Division Error?

  • Thread starter Thread starter rhl
  • Start date Start date
R

rhl

I just came across this weird division result with VB.NET.

2/3 = 0.66666666666666663

With VB6, I get

2/3 =0.666666666666667

Using the Windows Calculator results in
0.66666666666666666666666666666667

I'm sure I'm not the first one to notice this...what gives?

rhl
 
The differences are due to the math being done on different data types.

VB 6's, VB .NET's and Windows don't all have comprable data types.
 
Exact code ? I've got the same result than in VB6 with Debug.WriteLine(2 /
3)

Anyway, all those results are wrong and obviously you'll never be able to
display or even compute the correct result. More generally this is a well
know problem with "real" numbers.

You have several ways to workaround this depending on the problem you have
with this. You could use a better type (such as decimal) for increasing (and
possibly "guaranteed" precision), you could store this a a fraction etc...
 
Back
Top