VB.Net Maths Bug?? Please Help.

  • Thread starter Thread starter Scott Gunn
  • Start date Start date
S

Scott Gunn

Hello All,

In VB.Net 2003 why is this the result to a simple maths calculation?

3.53 - 3.52 = 0.00999999046325684

Its not just this one either try this:

1.001 - 0.001 = 0.999999999999989

Coding example:
Dim Calc1 As Double = 3.53F
Dim Calc2 As Double = 3.52F
Msgbox(Calc1 - Calc2)

Why? Please help

TBS
 
Hi Scott,

Scott Gunn said:
Hello All,

In VB.Net 2003 why is this the result to a simple maths calculation?

3.53 - 3.52 = 0.00999999046325684

Its not just this one either try this:

1.001 - 0.001 = 0.999999999999989

Coding example:
Dim Calc1 As Double = 3.53F
Dim Calc2 As Double = 3.52F
Msgbox(Calc1 - Calc2)

Why? Please help

TBS
Click to expand...
You are seeing the expected result of an operation on two numbers that
cannot be exactly represented in the IEEE format for floating point numbers,
whether single precision or double precision. It is not a bug. Go to this
site http://docs.sun.com/source/806-3568/ncg_goldberg.html and other similar
explanations of the limitations of limited precision math.

Charles
 
Welcome to the world of computers!

Computers do arithmetic in binary. The numbers 0.1, 0.01, etc., have the
same problem in binary that 1/3 does in decimal: no matter how many digits
you have, you can never get them exact. (1/3 is close to 0.3333, closer to
0.33333, and so forth.)

..NET Framework provides the Decimal data type for exact calculations with
multiples of 1/10 and 1/100.
 
Back
Top