Help! Numbers not adding properly

  • Thread starter Thread starter David Ricker
  • Start date Start date
D

David Ricker

I am having problems adding two numbers. I am trying to add 1.005 and
1.007 to come up with 2.012. Should be easy enough right? Problem is that
I keep getting 2.0119999999999996 as my result. Why is this happening? Is
there a way that I can get it to return the right result? Any help is
appreciated.

Thanks,
David J. Ricker II
 
Doubles. I have also tried simply typing 1.005+1.007 into the watch window
and it gives the same result.

Thanks,
David J. Ricker II
 
Strange, a decimal does work. Does anyone have any idea why a double won't
work in this situation... is it a bug?

Thanks CJ, I can use the decimals to get the desired results in my program.

Thanks Again,
David J. Ricker II
 
One would think so. But I don't know the honest answer. My best guess
would be back to high school chemistry days and understanding the use of Sig
Figs... Could be something to do with that, even though we only have 3 to
work with so I don't know what the big deal is.

I've had the same issue before, and something to do with decimal having a
precision scale is the reason I've heard before. But I quesiton everything.
=)

Then again... it could be a bug, but I'm probably wrong... but if it was,
that sounds like it would be processor level instead of code level.

-CJ
 
David,
In addition to CJ & Cor's posts.

http://msdn.microsoft.com/library/d...ry/en-us/vbcn7/html/vbconNonintegralTypes.asp

1.007 & 2.012 are by default a Double. Double & Single are base 2 floating
point numbers.

Decimal is a base 10 floating point number. (technally base 2, but think of
it as base 10 ;-))

Decimals are stored as a whole number (1234.) that is scaled, where as
Single & Double are stored as a fraction number (1.1234) that is scaled.
Seeing as both are represented as binary number internally, a binary whole
number can store all decimal numbers, where as a binary fraction cannot
store all decimal fractions. Hence the rounding issue you are seeing.

Hope this helps
Jay
 
Back
Top