Help! Numbers not adding properly

[David Ricker]

I am having problems adding two numbers. I am trying to add 1.005 and
1.007 to come up with 2.012. Should be easy enough right? Problem is that
I keep getting 2.0119999999999996 as my result. Why is this happening? Is
there a way that I can get it to return the right result? Any help is
appreciated.

Thanks,
David J. Ricker II

 

[CJ Taylor]

What data types are you using?

 

[David Ricker]

Doubles. I have also tried simply typing 1.005+1.007 into the watch window
and it gives the same result.

Thanks,
David J. Ricker II

 

[CJ Taylor]

What the he.....

Try a decimal. see what happens.

 

[David Ricker]

Strange, a decimal does work. Does anyone have any idea why a double won't
work in this situation... is it a bug?

Thanks CJ, I can use the decimals to get the desired results in my program.

Thanks Again,
David J. Ricker II

 

[Cor]

Hi David,

I think that this will help you.

I asume your decimal separator is a dot and not a comma

http://msdn.microsoft.com/library/d...ef/html/frlrfsystemdecimalclassroundtopic.asp

I hope this helps a little bit.

Cor

 

[CJ Taylor]

One would think so. But I don't know the honest answer. My best guess
would be back to high school chemistry days and understanding the use of Sig
Figs... Could be something to do with that, even though we only have 3 to
work with so I don't know what the big deal is.

I've had the same issue before, and something to do with decimal having a
precision scale is the reason I've heard before. But I quesiton everything.
=)

Then again... it could be a bug, but I'm probably wrong... but if it was,
that sounds like it would be processor level instead of code level.

-CJ

 

[Bob]

http://www.math.grin.edu/~stone/courses/fundamentals/IEEE-reals.html

Bob

 

[Jay B. Harlow [MVP - Outlook]]

David,
In addition to CJ & Cor's posts.

http://msdn.microsoft.com/library/d...ry/en-us/vbcn7/html/vbconNonintegralTypes.asp

1.007 & 2.012 are by default a Double. Double & Single are base 2 floating
point numbers.

Decimal is a base 10 floating point number. (technally base 2, but think of
it as base 10 ;-))

Decimals are stored as a whole number (1234.) that is scaled, where as
Single & Double are stored as a fraction number (1.1234) that is scaled.
Seeing as both are represented as binary number internally, a binary whole
number can store all decimal numbers, where as a binary fraction cannot
store all decimal fractions. Hence the rounding issue you are seeing.

Hope this helps
Jay

 

[David Ricker]

Thanks Jay,

At least now I know that it is by design.

 

[Herfried K. Wagner [MVP]]

I am having problems adding two numbers. I am trying to add 1.005 and
1.007 to come up with 2.012. Should be easy enough right? Problem is that
I keep getting 2.0119999999999996 as my result. Why is this happening? Is
there a way that I can get it to return the right result? Any help is
appreciated.

<http://research.microsoft.com/~hollasch/cgindex/coding/ieeefloat.html>
<http://support.microsoft.com/default.aspx?scid=kb;de;42980>