Trouble calculating exact values with double datatypes

  • Thread starter Thread starter Lothar Behrens
  • Start date Start date
L

Lothar Behrens

Hi,

I have found a problem in my unit tests that my code does not
calculate exact values.
My code produced 2.8000000000000003 but not 2.7000000000000003. In
both cases
I expected 2.8 / 2.7.

And when I change the calculation I get exact values. Why does this
happen?

I am using Visual Studio 2008 Professional 9.0.30729.1 SP and .NET 3.5
SP1.

Here is a sample from my watch window:

28.000000 * (100.000000 / 100.0000000) / (100.0000000 / 10.0000000)
2.8 double
28.000000 * (100.000000 / 100.0000000) * (10.0000000 / 100.0000000)
2.8000000000000003 double
27.000000 * (100.000000 / 100.0000000) * (10.0000000 / 100.0000000)
2.7 double

Thanks

Lothar
 
Lothar said:
I have found a problem in my unit tests that my code does not
calculate exact values.
My code produced 2.8000000000000003 but not 2.7000000000000003. In
both cases
I expected 2.8 / 2.7.

And when I change the calculation I get exact values. Why does this
happen?
Click to expand...

You get the precision that is possible with that number type
(http://en.wikipedia.org/wiki/Double_precision_floating-point_format)
with binary representation of numbers.
Using decimal instead of double would give you better precision. If you
want or need to use double then round as required.
 
You get the precision that is possible with that number type
(http://en.wikipedia.org/wiki/Double_precision_floating-point_format)
with binary representation of numbers.
Using decimal instead of double would give you better precision. If you
want or need to use double then round as required.
Click to expand...

I don't expect numbers like 2.8000000000000003 if I effectively divide
28 through 10.
If I divide 10 through 3, then I expect as much precision as the data
type will provide.

I have compared the code with C++ (not .NET) and there I also have
additionally compared
with using float. That would give me a more expectable result.

Rounding is an option, but only with the result of a calculation to
avoid rounding errors.
Actually I am using rounding in my unittests to get these passing.

I have to play a bit more with floating point values and
calculation :-)
Also I have to play with the datatypes I have choosen in my DB model.
(LinqToSQL / SQLSERVER)

Thanks

Lothar
 
I don't expect numbers like 2.8000000000000003 if I effectively divide
28 through 10.
Click to expand...

Then you clearly don't understand IEEE floating-point numbers. Don't feel
bad, most people don't. Even seemingly innocuous/"obvious" calculations can
lead to these odd values. Dividing by 10 when representing numbers in binary
is no "safer" than dividing by Pi. There's a URL that people post in another
group I read to help explain IEEE numbers to people who are having the same
problems you are. If I can find it I'll post it here.
 
Then you clearly don't understand IEEE floating-point numbers. Don't feel
bad, most people don't. Even seemingly innocuous/"obvious" calculations
can lead to these odd values. Dividing by 10 when representing numbers in
binary is no "safer" than dividing by Pi. There's a URL that people post
in another group I read to help explain IEEE numbers to people who are
having the same problems you are. If I can find it I'll post it here.
Click to expand...

Found one: http://support.microsoft.com/kb/42980/EN-US/
 
Back
Top