weird math problem!!

  • Thread starter Thread starter Todd
  • Start date Start date
T

Todd

OK...I have a form that has two text boxes. These two fields are being
filled from a file import, which works fine. I am then passing these
values into a custom class with both values typed as DOUBLE (IE:
double.parse(mytextbox1.text) ).

When I subtract these two values, I'm get an erroneous .00000000000002
being added to the result of the subtraction!!

The form fills the instance of the class...

myclass.value1 = double.parse(mytextbox1.text)
myclass.value2 = double.parse(mytextbox2.text)

....then in another procedure I'm trying to do the math...


Dim MyValue as double = myclass.value1 - myclass.value2

No matter what the source numbers, I always get a .00000000000002
added into the figure!!!! The class values are always correct, it's
just the subtraction that generates the incorect figure!!

This is driving me crazy!!!!!!

Anyone have any idea!?!?!?!?!?!?!?!
 
thats expected behavior. the datatype double which is implemented as
twos complement, does not hold 1/100's accurately, similar to decimal
not holding thirds.

use the decimal datatype.

-- bruce (sqlwork.com)
 
Hi Todd

I found this URL which may help to explain things a bit. In summary -
the double, like float is a floating point representation of a number.
As such there is some error built into it. The article advocates using
Decimal data types when you want the precision. Your alternative is to
round the value. Not having done any work with arithmetic in .NET, I
don't know how.

This is well known with the float data type. If you need an example,
try to determine how many $0.1 items you can purchase with $1.00 (the
decimal places are important). The answer (using float data types) is
9.

http://techembassy.blogspot.com/2006/07/data-types-single-double-and-decimal.html

Hope this helps
Mike
 
I can
not believe this is "expected behaviour". Then again, I just read the
article about Excel 2007 addition problem! Can this be called
"Microsoft Math"?
Click to expand...

Hi Todd

This isn't a Microsoft thing. It's the IEEE or ISO (can't remember
which) specification. It's built into the standard for floating point
data types like float and double. Microsoft are obviously storing
their numeric data as floating point numbers as it takes less storage
than something more accurate.

Regards
Mike
 
Back
Top