rounding problem

  • Thread starter Thread starter Isaak
  • Start date Start date
I

Isaak

In my code I have to calculate the difference between two
numeric values stored as strings and then store the result
as a string.

Using cstr(cdbl(strA) - cdbl(strB)) works fine most of the
time, but I encountered at least one exception:

when strA = 0.92 and strB = 0.85 the result always is:

cdbl(0.92) - cdbl(0.85) = 0.070000000000000062

Any ideas on why this is happening?

I think I have a workaround by using convert.todecimal
inplace of cdbl, but I'm afraid this can result in some
precision loss when converting from decimal back to string.
 
Isaak said:
In my code I have to calculate the difference between two
numeric values stored as strings and then store the result
as a string.

Using cstr(cdbl(strA) - cdbl(strB)) works fine most of the
time, but I encountered at least one exception:

when strA = 0.92 and strB = 0.85 the result always is:

cdbl(0.92) - cdbl(0.85) = 0.070000000000000062

Any ideas on why this is happening?
Click to expand...

Yes - it's because neither 0.92 nor 0.85 are exactly representable in
floating binary point.

See http://www.pobox.com/~skeet/csharp/floatingpoint.html for more
information.
 
Back
Top