rounding error

  • Thread starter Thread starter Steve Lloyd
  • Start date Start date
S

Steve Lloyd

Hi,

Can anyone please explain this rounding error on a windows forms project.
I take an input and need to convert from a percentage to a decimal so i do:
dim result as double = cdbl(textbox.text) / 100

when 9.85 is entered into the text box "result" is calcualted as
0.09849999999999999.

For debugging i changed it to cdbl(textbox.text) / 1000 and also /10. both
of these gave the accurate answers, 0.00985 and 0.985 respectively.

Any ideas?

Thanks for any help

Steve.
 
Hi,

Can anyone please explain this rounding error on a windows forms project.
I take an input and need to convert from a percentage to a decimal so i do:
dim result as double = cdbl(textbox.text) / 100

when 9.85 is entered into the text box "result" is calcualted as
0.09849999999999999.

For debugging i changed it to cdbl(textbox.text) / 1000 and also /10. both
of these gave the accurate answers, 0.00985 and 0.985 respectively.

Any ideas?

Thanks for any help

Steve.
Click to expand...

It all has to do with the accuracy of doubles and there isn't too much
you can do about it. You might do some searching for the exact "whys",
this topic has been covered numerous times on most programming
newsgroups.

Thanks,

Seth Rowe [MVP]
 
Use a Decimal rather than a Double for your result variable

Dim result As Decimal= cdec(textbox.text) / 100

hth

Guy
 
Back
Top