Excel precision issues with float/single

G

Guest

If I have a float variable with a value of 9.9 (as seen in the debugger), and
I set it to an Excel cell, the cell value is 9.89999961853027. However, if I
have a double variable with the same value of 9.9, setting that to an Excel
cell gives me the correct value.

(Note that the cell, in it's default size, looks correct, but if you expand
the cell, or click on the value and look on the editor area, the value is
incorrect.)

Am I doing something wrong? How do I fix this? Thanks much!
 
G

Guest

Cells hold all numbers as double, so when when the number is converted from
single to double, it takes on the imprecision. I suspect this has to do with
how floating point numbers are stored.
 

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments. After that, you can post your question and our members will help you out.

Ask a Question

Top