Can excel calculate?

  • Thread starter Thread starter Mike
  • Start date Start date
M

Mike

Binary rounding error. Because data is stored in binary
code in a computer there are certain decimal values that
don't quite work out properly when you perform
mathematical operations on them. Binary is inherently a
bad numbering system because it can really only handle
integer values well. In fact, it's so bad that:

64.1-64=0.999999999999943
64.2-64=0.200000000000003
64.3-64=0.299999999999997
64.4-64=0.400000000000006
64.5-64=0.5 (Hey, this one works!)
64.6-64=0.599999999999994
64.7-64=0.700000000000003
64.8-64=0.799999999999997
64.9-64=0.900000000000006
 
You're not serious are you? You mean, as I know the
difference will be 0.1, I should round the wrong result
from the calculation so that it fits 0.1 !!!???

Btw, I'm using this value as a seed for a filename, and
I'm happier if my file is called:
64_1.txt and not
64_0.999999999999946.txt ;-)

I guess you have to be VERY careful with numbers in VB...
Karl
oh, and C++ does it right! (see my answer to your 1st post)
 
Final remark:

That C++ does it wrong in this case does not mean that the
outcome of a floating point operation is, in general,
exact. It is not. In no compiler. Ever.

Anyway, there are simpler ways of replacing the decimal .
by an underscore.

As about the ignoring rounding errors (or making them fit,
as you call it): suppose we were dealing with money, why
would rounding the results to 2 decimals be so bad? I
don't know about you, but I can spare 5.7e-12 cents any
time.
 
Thanks for your answers anyway.
Best regards
Karl
-----Original Message-----
Final remark:

That C++ does it wrong in this case does not mean that the
outcome of a floating point operation is, in general,
exact. It is not. In no compiler. Ever.

Anyway, there are simpler ways of replacing the decimal .
by an underscore.

As about the ignoring rounding errors (or making them fit,
as you call it): suppose we were dealing with money, why
would rounding the results to 2 decimals be so bad? I
don't know about you, but I can spare 5.7e-12 cents any
time.

.
Click to expand...
 
Back
Top