Math issue

  • Thread starter Thread starter Swarfmaker
  • Start date Start date
S

Swarfmaker

Hi,
I don't use Excel a lot and am puzzled by this strange behaviour.

I put =1248.72-463 in a cell and get the expected answer of 785.72
I put 1248.72 in cell C5 and 463 in Cell E5 then do =C5-E5 and get
786.17457688

What's going on here?

TIA
Iain
 
If you format E5 as General and make the column wide enough you'll probably
find that its value is 462.5454231
 
I cannot reproduce this. When you say "put in" do you mean TYPE or are there
formulas in the cells.
If the latter then Niek has given you the answer.
best wishes
 
Many thanks.

Iain


Niek Otten said:
If you format E5 as General and make the column wide enough you'll
probably find that its value is 462.5454231


--
Kind regards,

Niek Otten
Microsoft MVP - Excel
 
Hi,

I'm going to have to second Bernard, I tested and could not reproduce the
problem.

I suspect that one of the cells has a formula and you are displaying it to
an accuracy of only two decimals when in fact Excel has calculated more
decimals, and you test by changing the format to display more decimals.

I tested in both Excel 2003 and 2007 are you using another version?
 
I have also been having isues with simple arithmatic operations producing
small discrepancies from the expected results.

The simplest example I have is as follows:

I open a new spreadsheet and type "=2.07-2" into A1.
The result displays 0.07. If I re-format to number and expand the decimals
to 16, the preview shows a different value (either + or - a small amount).

This is confirmed what is stored in the cell; if I type in "0.07" into B1; a
formula of "=A1=B1" will produce FALSE.
 
I have also been having isues with simple
arithmatic operations producing small
discrepancies from the expected results.

Yours is a common problem. For best results, make liberal use of the
ROUND function.

I open a new spreadsheet and type "=2.07-2" into A1.
The result displays 0.07.
[....]
if I type in "0.07" into B1; a formula of "=A1=B1"
will produce FALSE.

Enter =round(2.07-2,2) into A1.

Note: Rounding is not a panacea. There are still arithmetic
operations that can have surprising results. The problem is that most
decimal fractions cannot be represented exactly in a finite number of
digits. Moreover, normal internal representation (IEEE floating
point) limts the number of finite digits.


----- original posting -----
 
If you format E5 as General and make the column wide
enough you'll probably find that its value is 462.5454231

And apparently you are right, based on the OP's response to you.

So apparently the OP did __not__ "put =1248.72-463 in a cell" and "put
1248.72 in cell C5 and 463 in Cell E5".

Does it bother anyone else when people ask for help and completely
misrepresent the facts?

My concern is not about the OP's behavior; that is simply a reflection
on him/her. Who cares?

My concern is for people who later search for threads that might be
related to a problem they are having, and they are misled into
believing that there is a problem in Excel as simple as this.

For the record: there is not!
 
To expand on joeu2004's reply, this is not an Excel issue; it is common to
almost all computer software and hardware. The number 2.07 (like most
terminating decimal fractions) is a non-terminating binary fraction that can
only be approximated in finite precision (just as 1/3 can only be
approximated as a decimal fraction). The IEEE double precision approximation
to 2.07 is
2.069999999999999840127884453977458178997039794921875
You do not see the difference from 2.07 because (as documented in Help)
Excel displays at most 15 digits. However, when you subtract 2, then you
correctly get
0.069999999999999840127884453977458178997039794921875
Which displays in 15 digits as 0.0699999999999998

The math is right. To adjust your expectations without having to learn the
details of binary representation of numbers, remember that digits beyond the
15th may not be what you expect and protect yourself accordingly.
Possibilities include
Test for an appropriately small difference between numbers instead of
exact equality
Round results as appropriate
MS cannot do this for you because they cannot predict what calculations you
will do. Any attempt at a general solution would actually reduce accuracy
when its assumptions are not satisfied.

Jerry
 
A few remarks in response to Jerry's excellent post.

<it is common to almost all computer software and hardware>

Not entirely true. There are (software) implementations of the COBOL and
other languages that do not suffer from this problem, because they use
either scaled integer representations internally or true binary (not
floating point) representations of numbers.
Both systems have the "disadvantage" of smaller possible min/max values
(although this theoretically is not necessary). But in commercial
applications they may be perfectly suitable (and even desirable!).
These software solutions are sometimes supported by Binary Coded Decimal
(BCD) hardware instructions in the computer, but that is not necessary to
implement decimal arithmetic.
And, almost any pocket or desk calculator will do this kind of calculation
correctly, although with limited capacity. So did all punced card tabulators
for many decades.

<digits beyond the 15th may not be what you expect >

We should be careful with descriptions like "digits beyond the 15th". First,
it's *decimal* digits, but most of us would assume that anyway. But some
manuals quite rightly mention "15 significant digits". That is because in a
number like

0.000000000000000123456789012345

no precision is lost.

<then you correctly get
0.069999999999999840127884453977458178997039794921875 Which displays in 15
digits as 0.0699999999999998>

I don't think that is right. It's not just a matter of displaying less
precision, there really *is* less precision, as can easily be seen by
applying some more calculation.

Then I think we should be careful using the frase "*the* IEE
representation". There are several IEE floating point representations and
there is a new IEE 754-2008 standard.

The last remark I would like to make is that we (including myself) often
confuse IEEE FP and binary, like in "can't be represented exactly binary".
Some numbers can be represented exactly using true binary, but not in IEEE,
using the same number of bits. That is because in FP some bits are used for
the exponent.

It is my conviction that, since Excel is widely (and, I think, mainly) used
for commercial applications, true decimal arithmetic should be available, at
least as an option.

--
Kind regards,

Niek Otten
Microsoft MVP - Excel

I am also convinced that the performance implications will be relatively
small.
 
0.069999999999999840127884453977458178997039794921875
I don't think that is right.

Just for grins, here's a math program increasing the precision of a
number on my Intel computer.

SetPrecision[2.07 - 2, 50]
0.069999999999999840127884453977458178997039794921875

A small blurb from help:
"...the number is padded with zeros. The zeros are taken to be in base
2. In base 10, the additional digits are usually not zeros."

- - -
Dana DeLouis
 
Back
Top