Simple INT error?

  • Thread starter Thread starter Roger
  • Start date Start date
R

Roger

In the debug window in Excel 2000, why does ? int( 0.6 * 100) give an answer of 59?

I know it's due to binary storage or something - but how do I correct it?
 
It appears to be caused by come conversion that is being performed:

? int( 0.6 * 100)
59

? int( 0.6! * 100)
60
? int( 0.6# * 100)
59


0.6 is double
0.6# is double
0.6! is single.
 
Another Intel error then. I haven't encountered this one either. Could anyone running an
Athlon processor please test this ?

It's easy enough to fix that particular calculation, problem is that INT is used a lot to
remove decimals from calculated big numbers (bigger than integer limits) and it would be
impossible to know when this happens or not. Hmmm. I have to proofread tons of code
because of this, buying Macs to my 3000 collegues is unfortunately no option <g>.

Best wishes Harald
Followup to newsgroup only please.
 
Harald Staff said:
buying Macs to my 3000 collegues is unfortunately no option <g>.
Click to expand...

but have you tried??? Bet you could get a substantial discount!<g>
 
I've never encountered that before. In XLv.X:

? int( 0.6 * 100)
60
Click to expand...

That's odd.

Excel 2002(10.4524.4219) SP-2
in a worksheet =INT(0.6*100) --> 60

But in the VB6.3 that comes with it, I get 59 in the Immediate window, too.
?int(0.6*100)
59

XP Pro
Processor x86 Family 15 Model 2 Stepping 4 GenuineIntel ~1993 Mhz





--ron
 
Back
Top