Floating point division

  • Thread starter Thread starter Tom C
  • Start date Start date
T

Tom C

The following result is from the immediate window in vs2005 just to
illustrate what I am seeing in code. Quite obviously, .08 / .1 does
not equal 0.79999999999999993. What is going on here?

? 8/1
8.0
? .8/1
0.8
? .8/.1
8.0
? .08/.1
0.79999999999999993
? .08/.01
8.0
? .008/.001
8.0
? .008/1
0.008
? .008/.01
0.8
? .008/.1
0.08
? .08/.1
 
Every beginnig developer gets burned by this sooner or later.

http://floating-point-gui.de/

-ralph
Click to expand...

I have probably been programming longer than you are old but aside
from that, I have never seen this and I am looking for the cliff notes
answer not a college degree in math so please don't reply with another
antagonistic comment and link.

thanks
 
I have probably been programming longer than you are old but aside
from that, I have never seen this and I am looking for the cliff notes
answer not a college degree in math so please don't reply with another
antagonistic comment and link.
Click to expand...

Well, now, you have something to learn about floating-point
arithmetic. The site mentioned above appears to be a good
introduction to the issue.

The antagonism is actually yours. ralph's answer was very good.

You have some reading to do.

Sincerely,

Gene Wirchenko
 
I have probably been programming longer than you are old but aside
from that, I have never seen this and I am looking for the cliff notes
answer not a college degree in math so please don't reply with another
antagonistic comment and link.

thanks
Click to expand...

I apologize if I ruffled your feathers. Not intended, just wanted to
let you know this is a common issue EVERYONE runs into sooner or
later. Surprised if you have been programming for more years than I
(34 years) you haven't encountered this before.

I did provide the 'Cliff Notes' answer. If you want the real
nitty-gritty go here. (PS: Note the title.)

"What Every Computer Scientist Should Know About Floating-Point
Arithmetic"
http://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html

-ralph
 
Tom said:
I have probably been programming longer than you are old but aside
from that, I have never seen this and I am looking for the cliff notes
answer not a college degree in math so please don't reply with another
antagonistic comment and link.
Click to expand...

I've been programming for close to 30 years and I *still* get burned by fp
problems on occasion.
 
I apologize if I ruffled your feathers. Not intended, just wanted to
let you know this is a common issue EVERYONE runs into sooner or
later. Surprised if you have been programming for more years than I
(34 years) you haven't encountered this before.

I did provide the 'Cliff Notes' answer. If you want the real
nitty-gritty go here. (PS: Note the title.)

"What Every Computer Scientist Should Know About Floating-Point
Arithmetic"
http://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html

-ralph
Click to expand...

That hurt my head when I read it. The other article was more on my
level. Fluoride in the water must have dumbed me down :)

Good to know that I have been doing something right...just didn't knw it
until now.

TMA
 
Back
Top