Double+Int16

  • Thread starter Thread starter Doug Bell
  • Start date Start date
D

Doug Bell

Hi,
I am having a bit of a resolution problem.
I have a form that accepts elapsed time in various formats including 1.5 and
1:30

and converting it to minutes but I am getting a resolution problem.

Can someone explain this to me, my code:
dblMins = Val(stLoadSOs)

intHrs = Int(dblMins)

dblMins = dblMins - intHrs

intMins = intHrs * 60 + Int(dblMins * intTim)

if the input is '1:20' then the unshown code sets

stLoadSOs = "1.20" and intTim = 100

dblMins = 1.2 (correctly)

intHrs = 1 (correctly)

but then dblMins = dblMins - intHrs shows 19.999999 (incorrectly)

1.2 - 1 should equal 0.2

What is causing this resolution problem?



Thanks



Doug
 
Doug Bell said:
I am having a bit of a resolution problem.
I have a form that accepts elapsed time in various formats including 1.5 and
1:30

and converting it to minutes but I am getting a resolution problem.

Can someone explain this to me, my code:
dblMins = Val(stLoadSOs)

intHrs = Int(dblMins)

dblMins = dblMins - intHrs

intMins = intHrs * 60 + Int(dblMins * intTim)

if the input is '1:20' then the unshown code sets

stLoadSOs = "1.20" and intTim = 100

dblMins = 1.2 (correctly)

intHrs = 1 (correctly)

but then dblMins = dblMins - intHrs shows 19.999999 (incorrectly)

1.2 - 1 should equal 0.2

What is causing this resolution problem?
Click to expand...

Your misunderstanding of floating point numbers is faulty.

dblMins isn't actually 1.2, as that isn't representable in binary
floating point. That's just how the debugger (or whatever) is showing
it.

See http://www.pobox.com/~skeet/csharp/floatingpoint.html
 
Back
Top