Math problem using a Double

  • Thread starter Thread starter Samuel
  • Start date Start date
S

Samuel

Hi

I make the following calculation 1119.8*0.175 and I assign it to a double
then I round it using the Math.Round(Val,2) method and I get 195.96 instead
of 195.67 as I think it should be and the reason is due to the fact that the
value is the double variable before rounding is 195.96499999999997

Is it inaccuracy of the Double or this is the true result?

Can anyone suggest what can be done if it is the double which is inaccurate

Thank you,
Samuel
 
Samuel said:
Hi

I make the following calculation 1119.8*0.175 and I assign it to a double
then I round it using the Math.Round(Val,2) method and I get 195.96 instead
of 195.67 as I think it should be and the reason is due to the fact that the
value is the double variable before rounding is 195.96499999999997

Is it inaccuracy of the Double or this is the true result?

Can anyone suggest what can be done if it is the double which is inaccurate

Thank you,
Samuel
Click to expand...

195.96499999999997 rounded to two places should be 195.96. The .dd4 means
it rounds down.
 
Hi

I make the following calculation 1119.8*0.175 and I assign it to a double
then I round it using the Math.Round(Val,2) method and I get 195.96 instead
of 195.67 as I think it should be and the reason is due to the fact that the
value is the double variable before rounding is 195.96499999999997

Is it inaccuracy of the Double or this is the true result?

Can anyone suggest what can be done if it is the double which is inaccurate

Thank you,
Samuel
Click to expand...

Floating point math problem. It's just the way it works :) Try using a
Decimal type.
 
just Human math VS binary computer math

so doing it the human way will give you the expected result

Dim x As Decimal = 1119.8 * 0.175

x = Math.Round(x, 2, MidpointRounding.AwayFromZero)



hth



michel
 
Samuel said:
I make the following calculation 1119.8*0.175 and I assign it to a double
then I round it using the Math.Round(Val,2) method and I get 195.96
instead of 195.67 as I think it should be and the reason is due to the
fact that the value is the double variable before rounding is
195.96499999999997

Is it inaccuracy of the Double or this is the true result?
Click to expand...

(Complete) Tutorial to Understand IEEE Floating-Point Errors
<URL:http://support.microsoft.com/?scid=kb;EN-US;42980>

IEEE Standard 754 Floating Point Numbers
Can anyone suggest what can be done if it is the double which is
inaccurate
Click to expand...

Take a look at the 'Decimal' type.
 
Samuel said:
Hi

I make the following calculation 1119.8*0.175 and I assign it to a
double then I round it using the Math.Round(Val,2) method and I get
195.96 instead of 195.67 as I think it should be and the reason is
due to the fact that the value is the double variable before rounding
is 195.96499999999997
Is it inaccuracy of the Double or this is the true result?

Can anyone suggest what can be done if it is the double which is
inaccurate
Thank you,
Samuel
Click to expand...

Although it is worthwhile to read up on floating point, in your case, the issue
is really Banker's Rounding, that is, rounding to the nearest even digit.

If you try 1119.8 * 0.175 - .01, you will get 195.95499999999998. This also
rounds to 195.96, because the 6 is even.

As Michel pointed out, you can specify the type of rounding by using x =
Math.Round(x, 2, MidpointRounding.AwayFromZero), which says to round halves up
for positive numbers. Or go old school, and use x = CDbl(x.ToString("0.00")).
 
Back
Top