Single v. Double Help

  • Thread starter Thread starter Todd Heiks
  • Start date Start date
T

Todd Heiks

Why is 'ans1' true if double and false for single?

Module Module1
Sub Main()

Dim x As Double / Single
Dim y As Double / Single

Dim ans1 As Boolean
Dim ans2 As Boolean

x = 0.3000000000000001
y = Math.Round(x, 2)

ans1 = y = 0.3
ans2 = Math.Round(y, 2) = 0.3

End Sub
End Module

Thanks
 
Todd --

Am 08.04.2010 00:55, schrieb Todd Heiks:
Why is 'ans1' true if double and false for single?
[...]
Dim x As Double / Single
Dim y As Double / Single

Dim ans1 As Boolean
Dim ans2 As Boolean

x = 0.3000000000000001
y = Math.Round(x, 2)

ans1 = y = 0.3
ans2 = Math.Round(y, 2) = 0.3
Click to expand...

The reason is that the result cannot be represented exactly. It's in
general not recommended to compare floating point values of single and
double precision using the '=' operator.

More information:

(Complete) Tutorial to Understand IEEE Floating-Point Errors
<URL:http://support.microsoft.com/?scid=kb;EN-US;42980>
 
Back
Top