Is this a bug: 2.2 - 0.4 = 1.8 but 1.2 - 0.4 = 0.8000001 ?

  • Thread starter Thread starter Dave
  • Start date Start date
D

Dave

Hi,

Dim a, b As Single
a = 2.2
b = 0.4
Response.Write(a - b)

This gives: 1,8 (rendered according my regional settings)


But this gives: 0,8000001 (rendered according my regional settings):
Dim a, b As Single
a = 1.2
b = 0.4
Response.Write(a - b)


Any explantion for this, and how to solve this?? ( i rounded on two digits)
Thanks
Dave
 
Dave,
Try:

| Dim a, b As Decimal
| a = 2.2D
| b = 0.4D
| Response.Write(a - b)

| But this gives: 0,8000001 (rendered according my regional settings):
Is given as that is the approximate value that a Single can hold. A Single
holds base 2 floating point numbers, some (most) base 10 floating point
numbers cannot be exactly represented

For details see:

http://www.yoda.arachsys.com/csharp/floatingpoint.html

http://www.yoda.arachsys.com/csharp/decimal.html


NOTE: The D in 2.2D indicates it is a Decimal literal as opposed to 2.2
which is a Double literal, as opposed to 2.2F which is a Single literal.


--
Hope this helps
Jay B. Harlow [MVP - Outlook]
..NET Application Architect, Enthusiast, & Evangelist
T.S. Bradley - http://www.tsbradley.net


| Hi,
|
| Dim a, b As Single
| a = 2.2
| b = 0.4
| Response.Write(a - b)
|
| This gives: 1,8 (rendered according my regional settings)
|
|
| But this gives: 0,8000001 (rendered according my regional settings):
| Dim a, b As Single
| a = 1.2
| b = 0.4
| Response.Write(a - b)
|
|
| Any explantion for this, and how to solve this?? ( i rounded on two
digits)
| Thanks
| Dave
|
|
 
Thanks,

suppose i use variables a and b which are defined as decimals and gets their
values from anywhere else.
How can i put the "D" after the variable :
c=a-b D
 
| suppose i use variables a and b which are defined as decimals and gets
their
| values from anywhere else.
If you define the variable as Decimal, the variable will hold a Decimal.

| c=a-b D

a is a variable of type Decimal, it will return a Decimal value,
literals are not involved here, so you don't need the "literal type
character D".

The D is part of a Literal (constant if you will) value.

2.2 is a literal

a is a variable

b is a variable

c is a variable

2.2 is a Double literal

2.2D is a Decimal literal

2.2F is a Single literal (the F stands for Float).


--
Hope this helps
Jay B. Harlow [MVP - Outlook]
..NET Application Architect, Enthusiast, & Evangelist
T.S. Bradley - http://www.tsbradley.net


| Thanks,
|
| suppose i use variables a and b which are defined as decimals and gets
their
| values from anywhere else.
| How can i put the "D" after the variable :
| c=a-b D
|
|
|
|
| "Jay B. Harlow [MVP - Outlook]" <[email protected]> schreef in
| bericht | > Dave,
| > Try:
| >
| > | Dim a, b As Decimal
| > | a = 2.2D
| > | b = 0.4D
| > | Response.Write(a - b)
| >
| > | But this gives: 0,8000001 (rendered according my regional settings):
| > Is given as that is the approximate value that a Single can hold. A
Single
| > holds base 2 floating point numbers, some (most) base 10 floating point
| > numbers cannot be exactly represented
| >
| > For details see:
| >
| > http://www.yoda.arachsys.com/csharp/floatingpoint.html
| >
| > http://www.yoda.arachsys.com/csharp/decimal.html
| >
| >
| > NOTE: The D in 2.2D indicates it is a Decimal literal as opposed to 2.2
| > which is a Double literal, as opposed to 2.2F which is a Single literal.
| >
| >
| > --
| > Hope this helps
| > Jay B. Harlow [MVP - Outlook]
| > .NET Application Architect, Enthusiast, & Evangelist
| > T.S. Bradley - http://www.tsbradley.net
| >
| >
| > | > | Hi,
| > |
| > | Dim a, b As Single
| > | a = 2.2
| > | b = 0.4
| > | Response.Write(a - b)
| > |
| > | This gives: 1,8 (rendered according my regional settings)
| > |
| > |
| > | But this gives: 0,8000001 (rendered according my regional settings):
| > | Dim a, b As Single
| > | a = 1.2
| > | b = 0.4
| > | Response.Write(a - b)
| > |
| > |
| > | Any explantion for this, and how to solve this?? ( i rounded on two
| > digits)
| > | Thanks
| > | Dave
| > |
| > |
| >
| >
|
|
 
Back
Top