Integer to number, too many decimals

  • Thread starter Thread starter Eric
  • Start date Start date
E

Eric

Hi,

In a data table I have records numbered from 801 to 809.
I also use these numbers as row number then 8.01 to 8.09.

To do so I just devide the record number by 100 and put it in a variable of
type decimal (also tried double and single).

At one point I compare this value with a number to do an evaluation:

itemnr = CDec(rw("itemnr")) / 100

"If itemnr = 8.03 Or itemnr = 8.04 Or itemnr = 8.06 Or itemnr = 8.07 Then"

Well it should look like that, but it actually showing:
" If itemnr = 8.0299999999999994 Or itemnr = 8.0399999999999991 Or itemnr =
8.0600000000000005 Or itemnr = 8.0700000000000003 Then"

How can I get rid of the extra decimals?
I don't need them and it's confusing.

rg,
Eric
 
Am 17.06.2010 16:19, schrieb Eric:
Hi,

In a data table I have records numbered from 801 to 809.
I also use these numbers as row number then 8.01 to 8.09.

To do so I just devide the record number by 100 and put it in a variable of
type decimal (also tried double and single).

At one point I compare this value with a number to do an evaluation:

itemnr = CDec(rw("itemnr")) / 100

"If itemnr = 8.03 Or itemnr = 8.04 Or itemnr = 8.06 Or itemnr = 8.07 Then"

Well it should look like that, but it actually showing:
" If itemnr = 8.0299999999999994 Or itemnr = 8.0399999999999991 Or itemnr =
8.0600000000000005 Or itemnr = 8.0700000000000003 Then"

How can I get rid of the extra decimals?
I don't need them and it's confusing.
Click to expand...

If 'itemnr' is Decimal or Double, I can't repro the problem:

Dim value As Decimal = 803
Dim itemnr As Double 'As Decimal: same behavior

itemnr = value / 100

If itemnr = 8.03 Then
MsgBox("equal") 'always shown
End If
 
Eric said:
Hi,

In a data table I have records numbered from 801 to 809.
I also use these numbers as row number then 8.01 to 8.09.

To do so I just devide the record number by 100 and put it in a variable of
type decimal (also tried double and single).

At one point I compare this value with a number to do an evaluation:

itemnr = CDec(rw("itemnr")) / 100

"If itemnr = 8.03 Or itemnr = 8.04 Or itemnr = 8.06 Or itemnr = 8.07 Then"

Well it should look like that, but it actually showing:
" If itemnr = 8.0299999999999994 Or itemnr = 8.0399999999999991 Or itemnr =
8.0600000000000005 Or itemnr = 8.0700000000000003 Then"
Click to expand...

Fractional numbers are always approximate --- if you're going to
proceed with this, you're going to have to deal with the rounding
issue.

I don't understand what you're doing well enough to say what you should
do, but I'd reconsider it if I was you....
 
Hello, Eric,

The numbers 8.xx are being interpreted as floating point values. Appending
a "D" to them should correct the problem that you are seeing. That is, try:

"If itemnr = 8.03D Or itemnr = 8.04D Or ...

It is very important to have a clear understanding of the different number
types that can be used and what their limitations are. Floating point values
particularly seem to cause some programmers much confusion. See:

http://docs.sun.com/source/806-3568/ncg_goldberg.html

for a good detailed explanation.

Cheers,
Randy
 
Back
Top