Help converting a number.

  • Thread starter Thread starter Lance McGonigal
  • Start date Start date
L

Lance McGonigal

Folks...Thanks in advance for your help.

I have a program that totals up receipts and sales from an inventory system.
The action quantity is signed and a number with 4 decimals. When I sum them
up I get the following number: 1.11022302462516E-15.

I assume its a decimal that is greater than 4 places. Can anyone tell me
how to convert it to a number I can use?

Thanks!
 
Lance said:
Folks...Thanks in advance for your help.

I have a program that totals up receipts and sales from an inventory system.
The action quantity is signed and a number with 4 decimals. When I sum them
up I get the following number: 1.11022302462516E-15.

I assume its a decimal that is greater than 4 places. Can anyone tell me
how to convert it to a number I can use?
Click to expand...


Due to the lack of infinite precision in any floating point
calculation, whether it be on a computer or with pencil and
paper, many numbers are slightly off of what was entered.
Summing a lot of these small errors can sometimes result in
a noticable discrepency.

Depending on what you expect, there are two approaches to
deriving an acceptable total. One is really easy, just
round the result to 4 places. If you're displaying the
total in a text box, just set its Format property to
something like 0.0000

Ocassionally less accurate, but more understandable by
people that add up a column of numbers and complain that
your total is wrong, is to round the numbers to 4 places
before adding them to the total.

For situations where you have 4 or fewer places, you can get
100% accuracy by using the Currency data type (which is
fixed point).
 
It is scientific notation.
E-15 stand for 10 raised to the -15th power.
(Or shift the decimal point 15 places to the left!)

The number is so small you should not want to make use of it.
You want to round it away.
 
Back
Top