How do I make my computer unstupid?

  • Thread starter Thread starter Guest
  • Start date Start date
G

Guest

I tell my computer a math problem (19.98-20.00) and it gives me a wako answer
(0.01999999999999996) what is wrong with it?
 
Jam,

I don' know much about arithmetic, but I can think of a lot of "stupid"
answers to your question, such as: To make your computer unstupid,
don't unfix it!

Good luck.
 
a computer is not a human :-)

from a computers point of view the answer is completely valid

computers work with the binary system humans work with the decimal system

so to get a human expected answer you should convert the values to the
decimal system and then perform your math

(CType(19.98, Decimal) - CType(20.0, Decimal)


regards

Michel Posseth
 
M.Posseth,
I would recommend:

Dim d As Decimal = 19.98D - 20.0D

Which causes the 19.98 & 20.0 to be Decimal literals & not Double literals.

Being Decimal literals saves you from converting the values to Decimal at
run time...

At the very least it makes the expression easier to read ;-)

--
Hope this helps
Jay B. Harlow [MVP - Outlook]
..NET Application Architect, Enthusiast, & Evangelist
T.S. Bradley - http://www.tsbradley.net


|
| a computer is not a human :-)
|
| from a computers point of view the answer is completely valid
|
| computers work with the binary system humans work with the decimal system
|
| so to get a human expected answer you should convert the values to the
| decimal system and then perform your math
|
| (CType(19.98, Decimal) - CType(20.0, Decimal)
|
|
| regards
|
| Michel Posseth
|
|
|
|
| "JAM" wrote:
|
| > I tell my computer a math problem (19.98-20.00) and it gives me a wako
answer
| > (0.01999999999999996) what is wrong with it?
 
Back
Top