Error in CF Decimal-type calculations?

  • Thread starter Thread starter DAve
  • Start date Start date
D

DAve

Hi -

Anyone seen this? Using the 'Decimal' type to do calculations in the
Compact Framework, some calculations are going wrong ...

For instance, running the following three lines should give "60" but instead
gives "41.43..." !!

Dim k as Integer = 60
Dim d as Decimal = 1 / New Decimal(k)
MessageBox.Show(1 / d)

Many other values for 'k' also seem to go wrong (eg 60, 70, 77, 91, 92,
.....).

Any help greatly appreciated!
 
Dave,

Replace your last line with:

double r = 1/(double)d;
MessageBox.Show("inverting again yields: " + r.ToString()); // 60

-Darren Shaffer
 
Thanks for the idea, but really - if we were happy with the accuracy of the
Double type, then we'd be using that instead of Decimal!

If there were someway to know when the Decimal type is going to go wrong,
then we could perhaps just use Double then. Do you know how to decide if
the problem is going to occur in Decimal?

For instance, the problem occurs when trying to calculate
1/0.0010416666666666666666666667 but not for 1/0.0010416666666666666667.

Many thanks for any more ideas!
 
Dave,

I was able to recreate the issue and the only workaround I found was to
either do the whole
problem using floating point types or to cast to double before using the
128-bit number as the
denominator of the inversion. Just out of curiosity, what sort of
application are you building that
has to have 128-bit precision? Sounds like it might be a really interesting
use of mobile technology.

-Darren
 
Back
Top