Numeric data type issue

  • Thread starter Thread starter Guest
  • Start date Start date
G

Guest

Why in the world is this happening:

double MyCount=0.00;
MyCount = (100 * 2611508) / 37792454;
Response.Write(MyCount.ToString("F2");

Always return a whole # (in this case 6). The value should be:
6.9101307896015432075408492922952
(as large as the Windows calc would display).

I really want the 6.91 or rounded to the nearest 10th
Ive tried defining MyCount as float, double, decimal. Any string format I
use either "N2", "F2", "P", always returns .00

What am I doing wrong? grrrrrrr
 
JP said:
Why in the world is this happening:

double MyCount=0.00;
MyCount = (100 * 2611508) / 37792454;
Response.Write(MyCount.ToString("F2");

Always return a whole # (in this case 6). The value should be:
6.9101307896015432075408492922952
(as large as the Windows calc would display).

I really want the 6.91 or rounded to the nearest 10th
Ive tried defining MyCount as float, double, decimal. Any string format I
use either "N2", "F2", "P", always returns .00
Click to expand...

The compiler thinks that the numbers you are using in your calculation are
integer values, so the calculation is performed with integers, regardless
of the "target type". If you want the calculation to use floating point
values, you should use the "f" modifier with at least one, or better all
of the values. Like this:

MyCount = (100f * 2611508) / 37792454;

or (better because it doesn't depend on internal calculation order)

MyCount = (100f * 2611508f) / 37792454f;


Oliver Sturm
 
So if:

int GroupCount=2611508;
int TotalCount = 37792454;
double MyCount=0.00;

then how do I use the f modifer with a veriable name
Actual caculation looks like this:
MyCount=(100f *GroupCount)/TotalCount

thx
 
JP said:
int GroupCount=2611508;
int TotalCount = 37792454;
double MyCount=0.00;

then how do I use the f modifer with a veriable name
Actual caculation looks like this:
MyCount=(100f *GroupCount)/TotalCount
Click to expand...

Well, in this specific case it should work fine, I think. But other than
that, if you actually had no numeric values left in your calculation and
only integer variables to deal with, you'd need to insert one or more
casts to make sure the right types are used for the calculation. Like here:

int GroupCount=2611508;
int TotalCount = 37792454;
int hundred = 100;
double MyCount=0.00;

MyCount = ((double) hundred * GroupCount) / TotalCount;


Oliver Sturm
 
JP said:
Why in the world is this happening:

double MyCount=0.00;
MyCount = (100 * 2611508) / 37792454;
Response.Write(MyCount.ToString("F2");

Always return a whole # (in this case 6). The value should be:
6.9101307896015432075408492922952
(as large as the Windows calc would display).

I really want the 6.91 or rounded to the nearest 10th
Ive tried defining MyCount as float, double, decimal. Any string format I
use either "N2", "F2", "P", always returns .00

What am I doing wrong? grrrrrrr
Click to expand...
All the numeric literals in your calculation are integers. Division with
integers yields an integer result *before* the result is stored. Storing the
result in a double cannot restore the lost precision.
Try this instead:

MyCount = (100.0 * 2611508.0) / 37792454.0;

Now you are working with double precision floating point literals.

Peter [MVP Visual Developer]
Jack of all trades, master of none.
 
Back
Top