Can excel calculate?

  • Thread starter Thread starter Arne
  • Start date Start date
A

Arne

What, exactly, is the problem? Computer floating point
arithmetic is not exact. 0.999999999999943 is very close
to 1 (5.7E-12% deviation is negligible for most
applications).
 
Please note that C++ does it right!

#include <iostream>
#include <math.h>
using namespace std;
int main()
{
double tempRound;
double tempInt, tempDec;
double number;
number = 64.1;

tempRound =floor(number*10+0.5)/10;
tempInt = (int)(tempRound);
tempDec = 10*(tempRound - tempInt);

cout << "tempRound = " << tempRound << endl;
cout << "tempInt = " << tempInt << endl;
cout << "tempDec = " << tempDec << endl;

system("pause");
return 0;
}

gives the right result:

tempRound = 64.1
tempInt = 64
tempDec = 1

Hmmm. Can one trust VBA or not?
Karl
 
Please note that C++ does it right!
Click to expand...
So does VBA if you understand how to manipulate data types. I think C++
may use more bytes to store

Dim number, tempRound As Double

Dim tempInt As Long
Dim tempDec As Long

number = 64.111
tempRound = number
tempInt = tempRound
tempDec = (10 * (tempRound - tempInt))

Debug.Print "tempRound = " & tempRound
Debug.Print "tempInt = " & tempInt
Debug.Print "tempDec = " & tempDec


tempRound = 64.1
tempInt = 64
tempDec = 1
 
Please note that C++ does it right!
Click to expand...
So does VBA if you know how to manipulate the data types

Dim number, tempRound As Double

Dim tempInt As Long
Dim tempDec As Long

number = 64.111
tempRound = number
tempInt = tempRound
tempDec = (10 * (tempRound - tempInt))

Debug.Print "tempRound = " & tempRound
Debug.Print "tempInt = " & tempInt
Debug.Print "tempDec = " & tempDec

tempRound = 64.111
tempInt = 64
tempDec = 1

I don't know C++, but my guess is that 'cout' has some default value of
number of dp to display
 
Karl Itschen said:
Please note that C++ does it right!

#include <iostream>
#include <math.h>
using namespace std;
int main()
{
double tempRound;
double tempInt, tempDec;
double number;
number = 64.1;

tempRound =floor(number*10+0.5)/10;
tempInt = (int)(tempRound);
tempDec = 10*(tempRound - tempInt);

cout << "tempRound = " << tempRound << endl;
cout << "tempInt = " << tempInt << endl;
cout << "tempDec = " << tempDec << endl;

system("pause");
return 0;
}

gives the right result:

tempRound = 64.1
tempInt = 64
tempDec = 1
Click to expand...

Well of course it does

The default precison used for doubles by cout
is only 6 decimal places

Try

cout.precision(15);

if you want a real comparison.

http://msdn.microsoft.com/library/d...floating_point_numbers_may_lose_precision.asp

Keith
 
Back
Top