SqlDecimal calculations in .NET 2.0

  • Thread starter Thread starter Beat Bucheli
  • Start date Start date
B

Beat Bucheli

Hello,

I'm trying to do some simple calculations with System.Data.SqlDecimal on
..NET 2.0:

SqlDecimal one = new SqlDecimal(1);
SqlDecimal seven = new SqlDecimal(7);
SqlDecimal oneBySeven = SqlDecimal.Divide(one, seven);

The precision of the result in oneBySeven is way below my expectations. I
get 0.142857, but I expected something like 0.1428571428571428571428571429
or even more precise. Additional values reported by the debugger:

// Data[0] Data[1] Data[2] Data[3] Precision
Scale Value
// one 0x00000001 0x00000000 0x00000000 0x00000000 1 0
{1}
// seven 0x00000007 0x00000000 0x00000000 0x00000000 1 0
{7}
// oneBySeven 0x00022e09 0x00000000 0x00000000 0x00000000 7 6
{0.142857}

Any explanations? Thanks!
Beat
 
I'm trying to do some simple calculations with System.Data.SqlDecimal on
.NET 2.0:

SqlDecimal one = new SqlDecimal(1);
SqlDecimal seven = new SqlDecimal(7);
SqlDecimal oneBySeven = SqlDecimal.Divide(one, seven);

The precision of the result in oneBySeven is way below my expectations. I
get 0.142857, but I expected something like 0.1428571428571428571428571429
or even more precise. Additional values reported by the debugger:

// Data[0] Data[1] Data[2] Data[3] Precision
Scale Value
// one 0x00000001 0x00000000 0x00000000 0x00000000 1
0 {1}
// seven 0x00000007 0x00000000 0x00000000 0x00000000 1
0 {7}
// oneBySeven 0x00022e09 0x00000000 0x00000000 0x00000000 7
6 {0.142857}

Any explanations? Thanks!
Beat
Click to expand...

Sure. This explains all:

Precision, Scale, and Length (Transact-SQL)
http://msdn2.microsoft.com/en-us/library/ms190476.aspx


Given decimals e1 and e1, with precision p1 and p2 and scale s1 and s2
respectively, the quotient (e1 / e2)
will have precision = p1 - s1 + s2 + max(6, s1 + p2 + 1) and scale = max(6,
s1 + p2 + 1)


In a table or in TSQL a decimal type is declared with a precision and scale.
When you use this .NET constructor the precision and scale are inferred from
the constructor argument. If you want more precision in the quotient you
can specify more precision in the arguments:

SqlDecimal one = SqlDecimal.ConvertToPrecScale(new SqlDecimal(1), 20,
12);
SqlDecimal seven = SqlDecimal.ConvertToPrecScale(new SqlDecimal(7), 20,
12); ;
SqlDecimal oneBySeven = SqlDecimal.Divide(one, seven);

Console.WriteLine(oneBySeven);

David
 
Back
Top