U
Uma Abhyankar
Hello,
We are facing precision issues with addition, multiplication or division of
"double"
Issue1:
#####
The output of 0.1 + 0.2 is not 0.3
It is 0.30000000000000004
How do we tackle this??
Issue2:
#####
Minute value: 8.6994833333333332
Case1: From Minutes to Seconds:
=========================
Operation: 8.6994833333333332 * 60
Expected output = 521.968999999999992
Actual output = 521.969
-Give output of Case 1 to conversion again
Case2: From Seconds to Minutes:
=========================
Operation: 521.969 / 60
Expected output = same as input i.e. 8.6994833333333332
Actual output = 8.6994833333333333333333333333333
[As the input to Case2 was 521.969 and not 521.968999999999992]
How do we tackle this?
Thanks & Regards
Uma
We are facing precision issues with addition, multiplication or division of
"double"
Issue1:
#####
The output of 0.1 + 0.2 is not 0.3
It is 0.30000000000000004
How do we tackle this??
Issue2:
#####
Minute value: 8.6994833333333332
Case1: From Minutes to Seconds:
=========================
Operation: 8.6994833333333332 * 60
Expected output = 521.968999999999992
Actual output = 521.969
-Give output of Case 1 to conversion again
Case2: From Seconds to Minutes:
=========================
Operation: 521.969 / 60
Expected output = same as input i.e. 8.6994833333333332
Actual output = 8.6994833333333333333333333333333
[As the input to Case2 was 521.969 and not 521.968999999999992]
How do we tackle this?
Thanks & Regards
Uma