|
Precision of 'float', 'double', and 'decimal'
Before looking at the output of the following program, please take a guess at the values
of these expressions:
- ((float)1/3)*3
- ((double)1/3)*3
- ((decimal)1/3)*3
If you are an entry level programmer, your guess would be:
- ((float)1/3)*3 = 1
- ((double)1/3)*3 = 1
- ((decimal)1/3)*3 = 1
If you are an experienced programmer, your guess would be:
- ((float)1/3)*3 = 0.9999999
- ((double)1/3)*3 = 0.999999999999999
- ((decimal)1/3)*3 = 0.9999999999999999999999999999
But both answers are wrong. In C#, you will get:
- ((float)1/3)*3 = 1
- ((double)1/3)*3 = 1
- ((decimal)1/3)*3 = 0.9999999999999999999999999999
The following program demonstrates this answer in a slightly different way:
using System;
class Precision {
public static void Main() {
float f;
double d;
decimal m;
for (int i=1; i<=2; i++) {
f = (float)i/3;
d = (double)i/3;
m = (decimal)i/3;
Console.WriteLine("Testing {0}/3:", i);
Console.WriteLine(" f = {0}", f);
Console.WriteLine(" d = {0}", d);
Console.WriteLine(" m = {0}", m);
Console.WriteLine(" f*3 = {0}", f*3);
Console.WriteLine(" d*3 = {0}", d*3);
Console.WriteLine(" m*3 = {0}", m*3);
Console.WriteLine(" (double)f*3 = {0}", (double)f*3);
Console.WriteLine(" (decimal)f*3 = {0}", (decimal)f*3);
Console.WriteLine(" (decimal)d*3 = {0}", (decimal)d*3);
Console.WriteLine(" (double)((float)i/3)*3 = {0}",
(double)((float)i/3)*3);
}
}
}
Output:
Testing 1/3:
f = 0.3333333
d = 0.333333333333333
m = 0.3333333333333333333333333333
f*3 = 1
d*3 = 1
m*3 = 0.9999999999999999999999999999
(double)f*3 = 1.00000002980232
(decimal)f*3 = 0.9999999
(decimal)d*3 = 0.999999999999999
(double)((float)i/3)*3 = 1
Testing 2/3:
f = 0.6666667
d = 0.666666666666667
m = 0.6666666666666666666666666667
f*3 = 2
d*3 = 2
m*3 = 2.0000000000000000000000000001
(double)f*3 = 2.00000005960464
(decimal)f*3 = 2.0000001
(decimal)d*3 = 2.000000000000001
(double)((float)i/3)*3 = 2
Question 1: Why ((float)1/3)*3 = 1?
Don't try to answer it with the possibility that
the compiler is smart, and optimizes (/3)*3 as *1. I have tried to get the /3 part
from a variable like the demonstration program, from a method, or even from another
class that compiled separately. They all give me the same result.
I believe that the answer is related to the storage algorithm of the float data type.
But I don't have time to figure it out yet.
Question 2: If ((float)1/3)*3 = 1, then why ((decimal)1/3)*3 = 0.9999999999999999999999999999?
I also don't have a clear answer yet. But I believe that the answer is also related to
the storage algorithm of the data types.
No matter what the answer is, this behavior is telling us that "decimal" is not a natural
extension of "float", like "double" extending "float".
If you are a designer, don't ask your developers to change "double" to "decimal"
immediately. Do some experiments first.
Question 3: If ((float)1/3)*3 = 1, and ((decimal)1/3)*3 < 1, then is the float type more
accurate that the decimal type?
I will leave this question to you to answer.
Question 4: If f is (float)1/3 = 0.3333333, then why (double)f*3 = 1.00000002980232?
I am expecting (double)f*3 = 0.9999999, because casting f into "double" will force the
expression to be evaluated in double precision, giving 0.9999999. What do you think?
Question 5: If (double)f*3 = 1.00000002980232, why (double)((float)1/3)*3 = 1?
The difference here is that (float)1/3 is evaluated first and stored in a variable f.
My guess to what happened here is that all floating data expressions are evaluated
in double precision. What do you think?
|