且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

浮动正在转换我的价值观

更新时间:2023-02-27 09:27:49

http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems



简而言之,由于浮点数字代表实数,因此您分配给浮点数的数字并不总是您返回的数字。您指定的值将被转换为可用科学计数法表示的最接近的值,其幅度由基数2确定。在999999.99的情况下,可以表示为具有相同数量信号的浮点数的数字是7.6293945 * 2 17 = 999999.99504,当四舍五入为相同的信号时,数字是1,000,000.00。这可能不是确切的情况,但像这样的错误是固有的使用浮动。



不要使用浮点类型的情况下,数字的准确性在给定的精度水平是必需的。相反,使用小数类型,这将保留输入值的精度。


I have a method that tests a value is within the range allowed on fields. If it is outside the range returns null and if inside returns the value.

internal float? ExtractMoneyInRangeAndPrecision(string fieldValue, string fieldName, float min, float max, int scale, int lineNumber)
    {          
        float returnValue;

        //Check whether valid float if
        if (float.TryParse(fieldValue, out returnValue))
        {
            //Check whether in range
            if (returnValue >= min && returnValue <= max)
            {
                int decPosition = 0;
                decPosition = fieldValue.IndexOf('.');

                if (
                    (decPosition == -1) ||
                    ((decPosition != -1) && (fieldValue.Substring(decPosition, fieldValue.Length - decPosition).Length -1 <= scale))
                    )
                {
                    return returnValue;
                }
            }
        }            

        return null;
    }

Here is my unit test:

[TestMethod()]
    [DeploymentItem("ImporterEngine.dll")]
    public void ExtractMoneyInRangeAndPrecisionTest_OutsideRange()
    {
        MockSyntaxValidator target = new MockSyntaxValidator("", 0);
        string fieldValue = "1000000";
        string fieldName = "";
        float min = 1;
        float max = 999999.99f;
        int scale = 2;
        int lineNumber = 0;
        float? Int16RangeReturned;

        Int16RangeReturned = target.ExtractMoneyInRangeAndPrecision(fieldValue, fieldName, min, max, scale, lineNumber);

        Assert.IsNull(Int16RangeReturned);
    }

As you can see the max is 999999.99 but when the method takes it in it changes it to 1,000,000

Why is this?

http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems

In short, because of the way floating-point numbers represent real numbers, the number you assign to a float is not always the number you get back out. The value you specify is converted to the nearest value that can be represented in scientific notation with a magnitude determined by a base of 2.

In the case of 999999.99, the nearest number that can be represented as a float with the same number of sig figs is 7.6293945 * 217 = 999999.99504, which when rounded to the same sig figs is 1,000,000.00. This may not be the EXACT case, but error like this is inherent in the use of floats.

Do not use floating-point types in situations where the accuracy of the number at a given level of precision is required. Instead, use the decimal type, which will retain the precision of values entered.