Forum Discussion

msaunders's avatar
msaunders
Contributor
10 years ago
Solved

Check point failing with matching values

Hi all,



I have a checkpoint that is checking that the value is properly stored in an infragistics grid. it seems to fail at certain times through the test. I am stepping the value up by .1 at a time and then checking from 0.1 to 10.  below are the first 3 fail points from 0-1.















Actual value
0.3

Expected value
0.3















Actual value
0.8

Expected value
0.8















Actual value
0.9

Expected value
0.9





I think it is a floating point math failure as when i change it to Math.round( i )

the failure points change, as you can see its rounding right but not what i wanted. its not consistent either, why does 1.5 work and 3.5 but not .5 and 2.5?















Actual value
0.5

Expected value
1















Actual value
2.5

Expected value
3















Actual value
4.5

Expected value
5





So next i tried toPrecision  that was working until i got to 1 but then a precision of 1 ends up not matching. does anyone have a good idea of how to get the values to match?  I notice when i use toPrecision it changes it to a string also. I am pretty new to JS so i am not sure what is out there to help here. thanks in advance!
  • HKosova's avatar
    HKosova
    10 years ago
    Hi Mathew,



    You're right, it happens because of floating point math specifics. The proper way to compare floating-point numbers is within some very small tolerance, like this:



    // Replace aqObject.CheckProperty with this:

    var diff = Math.abs(ultraGridCLT.wValue(1, "Scale") - i);

    if (diff < 0.000001)

      Log.Checkpoint("The values match.");

    else

      Log.Error("The values are different.");

3 Replies

Sort By
  • msaunders's avatar
    msaunders
    Contributor
    10 years ago

    If it helps this is the actual function i am using. 



    function iterateScale(ultraGridCLT)


    {


      for (i = 0.1; i <= 10; i += 0.1){


        var textCLT = ultraGridCLT.ultraGrid1_EmbeddableTextBox;


        //set the scale


        ultraGridCLT.ClickCellXY(1, "Scale", 66, 8);


        textCLT.SetText( i );


        windowPlaceHolder.Click(63, 305);


        ultraGridCLT.Click(268, 948);


        //check that the value is really right


        aqObject.CheckProperty(ultraGridCLT, "wValue(1, \"Scale\")", cmpEqual, i );


      }


    }

  • HKosova's avatar
    HKosova
    SmartBear Alumni (Retired)
    10 years ago
    Hi Mathew,



    You're right, it happens because of floating point math specifics. The proper way to compare floating-point numbers is within some very small tolerance, like this:



    // Replace aqObject.CheckProperty with this:

    var diff = Math.abs(ultraGridCLT.wValue(1, "Scale") - i);

    if (diff < 0.000001)

      Log.Checkpoint("The values match.");

    else

      Log.Error("The values are different.");

  • msaunders's avatar
    msaunders
    Contributor
    10 years ago
    Thank you, i figured it had to be something along those lines. I was not sure how to compare a value without aqutils checkpoint. That was really helpful seeing that i can just pass ultraGridCLT.wValue(1, "Scale") , i didn't realize i could do that.   It passes now and i got to see the actual error because of the new method. within the cell i was checking the value changed from "0.1" format to large floating point values, see attached. Thanks i learned a lot from this advice and i will use it towards my other checks.