Forum Discussion

p_mariet's avatar
p_mariet
Occasional Contributor
11 years ago

Testcomplete subsctraction problem VBScript

Hello, I'm having a hard time with a simple substraction in Testcomplete (I know that sounds a bit stupid...). Here is an example: Dim one, two   one = (79.10 - 67...
  • karkadil's avatar
    karkadil
    11 years ago
    Working with float types is always a bottleneck in scripting language.



    I am not an expert in this and can't explain why exactly this example works like this, but I've found a solution for you




    Dim one, two


     


    one = Round(79.10 - 67.80, 2)


    two = Round(11.30, 2)


     

hlalumiere's avatar
hlalumiere
Regular Contributor
11 years ago
The reason is that FLOATs and DOUBLEs cannot properly express integer numbers. See http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems . They are meant to represent fractions, not integers,



An alternative would be to use the DECIMAL format, which internally is stored as an unpacked BCD, which solves the issues above. Of course it does not have the precision of a DOUBLE...