Forum Discussion

JeanL's avatar
JeanL
Contributor
14 years ago

Counting with doubles (fractions)

Hey,



I ran into this strange bug with TC 8.60.665.70 when counting with double values:



17.08-(17.10-0.02) = -3,5527136788005E-15

17.07-(17.09-0.02) = 0



They should both give 0, right?



Is this because computers can't process fractions correctly or is this bug in VBSCript, TestComplete or in my computer or even in my head? Am I missing something here?

2 Replies

  • tristaanogre's avatar
    tristaanogre
    Esteemed Contributor
    14 years ago
    I think, generally, it is a problem with "doubles" as a data type.  You'll run into the same problem in SQL Server.  This is because a decimal such as 17.09 is not actually stored like that but as a set of bytes and such that, when being translated into human readable text, may actually display differently.
  • AlexeyK's avatar
    AlexeyK
    Icon for Alumni rankAlumni
    14 years ago

    Jean,


    This happens due to the format of floating point numbers used in VBScript.

    Specifics of floating point representation causes various interesting situations (you can read more in wikipedia, for example).


    Anyhow, VBScript is not the programming language for scientific calculations, is it? :-)