Forum Discussion

JohanH's avatar
JohanH
Occasional Contributor
10 years ago

Casting lager float numbers to int does not work when using python

I'm having issues casting a large float to int in python.

It works fine when running python from the console but not in TestComplete.

I need to do some bit operations, using a timestamp as base. The timestamp is measured in microseconds since epoc (i.e. 1970).

 

Here is the code that fails:

import datetime

dt = datetime.datetime(2015,10,30,12)  # Create date and time
ms = dt.timestamp() * 1000 * 1000  # Get timestamp in microseconds
t_bits = (int(ms) & 0x7FFFFFFFFFFFFF) << 8  # Apply mask and shift

in the above example int(ms) will return -1... python operates with 'bignum' under the hood and as stated in the TestComplete documentation the number here is well within range.

"An integer value can accept zero, positive and negative numbers within the range ±1.7976931348623157x10^308"

 

I also discovered that math.floor returns -1 as well when I was trying to find a workaround.

 

Any thougts on how to solve this?

  • joseph_michaud's avatar
    joseph_michaud
    10 years ago

    I think you may be seeing a bug with how TestComplete is converting python variables.  If you convert any of the variables to a string and print out the string, the variable seems to have the correct value.  From your example:

     

     

    import datetime
import math

def test():
  dt = datetime.datetime(2015,10,30,12,29,5,987)
  ms = dt.timestamp() * 1000 * 1000
  Log.Message("aqConvert.FloatToStr(ms) = " + aqConvert.FloatToStr(ms))
  Log.Message("str(ms) = " + str(ms))
  
  intms = int(ms)
  Log.Message("aqConvert.IntToStr(intms) = " + aqConvert.IntToStr(intms))
  Log.Message("str(intms) = " + str(intms))

     

    produces this output

     

    aqConvert.FloatToStr(ms) = 1.44622254500099E+15
str(ms) = 1446222545000987.0
aqConvert.IntToStr(intms) = -1
str(intms) = 1446222545000987

    When debugging, the local variable also (incorrectly) shows the intms value as -1.

     

    I haven't pursued this much further, but I think your calculations will be OK and that you should use the Python string conversion to look at your values for the time being.

     

  • JohanH's avatar
    JohanH
    Occasional Contributor
    10 years ago

    After much pondering I came up with the following solution, but it is not pretty...

    maxint = 0x100000000  # this works even if TestComplete says this is -1
# split the float into two 32-bit integers
# divide with maxint is equivalent to shifting an integer 32 bits right (>> 32)
t_bits_u = int(ms / maxint)
t_bits_l = int(ms - t_bits_u * maxint)
# apply mask and shift
t_bits_u = ((t_bits_u  & 0x7FFFFF) << 8) | ((t_bits_l & 0xFF000000) >> 24)
t_bits_l = (t_bits_l & 0x00FFFFFF) << 8

    ...

    'Tx{0:08X}{1:08X}'    .format(t_bits_u, t_bits_l)

    Fortunately I'm hex encoding the result to a string to send to the tested app. This enables me to encode each 32 bit part and concatenate the result. If further calculations where needed I would be doomed....

     

    • joseph_michaud's avatar
      joseph_michaud
      Moderator
      10 years ago

      I think you may be seeing a bug with how TestComplete is converting python variables.  If you convert any of the variables to a string and print out the string, the variable seems to have the correct value.  From your example:

       

       

      import datetime
import math

def test():
  dt = datetime.datetime(2015,10,30,12,29,5,987)
  ms = dt.timestamp() * 1000 * 1000
  Log.Message("aqConvert.FloatToStr(ms) = " + aqConvert.FloatToStr(ms))
  Log.Message("str(ms) = " + str(ms))
  
  intms = int(ms)
  Log.Message("aqConvert.IntToStr(intms) = " + aqConvert.IntToStr(intms))
  Log.Message("str(intms) = " + str(intms))

       

      produces this output

       

      aqConvert.FloatToStr(ms) = 1.44622254500099E+15
str(ms) = 1446222545000987.0
aqConvert.IntToStr(intms) = -1
str(intms) = 1446222545000987

      When debugging, the local variable also (incorrectly) shows the intms value as -1.

       

      I haven't pursued this much further, but I think your calculations will be OK and that you should use the Python string conversion to look at your values for the time being.

       

      • JohanH's avatar
        JohanH
        Occasional Contributor
        10 years ago

        Thanks for your effort.

         

        I do however believe that making calculations in the value did not work... at least not the bitwise operations but I'll give it another spin and see how it turns out...