Forum Discussion

JohanH's avatar
JohanH
Occasional Contributor
10 years ago

Casting lager float numbers to int does not work when using python

I'm having issues casting a large float to int in python. It works fine when running python from the console but not in TestComplete. I need to do some bit operations, using a timestamp as base. ...
  • joseph_michaud's avatar
    joseph_michaud
    10 years ago

    I think you may be seeing a bug with how TestComplete is converting python variables.  If you convert any of the variables to a string and print out the string, the variable seems to have the correct value.  From your example:

     

     

    import datetime
import math

def test():
  dt = datetime.datetime(2015,10,30,12,29,5,987)
  ms = dt.timestamp() * 1000 * 1000
  Log.Message("aqConvert.FloatToStr(ms) = " + aqConvert.FloatToStr(ms))
  Log.Message("str(ms) = " + str(ms))
  
  intms = int(ms)
  Log.Message("aqConvert.IntToStr(intms) = " + aqConvert.IntToStr(intms))
  Log.Message("str(intms) = " + str(intms))

     

    produces this output

     

    aqConvert.FloatToStr(ms) = 1.44622254500099E+15
str(ms) = 1446222545000987.0
aqConvert.IntToStr(intms) = -1
str(intms) = 1446222545000987

    When debugging, the local variable also (incorrectly) shows the intms value as -1.

     

    I haven't pursued this much further, but I think your calculations will be OK and that you should use the Python string conversion to look at your values for the time being.

     

JohanH's avatar
JohanH
Occasional Contributor

After much pondering I came up with the following solution, but it is not pretty...

maxint = 0x100000000  # this works even if TestComplete says this is -1
# split the float into two 32-bit integers
# divide with maxint is equivalent to shifting an integer 32 bits right (>> 32)
t_bits_u = int(ms / maxint)
t_bits_l = int(ms - t_bits_u * maxint)
# apply mask and shift
t_bits_u = ((t_bits_u  & 0x7FFFFF) << 8) | ((t_bits_l & 0xFF000000) >> 24)
t_bits_l = (t_bits_l & 0x00FFFFFF) << 8

...

'Tx{0:08X}{1:08X}'.format(t_bits_u, t_bits_l)

Fortunately I'm hex encoding the result to a string to send to the tested app. This enables me to encode each 32 bit part and concatenate the result. If further calculations where needed I would be doomed....

 

joseph_michaud's avatar
joseph_michaud
Moderator
10 years ago

I think you may be seeing a bug with how TestComplete is converting python variables.  If you convert any of the variables to a string and print out the string, the variable seems to have the correct value.  From your example:

 

 

import datetime
import math

def test():
  dt = datetime.datetime(2015,10,30,12,29,5,987)
  ms = dt.timestamp() * 1000 * 1000
  Log.Message("aqConvert.FloatToStr(ms) = " + aqConvert.FloatToStr(ms))
  Log.Message("str(ms) = " + str(ms))
  
  intms = int(ms)
  Log.Message("aqConvert.IntToStr(intms) = " + aqConvert.IntToStr(intms))
  Log.Message("str(intms) = " + str(intms))

 

produces this output

 

aqConvert.FloatToStr(ms) = 1.44622254500099E+15
str(ms) = 1446222545000987.0
aqConvert.IntToStr(intms) = -1
str(intms) = 1446222545000987

When debugging, the local variable also (incorrectly) shows the intms value as -1.

 

I haven't pursued this much further, but I think your calculations will be OK and that you should use the Python string conversion to look at your values for the time being.

 

  • JohanH's avatar
    JohanH
    Occasional Contributor
    10 years ago

    Thanks for your effort.

     

    I do however believe that making calculations in the value did not work... at least not the bitwise operations but I'll give it another spin and see how it turns out...

  • JohanH's avatar
    JohanH
    Occasional Contributor
    10 years ago

    Now I've tried reverting to my original calculations and using the 'Watch' window, and '{0} [0x{0:016X}]'.format(value), instead of 'Locals' to view the intermediat results during debugging. It works like a charm!

     

    Summary: The calculations are correct but the 'Locals' debug window does a poor job of converting theese large integers to strings for python. If you are working with python and large integers please be advised that you cannot trust what you see uneder 'Locals', and use my trick of converting the variables to strings using e.g. format or some other python function.