Forum Discussion

bmarcantel's avatar
bmarcantel
New Contributor
6 years ago

Consistency running test suites in Test Complete vs. Test Execute

Greetings,  I am curious if anyone has noticed differences in test consistency between Test Complete and Test Execute. Currently our organization's work flow consists of creating groups of keyword ...
  • tristaanogre's avatar
    tristaanogre
    6 years ago

    Not on this end....at least not in major ways.

    One thing to keep in mind has to do with execution speed.  I have noticed that automations run on TestComplete run a little slower... there's an overhead of the debugger, the IDE, etc.. TestExecute is lighterweight and so tests run faster.  We have noticed inconsistency ONLY with regards to timing issues... automation runs faster on TestExecute so we need to be more dligent and consistent with proper coding for these timings... use of "Wait", "WaitChild", etc.  That's the advice I give you.

tristaanogre's avatar
tristaanogre
Esteemed Contributor
6 years ago

Not on this end....at least not in major ways.

One thing to keep in mind has to do with execution speed.  I have noticed that automations run on TestComplete run a little slower... there's an overhead of the debugger, the IDE, etc.. TestExecute is lighterweight and so tests run faster.  We have noticed inconsistency ONLY with regards to timing issues... automation runs faster on TestExecute so we need to be more dligent and consistent with proper coding for these timings... use of "Wait", "WaitChild", etc.  That's the advice I give you.

  • bmarcantel's avatar
    bmarcantel
    New Contributor
    6 years ago

    This is consistent with what I had noticed watching the tests execute and was my main hypothesis. I'll update how and where in the scripts adding "waits: effects things.

     

    Thanks!

  • bmarcantel's avatar
    bmarcantel
    New Contributor
    6 years ago

    Two quick follow up questions:

     

    1. Why is adding delays neccessary for the automation to run properly? i.e. does TestComplete/Execute have the equivalent of a document on load or on ready listener it uses to check for page resources, DOM elements, etc. to complete loading?

     

    2. Assuming no changes to a site and a static group of well written fully passing tests, is the expectation of a recurring 100% success rate reasonable? 

    • tristaanogre's avatar
      tristaanogre
      Esteemed Contributor
      6 years ago

      1) Not delays necessarily in the way of hard coded "sleeps" or call to aqUtils.Delay... but "smart" wait times.  And yes, it does... Look up the documentation on the "Page" object and specifically look at the "Wait" method.    Why is this necessary?  Because TestComplete attempts to execute as quickly as possible.  There are some global values for waiting for when a component or such is added but they aren't always sufficient.  So, to allow the application to "think" until it's ready before interacting with the UI, you need these delays.  Consider a human being.  Before clicking on a button, that human being "waits" until the button is ready to be clicked.  You are, when writing test automation, acting like a robotic human so you need to program such "wait" calls into the code.  Common practice... regardless of tool.

       

      2) no, it's not unreasonable.  But it is not something that just happens... it takes good, smart, robust programming and proper practices to get to that point... bullet-proofing your code, adding in proper wait times, accounting for dynamically changing object identifciation factors, etc....