Forum Discussion

ameykhopade's avatar
ameykhopade
Contributor
8 years ago

test scripts fail in full test run but pass when executed individually with no changes made in tests

Hi,

 

I am testing an web application in IE.

 I have a framework written in VBscript. It a data driven framework and takes test scripts in Excel sheet as input and executes the test.

one excel sheet consists of one test script, and i have created 100 of such test scripts.

 

now when i execute all the test scripts in one run (have another excel sheet as the config excel which controls the number of test scripts to be executed) 

 

so when I run 100 test scripts out this 100 only 5-6 test scripts fails and when I re execute these test scripts Individually they are executed successfully with no error.

 

again when when I executed the all 100 test scripts now some different 4-5 tests fail and again this tests pass when they are executed individually 

 this continues for other runs as well I am not able to get a clean run where all the 100 test are passed without any error.

 

And the reason for test scripts failing is also not a major one its like sometimes it is not able to find a particular object or sometimes it is not able to perform a click action on a object which is present. or sometimes the page is not loaded completely and the test timeouts.

 

All the these issue are not reproduced when the same test are run individually on in some other run.

 

has anyone faced the same issue.

can you suggest me anything on this.?

any particular reason for this behavior.

 

Note: the full test run take around 4 hrs to execute and cant tell which test will fail ;(

 

thanks,

Amey

7 Replies

Sort By
  • tristaanogre's avatar
    tristaanogre
    Esteemed Contributor
    8 years ago

    What it sounds like is that, generally speaking, you're dealing with timing issues with your tests.  It takes JUST a little bit too long for an object to be available for interaction, or visible on screen, or to be refreshed after making a change... and the tests fail.

     

    Running them individually, you have less processes running, most likely, and the system responds better.  Just off the top of my head, I'm thinking you will need to increased your auto-wait timeout, either globally on the project or in specific "Wait" calls you are making.

     

    It may help us help you more if you can give us screenshots of the logs that demonstrate these particular problems.  We may be able to target a solution better.

    • ameykhopade's avatar
      ameykhopade
      Contributor
      8 years ago

      Hi

       

      Yes i too had the same doubt regarding the timing issue and for that I tried to add a exist check.

       

      While Not object.VisibleOnScreen'or object.Exists
 aqUtils.Delay 1000 ' 1 sec delay
  If counter > 60 then 
  Break()
 Else
 counter = counter + 1
  End If
WEnd

      But here the Break statement is not working and o the infinite loop as I need to time out the test if the object is not available after some time.

      Not sure What i am missing here.

       

      And what exactly do you need from the testlog should i share the error message or any other thing from the log

       

      thanks,

      amey

       

      • tristaanogre's avatar
        tristaanogre
        Esteemed Contributor
        8 years ago

        Instead of just calling the Exists or VisibleOnScreen property, simply use WaitChild or WaitAliasChild with a timeout delay set to 60 seconds.  Something more like this (sorry, I don't know VB syntax very well so I'm doing it in Java)

         

        if (!parentobject.WaitAliasChild('myObject', 60000).Exists) {
    Log.Error('Object did not appear within 60 seconds);
}

        As for the test log, it would be good to know 

         

        1) What the actual text is of the error message you're getting

        2) Information from the "Additional Info" panel for the error message

        3) Some sample of the code that is generating the error message.

         

        These will help to debug the problem.  As mentioned, it is most likely a timing issue.

        Something you can do, more globally, is in Tools | Current Project Properties | Playback, increase your auto-wait timeout.  The default is typically 10 seconds.  Bump that up to 20 or 30 seconds to see if this corrects the problem.