Forum Discussion

DanNad's avatar
DanNad
Contributor
10 months ago

No Log.ErrCount when test is being stopped manually

I've opened a support issue and but they couldnt help me on this so im hoping maybe someone here knows the solution to following problem.

At the end of each of my tests the general event OnStopTest is being called and writes some results and performance data into a sql database.
Writing into database should only happen when the test has passed without error. To check if the testrun is failed  in addition to OnLogError Event i check for Log.ErrCount inside OnStopTest function.

The only problem i cant solve is: When i manually stop the testrun following happens:
1. Test is stopped

2. OnStopTest event is being called -> Log.ErrCount is 0 (but should be > 0 || 1)

At this point -> The Test is failed but Log.ErrCount is 0 and my function writes wrong results into database
3. Error "The script execution was interrupted" is Logged after the OnStopTest Event finished (i assume that only at this point Log.ErrCount is being increased by 1)


I was thinking about the OnStopTest Event (Manual Testing Events) but i dont know if this is even usable also its marked as deprecated and it'll be removed in the future so i didnt try.

Is there a way to solve this?

Thanks in advance

13 Replies

    • DanNad's avatar
      DanNad
      Contributor

      Thank you for clarification. So the easy solution would be the child process to raise an error by default if the stop button is clicked and not the parent?

  • eykxas's avatar
    eykxas
    Frequent Contributor

    Hi, this is a potential workaround, but it's really ugly : you need boolean variable in your project "testPassed". Each test case initialize this variable to false. And at a last operation, each test set this variable to true.

     

    Then in the OnStopTest event, check both variable. If Log.ErrCount > 0 or testPassed = false : don't log perfs in SQL.


    I haven't tried it but in theory it should work.

    • DanNad's avatar
      DanNad
      Contributor

      eykxas wrote:

      Hi, this is a potential workaround, but it's really ugly : you need boolean variable in your project "testPassed". Each test case initialize this variable to false. And at a last operation, each test set this variable to true.

       

      Then in the OnStopTest event, check both variable. If Log.ErrCount > 0 or testPassed = false : don't log perfs in SQL.


      I haven't tried it but in theory it should work.


      eykxas Thank you for your idea. This way i could solve the problem but this is indeed not ideally.

       

      I created a feature request (https://community.smartbear.com/t5/TestComplete-Feature-Requests/Clicking-on-Stop-Button-should-raise-Log-ErrCount-by-1/idi-p/250413) as rraghvani suggested and hopefully it will get some attention from smartbear.

      In my opinion this behaviour of testcomplete should be seen as bug because a manually stopped test should also be markable as failed by checking Log.ErrCount within GeneralEvent_OnStopTest but at testcompletes current state its not.

  • rraghvani's avatar
    rraghvani
    Champion Level 3

    Log.ErrCount returns the number of error messages (that is, messages added via Log.Error) posted to the test log during the current run of the current test item.

     

    In this example, I'm calling Log.Error() method twice, and Log.ErrCount shows 2 

    If I stop the automation after the first Log.Error() during the 10 seconds, then Log.ErrCount shows 1

    Which is normal behaviour.

     

    • DanNad's avatar
      DanNad
      Contributor

      rraghvani wrote:

      Log.ErrCount returns the number of error messages (that is, messages added via Log.Error) posted to the test log during the current run of the current test item.

       

      In this example, I'm calling Log.Error() method twice, and Log.ErrCount shows 2 

      If I stop the automation after the first Log.Error() during the 10 seconds, then Log.ErrCount shows 1

      Which is normal behaviour.

       


      Yes i undestand that and thats the problem.

       


      Which is normal behaviour.

      Normal behaviour doesnt mean its good.

  • rraghvani's avatar
    rraghvani
    Champion Level 3

    Sorry, I may have miss understood. When you stop the automation, and no Log.Error() methods are called, what should Log.ErrCount be?

     

    OnStopTest is different to OnTestStop event.

    • DanNad's avatar
      DanNad
      Contributor

      Log.ErrCount should be 1

      Ideally the "The script execution was interrupted", which looks to me like a Log.Error should be logged at the time i click the stop button and not at the very end of the log. But probably thats not a thing smartbear is going to change in near future so i need to find a workaround for this.

  • rraghvani's avatar
    rraghvani
    Champion Level 3

    The way I see it, is that you have a parent process being TestComplete, which spools up a child process to run the automation. When you stop the automation, you are stopping the child process, and the message "The script execution was interrupted" is coming from the parent process. Therefore, Log.ErrCount will be zero, as the child process hasn't raised an error.

     

    I don't think there's a way of capturing "The script execution was interrupted"

     

     

    • DanNad's avatar
      DanNad
      Contributor

      Still not acceptable if the user has to find workaround and this makes it more difficult to work with testcomplete. Atleast give the user more control over such things by adding something that helps with such situations. Add more Global Events for example OnStopButtonClicked or OnStopTest or OnPauseTest etc.
      I know its maybe a problem a minor portion of users will have but still (and this has not changed): Usability or ease of use does play a big role into customer satisfaction.

  • rraghvani's avatar
    rraghvani
    Champion Level 3

    Is it necessary to stop the automation manually? What is the reason to stop the automation manually?

     

    I'm just trying to understand, as there could be alternative ways or resolving what you are trying to do.

    • DanNad's avatar
      DanNad
      Contributor

      Inside of GeneralEvents_OnStopTest we are collecting performance data (test-duration, cpu usage of AUT, memory usage of AUT etc.) and log the data into our database. But this data should only be written into the database if the test is passed.

      To determine if the test is passed or failed we defined a variable "ErrorStatus". This ErrorStatus variable will be set to 1 (for failed) if: GeneralEvents_OnLogError has been called or if Log.ErrCount is greater than 0

      So we need to have the test marked as failed when:
      1. Log.Error(...) occured --> GeneralEvents_OnLogError(...) was called
      2. JScript runtime exception occured -> Log.ErrCount is raised by 1 but does not call GeneralEvents_OnLogError(...)
      3. The test was stopped by user -> does not raise Log.ErrCount and test is marked as "Passed" inside of GeneralEvents_OnStopTest(...) -> Only problem left for me to solve

      We use the performance data to calculate a percentage deviation to previous test runs (test-duration, cpu usage, memory usage) and in order to ensure that the performance data comes only from test runs that are initailly "passed" we have to prevent that data from a failed test is being written into our database.

       

      And of course we do stop the tests every now and then especially if we have to debug and fix our test scripts.

  • rraghvani's avatar
    rraghvani
    Champion Level 3

    Here's a simple example where passed tests will write to the database, and failed test will not write to the database.

    However, you should not interrupt the automation while it's running, otherwise, since isError is equal to 0, it's considered to be passed, and will write to the database. You can not capture 3. The test was stopped by user