Forum Discussion

PramodDesai's avatar
PramodDesai
New Member
11 days ago

Generate a segregated report identifying error as script failure or an application issue

Hi Team,

Problem Statement : How can you identify whether an error is a script failure or an application issue through logs, and generate a segregated report at the end of the test execution?

========================================================================
Here, I'm trying to understand if anyone has tried to create a framework (or has already developed and is using one) in such a way that, once the test execution completes, we will have logs attached to the test execution report. I want to go even deeper and segregate failures (if any) based on script and application issues.

I understand that until we analyze, the failures cannot be easily segregated. But there could be some errors that can be clearly identified as script errors rather than application issues. Does TestComplete provide anything to identify such errors? If yes, what are those?

What am I aiming at? -- To reduce the analysis time (so the report can be shared earlier) as not all the errors from logs need to be given the same priority.

Thank you in advance for your thoughts and inputs.

Test Results

2 Replies

  • JDR2500's avatar
    JDR2500
    Frequent Contributor
    11 days ago

    We have addressed this to a degree by having the majority of our 'checkpoints' for application failures generate warnings rather than errors.  In the results summaries when we see warnings we know those are application failures.  Conversely, errors generally reflect script issues.
    As an example, we use property comparisons with the MessageType set to lmWarning:
    Call aqObject.CompareProperty(prop, cmpEqual, value, true, lmWarning)

    Additionally, we have our own comparison routines for clipboard comparisons, CSV files, text files, etc. that all return warnings rather than errors for failures.

    Not sure if that approach will work for you, but it serves us reasonably well.

     

  • rraghvani's avatar
    rraghvani
    Icon for Champion Level 3 rankChampion Level 3
    11 days ago

    Script Failure - errors caused by issues in the test script itself e.g. object not found due to incorrect alias, syntax error, logic error, etc.

    Application Issue - errors due to the application under test misbehaving e.g. crash, incorrect UI behaviour, unexpected error dialog, etc.

    You could wrap test steps within try-catch blocks and categorise the errors accordingly i.e. using pmHigher in Log.Error() method. You can also iterate over the log items and segregate based on ErrorType.

    Refer to https://support.smartbear.com/testcomplete/docs/reference/project-objects/test-log/log/index.html and https://support.smartbear.com/testcomplete/docs/reference/project-objects/project/index.html including sub topics.

    This is something that I have not implemented, as I have kept the automation as simple as possible, so that other QA's can contribute to scripting without knowing much about programming, and diagnose issues too.