Forum Discussion

IAG's avatar
IAG
Occasional Contributor
6 years ago

Test Result analysis with automation tool : "first detection" Vs "already known" defects

Hello, 

After execution of test case we must analyze the test result and check the defect detected by tool like Test Complete, tosca tricentis, selenium or other. In the first iteration, we can easily show the failed test steps highlighted in “Red” and the pass Test Steps highlighted in “Green”. However, in the next iterations it’s very difficult to do easily the same analyzes and focus the Tester and Developer attention at “first detection” defects. This last category is highlighted in “Red” with the “already known” defect. Is there any tool, addon, or mean that allow to agreggate test results from one iteration to another and make distinction between "first detection" and "already known" defects ? The idea is to have something like this : "first detection” defects highlighted in “Red”, "already known" defects [because detected by automation tool at the previous iteration] highlighted in orange color for example.

Thank you in advance for your help.

Ismail Agr

  • Not within the TestComplete tool itself, no.  The tool is what it is: a development and execution tool for automated functional tests.  The output from the tool is aimed at simply reporting the results of a test run as executed.

     

    For test result analysis, that's what tools like QA Complete are for.  Investigate in that direction.

  • Hi Ismail,

     

    One of the possible ways (exact implementation depends on the used tools/components):

    -- Prerequisite: a) issue tracking system (ITS) is used to register problems; and b) it is possible to query ITS via some means (DCOM, http, tcp, etc.);

    -- Within your common reusable test code, implement assert method that accepts condition (true/false) and issue identifier as parameters;

    -- Within this assert method, you should check the condition. If the condition is true, then everything is fine;

    -- If the condition fails, then check issue identifier. If it is empty (or zero, or null, ...) then this means that this is the 'first detection' that must be reported;

    -- After you analyse test log and decide that the reported problem is a problem indeed, you must report it to the ITS and provide its identifier in test code as a parameter to the relevant call to assert;

    -- On subsequent test runs, when condition fails again, you need to query the ITS for the issue status. If the status is not the final one (e.g. 'fixed/deployed'), then this means that the problem is known but not fixed yet. So you may report it as an 'already known' one;

    -- If the condition fails but the status of the issue is the final one, this means that this is 'regression' detection.

     

6 Replies

  • AlexKaras's avatar
    AlexKaras
    Champion Level 3

    Hi Ismail,

     

    One of the possible ways (exact implementation depends on the used tools/components):

    -- Prerequisite: a) issue tracking system (ITS) is used to register problems; and b) it is possible to query ITS via some means (DCOM, http, tcp, etc.);

    -- Within your common reusable test code, implement assert method that accepts condition (true/false) and issue identifier as parameters;

    -- Within this assert method, you should check the condition. If the condition is true, then everything is fine;

    -- If the condition fails, then check issue identifier. If it is empty (or zero, or null, ...) then this means that this is the 'first detection' that must be reported;

    -- After you analyse test log and decide that the reported problem is a problem indeed, you must report it to the ITS and provide its identifier in test code as a parameter to the relevant call to assert;

    -- On subsequent test runs, when condition fails again, you need to query the ITS for the issue status. If the status is not the final one (e.g. 'fixed/deployed'), then this means that the problem is known but not fixed yet. So you may report it as an 'already known' one;

    -- If the condition fails but the status of the issue is the final one, this means that this is 'regression' detection.

     

    • IAG's avatar
      IAG
      Occasional Contributor

      Hi Alex,

      the algorithm is nice :-). Thank you for this very useful information that enlighten me and I think this is the approach I will use to avoid paying the cost of a commercial tool. the algorithm is nice.

      Best regards

  • tristaanogre's avatar
    tristaanogre
    Esteemed Contributor

    Not within the TestComplete tool itself, no.  The tool is what it is: a development and execution tool for automated functional tests.  The output from the tool is aimed at simply reporting the results of a test run as executed.

     

    For test result analysis, that's what tools like QA Complete are for.  Investigate in that direction.

    • IAG's avatar
      IAG
      Occasional Contributor

      Hi, thank you for this response : I poc the tosca tool also does not allow to have an aggregation of the results of the different executions. I want to go through what is done before developing a solution to achieve aggregation. I was hoping to find an application or open source or even commercial tool that allows to have a synthesis by 3 colors: red (first detection), already knwon (detected previouseley) and green for passed.

      Best regards

      • AlexKaras's avatar
        AlexKaras
        Champion Level 3

        Hi,

         

        > to have a synthesis by 3 colors

        TestComplete has three basic message types: Error, Warning and Message. (Also Checkpoint to get a nice green circle). Every type has Attributes parameter that you can use to colorize the text/background as you like. Does this help?

         

        > to achieve aggregation

        As said by Robert, TestComplete (and TestExecute) is just an execution tool. Look at QAComplete (https://qacomplete.com/) that is a management/aggregation/reporting tool complimentary with TestComplete.