cancel
Showing results for 
Search instead for 
Did you mean: 

Test Result analysis with automation tool : "first detection" Vs "already known" defects

SOLVED
Highlighted
IAG
Occasional Contributor

Test Result analysis with automation tool : "first detection" Vs "already known" defects

Hello, 

After execution of test case we must analyze the test result and check the defect detected by tool like Test Complete, tosca tricentis, selenium or other. In the first iteration, we can easily show the failed test steps highlighted in “Red” and the pass Test Steps highlighted in “Green”. However, in the next iterations it’s very difficult to do easily the same analyzes and focus the Tester and Developer attention at “first detection” defects. This last category is highlighted in “Red” with the “already known” defect. Is there any tool, addon, or mean that allow to agreggate test results from one iteration to another and make distinction between "first detection" and "already known" defects ? The idea is to have something like this : "first detection” defects highlighted in “Red”, "already known" defects [because detected by automation tool at the previous iteration] highlighted in orange color for example.

Thank you in advance for your help.

Ismail Agr

2 ACCEPTED SOLUTIONS

Accepted Solutions
Community Hero

Re: Test Result analysis with automation tool : "first detection" Vs "already known&q

Not within the TestComplete tool itself, no.  The tool is what it is: a development and execution tool for automated functional tests.  The output from the tool is aimed at simply reporting the results of a test run as executed.

 

For test result analysis, that's what tools like QA Complete are for.  Investigate in that direction.


Robert Martin
[Community Expert Group]
Please consider giving a Kudo if I write good stuff
----

Why automate?  I do automated testing because there's only so much a human being can do and remain healthy.  Sleep is a requirement.  So, while people sleep, automation that I create does what I've described above in order to make sure that nothing gets past the final defense of the testing group.
I love good food, good books, good friends, and good fun.

Mysterious Gremlin Master
Vegas Thrill Rider
Extensions available

View solution in original post

Community Hero

Re: Test Result analysis with automation tool : "first detection" Vs "already known&q

Hi Ismail,

 

One of the possible ways (exact implementation depends on the used tools/components):

-- Prerequisite: a) issue tracking system (ITS) is used to register problems; and b) it is possible to query ITS via some means (DCOM, http, tcp, etc.);

-- Within your common reusable test code, implement assert method that accepts condition (true/false) and issue identifier as parameters;

-- Within this assert method, you should check the condition. If the condition is true, then everything is fine;

-- If the condition fails, then check issue identifier. If it is empty (or zero, or null, ...) then this means that this is the 'first detection' that must be reported;

-- After you analyse test log and decide that the reported problem is a problem indeed, you must report it to the ITS and provide its identifier in test code as a parameter to the relevant call to assert;

-- On subsequent test runs, when condition fails again, you need to query the ITS for the issue status. If the status is not the final one (e.g. 'fixed/deployed'), then this means that the problem is known but not fixed yet. So you may report it as an 'already known' one;

-- If the condition fails but the status of the issue is the final one, this means that this is 'regression' detection.

 

Regards,
Alex
[Community Expert Group]
____
[Community Expert Group] members are not employed by SmartBear Software but
are just volunteers who have some experience with the tools by SmartBear Software
and a desire to help others. Postings made by [Community Expert Group] members
may differ from the official policies of SmartBear Software and should be treated
as the own private opinion of their authors and under no circumstances as an
official answer from SmartBear Software.
[Community Expert Group] signature is used with permission by SmartBear Software.
http://smartbear.com/forums/f83/t86934/community-experts/
================================

View solution in original post

6 REPLIES 6
Community Hero

Re: Test Result analysis with automation tool : "first detection" Vs "already known&q

Not within the TestComplete tool itself, no.  The tool is what it is: a development and execution tool for automated functional tests.  The output from the tool is aimed at simply reporting the results of a test run as executed.

 

For test result analysis, that's what tools like QA Complete are for.  Investigate in that direction.


Robert Martin
[Community Expert Group]
Please consider giving a Kudo if I write good stuff
----

Why automate?  I do automated testing because there's only so much a human being can do and remain healthy.  Sleep is a requirement.  So, while people sleep, automation that I create does what I've described above in order to make sure that nothing gets past the final defense of the testing group.
I love good food, good books, good friends, and good fun.

Mysterious Gremlin Master
Vegas Thrill Rider
Extensions available

View solution in original post

Community Hero

Re: Test Result analysis with automation tool : "first detection" Vs "already known&q

Hi Ismail,

 

One of the possible ways (exact implementation depends on the used tools/components):

-- Prerequisite: a) issue tracking system (ITS) is used to register problems; and b) it is possible to query ITS via some means (DCOM, http, tcp, etc.);

-- Within your common reusable test code, implement assert method that accepts condition (true/false) and issue identifier as parameters;

-- Within this assert method, you should check the condition. If the condition is true, then everything is fine;

-- If the condition fails, then check issue identifier. If it is empty (or zero, or null, ...) then this means that this is the 'first detection' that must be reported;

-- After you analyse test log and decide that the reported problem is a problem indeed, you must report it to the ITS and provide its identifier in test code as a parameter to the relevant call to assert;

-- On subsequent test runs, when condition fails again, you need to query the ITS for the issue status. If the status is not the final one (e.g. 'fixed/deployed'), then this means that the problem is known but not fixed yet. So you may report it as an 'already known' one;

-- If the condition fails but the status of the issue is the final one, this means that this is 'regression' detection.

 

Regards,
Alex
[Community Expert Group]
____
[Community Expert Group] members are not employed by SmartBear Software but
are just volunteers who have some experience with the tools by SmartBear Software
and a desire to help others. Postings made by [Community Expert Group] members
may differ from the official policies of SmartBear Software and should be treated
as the own private opinion of their authors and under no circumstances as an
official answer from SmartBear Software.
[Community Expert Group] signature is used with permission by SmartBear Software.
http://smartbear.com/forums/f83/t86934/community-experts/
================================

View solution in original post

IAG
Occasional Contributor

Re: Test Result analysis with automation tool : "first detection" Vs "already known&a

Hi, thank you for this response : I poc the tosca tool also does not allow to have an aggregation of the results of the different executions. I want to go through what is done before developing a solution to achieve aggregation. I was hoping to find an application or open source or even commercial tool that allows to have a synthesis by 3 colors: red (first detection), already knwon (detected previouseley) and green for passed.

Best regards

IAG
Occasional Contributor

Re: Test Result analysis with automation tool : "first detection" Vs "already known&a

Hi Alex,

the algorithm is nice :-). Thank you for this very useful information that enlighten me and I think this is the approach I will use to avoid paying the cost of a commercial tool. the algorithm is nice.

Best regards

Community Hero

Re: Test Result analysis with automation tool : "first detection" Vs "already known&a

Hi,

 

> to have a synthesis by 3 colors

TestComplete has three basic message types: Error, Warning and Message. (Also Checkpoint to get a nice green circle). Every type has Attributes parameter that you can use to colorize the text/background as you like. Does this help?

 

> to achieve aggregation

As said by Robert, TestComplete (and TestExecute) is just an execution tool. Look at QAComplete (https://qacomplete.com/) that is a management/aggregation/reporting tool complimentary with TestComplete.

 

Regards,
Alex
[Community Expert Group]
____
[Community Expert Group] members are not employed by SmartBear Software but
are just volunteers who have some experience with the tools by SmartBear Software
and a desire to help others. Postings made by [Community Expert Group] members
may differ from the official policies of SmartBear Software and should be treated
as the own private opinion of their authors and under no circumstances as an
official answer from SmartBear Software.
[Community Expert Group] signature is used with permission by SmartBear Software.
http://smartbear.com/forums/f83/t86934/community-experts/
================================
IAG
Occasional Contributor

Test Result analysis with automation tool : "first detection" Vs "already known&a

I'm examining the solution proposed by Robert and I'm looking so how to uses the approach you proposed by using firstly excel sheet. The Goal to deal with several test result analyzis and compare the result between several release. Thank you Alex and Robert for your implication and reactivity. 

Kind regards