TestExecute and Azure Pipelines: Understanding and upgrading test result format

[zgibanez]

Hi there,

I have set up a Pipeline as part of a build. The test is executed successfully on the self-hosted agent, which has a TestExecute license and a ".trx" file is produced.

I have noticed that this file is very scarce in content. Regarding the individual tests, I can only gather the outcome and test duration.

See the example below, most of the information is made up of identifiers.

<?xml version="1.0" encoding="utf-8"?>
<TestRun id="f20edbe2-19c8-485f-85fd-c3fd3a76de11" name="XXXXX 2023-05-22 09:07:42" runUser="XXXXXX" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Times creation="2023-05-22T09:07:42.3114566+02:00" queuing="2023-05-22T09:07:42.3114566+02:00" start="2023-05-22T09:07:16.1427079+02:00" finish="2023-05-22T09:07:46.3304395+02:00" />
  <TestSettings name="default" id="bcc38cab-07e3-4179-9708-f8d4e64eb4f4">
    <Deployment runDeploymentRoot="buildserver_XXX_2023-05-22_09_07_42" />
  </TestSettings>
  <Results>
    <UnitTestResult executionId="da6d90e9-e78d-4dfc-bc35-6c0423463656" testId="5b32f886-8953-8e58-5cce-00235d5daff7" testName="Launch application" computerName="XXX" duration="00:00:18.1977103" startTime="2023-05-22T09:07:22.7142290+02:00" endTime="2023-05-22T09:07:40.9119393+02:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="da6d90e9-e78d-4dfc-bc35-6c0423463656" />
  </Results>
  <TestDefinitions>
    <UnitTest name="Launch application" storage="XXXX" id="5b32f886-8953-8e58-5cce-00235d5daff7">
      <Execution id="da6d90e9-e78d-4dfc-bc35-6c0423463656" />
      <TestMethod codeBase="XXX\Suite.pjs" adapterTypeName="executor://tctestexecutor/v2" className="AppSuite" name="XXX" />
    </UnitTest>
  </TestDefinitions>
  <TestEntries>
    <TestEntry testId="5b32f886-8953-8e58-5cce-00235d5daff7" executionId="da6d90e9-e78d-4dfc-bc35-6c0423463656" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
  </TestEntries>
  <TestLists>
    <TestList name="Results Not in a List" id="8c84fa94-04c1-424b-9868-57a2d4851a1d" />
    <TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
  </TestLists>
  <ResultSummary outcome="Completed">
    <Counters total="1" executed="1" passed="1" failed="0" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
  </ResultSummary>
</TestRun>

I have some questions regarding this file and in general what can I expect from testExecute:

1) First, where can I find a format guide for `.trx` files? 

2) Can I customize it to include custom metrics? I'm pondering if this a suitable format for aggregating a database of test results

3) Is it possible to obtain a more full-fledged result format as with TestComplete logs? So a html zip with screenshots, expected and actual results, etc

Thank you!

[rraghvani]

See https://fileinfo.com/extension/trx. The file just contains test results, but it is possible to parse the XML file and capture aggregate data. See Test Results and sub topics.

There's also options in TestComplete Command Line, to export the results which include screenshots too.

[zgibanez]

Hi rraghvani, thanks for answering.

The link you posted about the trx format does not really explain its parts or how is it generated, which is what I wanted. I know that I can parse it and aggregate it, my question is if I can expand it with custom information.

Keep in mind I am asking about TestExecute, which is the tool I am using with Azure Pipelines, not TestComplete.

If TestExecute has more limited logging, that's fine, but I would like to know what can I already do with it.

[Luukdb · Accepted solution]

Hi Zgibanez,

I had the same issue for logging as you do until I found the setting below.

1. When editing a pipline, go the the TC test adapter installer.
2. Search for the "Save logs for" option.
3. 

   Select the desired option

4. 

   Click on the i icon

5. 

   Link the TestComplete log to the test result

6. 

After you run the pipeline you will get the logs just as in TestComplete itself. To find the logs open the pipeline who is finished and select the "Tests" tab.

Your screen will look like this.

Small tip: select the aborted filter en select the passed filter. This way you will also see the logs of passed tests.

Select the testrun below, and in the attachments tab you will find the logs, and you will be able to download the logs.

And your logs will look like the one from TC itself

I am not sure if you are able to customize this outcome but this is way more detailed then the log created by Azure itself.

I hope this is what you are looking for!

[rraghvani]

You will have to refer to the pipeline setup when you specify Visual Studio Test Platform and VsTest. You should be able to find more information on Azure DevOps.

There's also TestExecute Command Line, which also allows to export logs.

I export XML results, which I then parse to generate the following HTML summary email,

at the bottom of the email, I have a link that takes me to detailed results

[mkharsha]

How you parsed XML results? If you can shared details will be useful.

[rraghvani]

Using PowerShell script, https://stackoverflow.com/questions/18032147/parsing-xml-using-powershell. There's plenty of examples on the internet.

[mkharsha]

Above link is to parse XML result to html?

[mkharsha]

Do you have any example to convert Testcomplete log to html?

[rraghvani]

XML is just data, which I convert to HTML to make it readable.

See Exporting Test Results