More correct assertion of testcase

[fwseb]

I use a groovy script to cancel (or abort) the test execution (if some precondition is satisfied):

// test step "SendSellOrderVerifyRequest" is a setup/precondition of the testcase.

def rspHeader = context.testCase.testSteps["SendSellOrderVerifyRequest"].testRequest.response.responseHeaders
if (! rspHeader["#status#"].contains("HTTP/1.1 200 OK")) {
    context.testRunner.cancel("TestCase cancelled, because ...... ")
}

This gives the test case "Cancelled" result in ReadyAPI, as well as in the test log:

2021-03-23T13:51:45.4138758Z 14:51:45,399 INFO  [SoapUIProTestCaseRunner] Finished running TestCase [xxxxxx], time taken: 1259ms, status: CANCELED

But, in Azure pipeline, this test case is counted as "Passed", probably because "Canceled" is not a supported outcome of test result in Azure:

> There are 15 possible outcomes for a test result: Aborted, Blocked, Error, Failed, Inconclusive, In progress, None, Not applicable, Not executed, Not impacted, Passed, Paused, Timeout, Unspecified, and Warning.

https://docs.microsoft.com/en-us/azure/devops/pipelines/test/test-glossary?view=azure-devops

My question is:

How many test result outcomes are ReadyAPI supporting, besides Passed, Failed and Canceled? is there any way to set the test assertion to "Aborted" or "Inconclusive" etc that Azure supports, to align with Azure? If not can you add the feature of having more assertion outcomes such as "Aborted"?

[nmrao]

---

@fwseb wrote:

This gives the test case "Cancelled" result in ReadyAPI, as well as in the test log:

2021-03-23T13:51:45.4138758Z 14:51:45,399 INFO  [SoapUIProTestCaseRunner] Finished running TestCase [xxxxxx], time taken: 1259ms, status: CANCELED

---

Does this work when the same project is called / execute from command-line using testrunner utility?

NOTE: No-voice in Azure pipeline.

By the way, what command do you use in Azure pipeline to execute the project?

[fwseb]

I haven't tried to run the test project by command line myself, but the command line that is used in pipeline is:

C:\WINDOWS\system32\cmd.exe /D /S /C ""C:\Program Files\Maven\apache-maven-3.5.2\bin\mvn.cmd" -f <my test project pom>.xml test --fail-at-end"

[nmrao]

Thank you for that.
Do you see the same behavior if you run the command locally to execute the project?