server version: uploading test results to Zephyr Scale via the automation API

Test Zephyr
, , , , , ,
1

Originally posted by fede1 on July 15, 2021

Hello!

I’m using Zepyhr Scale Server and I would like to upload to Zephyr the results of my automation testsuite made with pytest. I’ve tried this POST request:

post(url="https://{my-jira-host}/rest/atm/1.0/automation/execution/{projectKey}", auth=({my_username}, {my_password}), files={"file":open("test_results.zip","rb")})

but it doesn’t work because the response is “errorMessages”:[“Invalid Custom Format JSON file”]}'.

I’m uploading a zip file containing one xml file generated with

pytest --junitxml=output/junitxml_report.xml

as it’s explained here https://support.smartbear.com/zephyr-scale-cloud/docs/test-automation/pytest-integration.html

I’ve tried to make the same request with an API client (Postman) and the error is “Invalid ZIP file”, even if I fail the authentication with a wrong username or even if I upload the xml file only.

Maybe someone does the same thing and could help me? I’m a newbie :slightly_smiling_face: thanks!

2

Originally posted by abioneperhobby on August 23, 2021

Hi,

i’m facing the same issue with both the junit.xml file generated by pytest and the junit.xml generated by jest, and neither one works.

Someone has a working xml file?

3

Originally posted by MisterB on August 24, 2021

Just checking as I notice that the documentation link you posted is for the Cloud version but you say you’re on the Server version. The APIs might not be interchangeable between the versions.

Also, maybe some of these previous posts to the community might help: Search - SmartBear Community

4

Originally posted by fede1 on August 24, 2021

thanks for the answer! yes you’re right, I checked the cloud documentation because it’s written how to make the integration with pytest, and in the server documentation it isn’t reported. Is it impossible? but I used the server version API that I found here https://support.smartbear.com/zephyr-scale-server/api-docs/v1/

5

Originally posted by MisterB on August 24, 2021

Glad that’s helped clear up part of the issue. I’ve not used the APIs that much, and not for what you’re trying to do but it seems to me that the API for server is generic and as long as you follow the syntax it should work. Has it worked for you?

6

Originally posted by abioneperhobby on August 24, 2021

i’m using the server documentation,

maybe the problem can be with the endpoint?

following documentation https://support.smartbear.com/zephyr-scale-server/api-docs/v1/ it should to be something like

http://your-jira-host:port/your-jira-context/rest/atm/1.0/

but taking a look at the network in chrome i found a call with this endpoint

https://jira.namirial.com/rest/tests/1.0/testcase/search?archived=false&fields=id,key,projectId,name,averageTime,estimatedTime,labels,folderId,componentId,statusId,priorityId,lastTestResultStatus(name,i18nKey,color),majorVersion,createdOn,createdBy,updatedOn,updatedBy,customFieldValues,owner&maxResults=40&query=testCase.projectId+IN+(12804)&startAt=0

and the api works using /rest/tests/1.0/testcase/search but not using /rest/atm/1.0/ as described

abioneperhobby_2-1629793875567.png
abioneperhobby_2-1629793875567.png400×104 7.54 KB

but if i point to

https://jira.namirial.com/rest/tests/1.0/automation/execution/CUSTOMPRJ

i obtain

<?xml version="1.0" encoding="UTF-8" standalone="yes"?> 404 null for uri:

instead if i point to https://jira.namirial.com/rest/atm/1.0/automation/execution/CUSTOMPRJ

the endpoint is recognized but the api respond that there are a invalid custom format json file

abioneperhobby_3-1629794873243.png
abioneperhobby_3-1629794873243.png399×283 20.3 KB

i tried with

  • xml generated by pytest
  • xml generated by jest
  • xml downloaded online
7

Originally posted by MisterB on August 24, 2021

As a test, you could use another rest/atm endpoint to make sure it’s working for you. This endpoint works for me:

[https://jira.YOUR INSTANCE].com/rest/atm/1.0/testrun/[YOUR TEST CYCLE]/testresults

If that’s working for you, and the JSON file you’re providing is in the correct format as mentioned in the documentation here (smartbeartm4j / tm4j-junit-integration — Bitbucket) then I’m sorry I can’t help. I’ve only used the APIs for GET functions and not very often.

8

Originally posted by abioneperhobby on August 24, 2021

i make other tests and i understand that the api doesn’t want a junit.xml file but a json file formatted as zephyr wants, as for example

{
   "version": 1,
   "executions":[
      {
         "source":"CalculatorSumTest.sumTwoNumbersAndPass",
         "result":"Passed",
         "testCase": {
            "key": "QA-T1"
         }
      },
      {
        "source":"CalculatorSumTest.sumTwoNumbersAndFail",
        "result":"Failed",
         "testCase": {
            "key": "QA-T2"
         }
      },
      {
        "source":"CalculatorSumTest.notMappedToTestCaseAndPass",
        "result":"Passed"
      },
      {
        "source":"CalculatorSumTest.mappedToTestCaseNameAndPass",
        "result":"Passed",
         "testCase": {
            "name": "Mapped to Test Case Name and Pass"
         }
      }
   ]
}

as described here https://bitbucket.org/smartbeartm4j/tm4j-junit-integration/src/master/

but here https://support.smartbear.com/zephyr-scale-cloud/docs/test-automation/pytest-integration.html documentation talk about uploading a simple xml file, not a json file, but using this api https://support.smartbear.com/zephyr-scale-cloud/api-docs/#operation/createJUnitExecutions that doesn’t exists for the server version.

Can Zephyr Server version be updated to accept the createJUnitExecutions api?

9

Originally posted by MisterB on August 24, 2021

Sorry but I can only suggest raising a request with the SmartBear support team at this link: Login - Jira Service Management (atlassian.net)

But josh42 might also be able to help?

10

Originally posted by josh42 on August 29, 2021

Hi guys,

I work with the server API quite extensively, but have not used the JUnit integration, yet. I can only refer you to the docs

https://support.smartbear.com/zephyr-scale-server/docs/test-automation/integrations/junit.html

https://support.smartbear.com/zephyr-scale-server/docs/test-automation/integrations/annotations-with-other-ci-tools-running-locally.html#_scripts_for_local_testing_other_ci_tools

One thing, I can clear up is the difference between the API endpoints using rest/atm/… and the ones using rest/tests/…

The atm ones are the public API endpoints of Zephyr Scale which are documented for public use.

The tests endpoints are the internal ones that ZS uses. So those are not meant for public use and thus are not documented publicly. They can be quite handy, if you are savvy enough to figure them out (because they return a lot more information - like internal ids and such - than the public endpoints) but you will always run the risk of running into undocumented changes in the API and not getting the same sort of support than you get for the official API (also, you can break stuff quite easily using these endpoints, so be careful).

As for the rest, I’m with MisterB , raise a feature request for the server API and in the meantime contact support directly to have them help you figure out, why your JUnit results do not have the right format.

Cheers

Josh

11

Originally posted by abioneperhobby on August 29, 2021

Support team answered me that there is no junit impor funcionality in server version, but only in cloud version.

i implemented by myself parsing the xml and using server api to create-update test results.