server version: uploading test results to Zephyr Scale via the automation API

[fede1]

Hello!

I'm using Zepyhr Scale Server and I would like to upload to Zephyr the results of my automation testsuite made with pytest. I've tried this POST request: 

post(url="https://{my-jira-host}/rest/atm/1.0/automation/execution/{projectKey}", auth=({my_username}, {my_password}), files={"file":open("test_results.zip","rb")})

 but it doesn't work because the response is "errorMessages":["Invalid Custom Format JSON file"]}'.

I'm uploading a zip file containing one xml file generated with 

pytest --junitxml=output/junitxml_report.xml

as it's explained here https://support.smartbear.com/zephyr-scale-cloud/docs/test-automation/pytest-integration.html

I've tried to make the same request with an API client (Postman) and the error is "Invalid ZIP file", even if I fail the authentication with a wrong username or even if I upload the xml file only.

Maybe someone does the same thing and could help me? I'm a newbie 🙂 thanks!

[abioneperhobby]

Hi,

i'm facing the same issue with both the junit.xml file generated by pytest and the junit.xml generated by jest, and neither one works.

Someone has a working xml file?

[MisterB]

Just checking as I notice that the documentation link you posted is for the Cloud version but you say you're on the Server version.  The APIs might not be interchangeable between the versions.

Also, maybe some of these previous posts to the community might help: Search - SmartBear Community

[fede1]

thanks for the answer! yes you're right, I checked the cloud documentation because it's written how to make the integration with pytest, and in the server documentation it isn't reported. Is it impossible? but I used the server version API that I found here https://support.smartbear.com/zephyr-scale-server/api-docs/v1/

[MisterB]

Glad that's helped clear up part of the issue.  I've not used the APIs that much, and not for what you're trying to do but it seems to me that the API for server is generic and as long as you follow the syntax it should work.  Has it worked for you?

[abioneperhobby]

i'm using the server documentation, 

maybe the problem can be with the endpoint?

following documentation https://support.smartbear.com/zephyr-scale-server/api-docs/v1/ it should to be something like

http://your-jira-host:port/your-jira-context/rest/atm/1.0/

but taking a look at the network in chrome i found a call with this endpoint

https://jira.namirial.com/rest/tests/1.0/testcase/search?archived=false&fields=id,key,projectId,name... 

and the api works using /rest/tests/1.0/testcase/search but not using /rest/atm/1.0/ as described

but if i point to

https://jira.namirial.com/rest/tests/1.0/automation/execution/CUSTOMPRJ

i obtain

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<status>
<status-code>404</status-code>
<message>null for uri: https://jira.namirial.com/rest/tests/1.0/automation/execution/CUSTOMPRJ</message>
</status>

instead if i point to https://jira.namirial.com/rest/atm/1.0/automation/execution/CUSTOMPRJ 

the endpoint is recognized but the api respond that there are a invalid custom format json file

i tried with

• xml generated by pytest 
• xml generated by jest
• xml downloaded online

[MisterB]

As a test, you could use another rest/atm endpoint to make sure it's working for you.  This endpoint works for me:

https://jira.[YOUR INSTANCE].com/rest/atm/1.0/testrun/[YOUR TEST CYCLE]/testresults

If that's working for you, and the JSON file you're providing is in the correct format as mentioned in the documentation here (smartbeartm4j / tm4j-junit-integration — Bitbucket) then I'm sorry I can't help.  I've only used the APIs for GET functions and not very often.

[abioneperhobby]

i make other tests and i understand that the api doesn't want a junit.xml file but a json file formatted as zephyr wants, as for example

{
   "version": 1,
   "executions":[
      {
         "source":"CalculatorSumTest.sumTwoNumbersAndPass",
         "result":"Passed",
         "testCase": {
            "key": "QA-T1"
         }
      },
      {
        "source":"CalculatorSumTest.sumTwoNumbersAndFail",
        "result":"Failed",
         "testCase": {
            "key": "QA-T2"
         }
      },
      {
        "source":"CalculatorSumTest.notMappedToTestCaseAndPass",
        "result":"Passed"
      },
      {
        "source":"CalculatorSumTest.mappedToTestCaseNameAndPass",
        "result":"Passed",
         "testCase": {
            "name": "Mapped to Test Case Name and Pass"
         }
      }
   ]
}

as described here https://bitbucket.org/smartbeartm4j/tm4j-junit-integration/src/master/ 

but here https://support.smartbear.com/zephyr-scale-cloud/docs/test-automation/pytest-integration.html documentation talk about uploading a simple xml file, not a json file, but using this api https://support.smartbear.com/zephyr-scale-cloud/api-docs/#operation/createJUnitExecutions that doesn't exists for the server version.

Can Zephyr Server version be updated to accept the createJUnitExecutions api?

[MisterB]

Sorry but I can only suggest raising a request with the SmartBear support team at this link: Login - Jira Service Management (atlassian.net)

But @josh42 might also be able to help?

[josh42]

Hi guys,

I work with the server API quite extensively, but have not used the JUnit integration, yet. I can only refer you to the docs

https://support.smartbear.com/zephyr-scale-server/docs/test-automation/integrations/junit.html

https://support.smartbear.com/zephyr-scale-server/docs/test-automation/integrations/annotations-with...

One thing, I can clear up is the difference between the API endpoints using rest/atm/... and the ones using rest/tests/...

The atm ones are the public API endpoints of Zephyr Scale which are documented for public use.

The tests endpoints are the internal ones that ZS uses. So those are not meant for public use and thus are not documented publicly. They can be quite handy, if you are savvy enough to figure them out (because they return a lot more information - like internal ids and such - than the public endpoints) but you will always run the risk of running into undocumented changes in the API and not getting the same sort of support than you get for the official API (also, you can break stuff quite easily using these endpoints, so be careful).

As for the rest, I'm with @MisterB , raise a feature request for the server API and in the meantime contact support directly to have them help you figure out, why your JUnit results do not have the right format.

Cheers

Josh