TestComplete integration with TestRail

[chandreshp]

Hi, 



I am using TestRail to manage my Test suites and I am using TestComplete to perform automation testing of my application.





TestComplete allows to set log defects to JIRA. But does it provide any kind of API or Plugin which can help me to integrate TestComplete execution result to TestRail ? I mean is there any way in which I can update my test case result directly from TestComplete ?



Thanks,

Chandresh Parmar

[leandropoblet]

Hi Colin,

Thanks again for taking time to reply.
There's no paths involved here. If you look the function I'm calling:

function testUpdateTestRail() {
testrail.retestCase(13, 3486);
}

has only two parameters on it. Test run and Test case number both hardcoded integers just for testing the connection between TestComplete and TestRail.

If you follow the scripts posted by schef you'll see there's no file paths either.

On the other hand... how do I get to see the Json being sent.... no idea and it'd be great to know even for future debugging.

Cheers,
Leandro

[Colin_McCrae]

Sorry. My mistake in my previous post.

You are quite right. It's the JSON parser that's throwing the error.

I'm afraid I don't know why. The TestRail method must be passing something into it, which it doesn't like. (Or the DLL is bugged I guess?) Not sure how you debug it to allow you to actually inspect the JSON. Don't know what options are available to you here. I doubt you can step into external DLL code via the TC debugger?

Having never used any of this stuff, and not being an expert in JSON parsing, I'll need to leave this one for someone who knows the area better.

Looking at it, this is the TestRail method it ends up in:

http://docs.gurock.com/testrail-api2/reference-results#add_result_for_case

You only pass it a Test ID and a Run ID. So it must be interpreting something in the TC run to generate it's status to hand off to the API. As that's the point of the call. To update the test result/status.

The request should be generated to look the same as:

http://docs.gurock.com/testrail-api2/reference-results#add_result

... which can accept various additional fields. But the script looks like it only adds a status. (And any additional stuff you put in the call .... which is nothing in this case)

** BRAINWAVE! **

Hang on. This may be the same as TFS. In TFS-Land (I wrote my own extensions to do much the same as you're doing with TestRail) a Test Case moves from run to run. The Case ID stays the same, but the Test ID changes as you move it on from run to run.

Your method has to take in a Run ID and Case ID.

You mention passing it a Test ID. I wonder if you need to use one of the other methods (ie. runIdAndCaseIdFromTestId ) to get the Case ID and then use the ID it passed back to update the actual result?

Maybe give that a try?

[leandropoblet]

Hi Colin,

From the original usage post I'm trying the simplest one "retestCase" which has no comments, no version, only test run and test case numbers as explained in the post.

//USEUNIT TestRail

function testUpdateTestRail() {
    testrail.retestCase(4683, 866867);
    testrail.passCase(4683, 866868, {version: 'Build 10.0.0.1234'});
    testrail.blockedCase(4683, 866869, {comment: 'Blocked due to JIRA-4321'});
    testrail.failCase(4683, 866870, {comment: 'Failed. JIRA-5678 logged.', version: 'Build 10.0.0.1234'});
}

[joseph_michaud]

Everything goes through testrail.addResultForCase() to do a POST.  Is sendPost() having a problem trying to parse dataObj created in the call?  For the purpose of experimentation, what would happen if you hacked it to post something simple (like a string)?

testrail.addResultForCase = function(runId, caseId, additionalFields) {
    // dataObj = testrail.dataDictonary(additionalFields);
    testrail.sendPost("add_result_for_case/" + runId + "/" + caseId, "simplestring");
};

Do you still get the parse exception?

[schef]

Hello,

Sorry I'm a bit late to the party here. I no longer use testcomplete and testrail, but I'll try to help where I can. 

@leandropoblet, you posted an image that had an exception message that pointed to the testrail script file, line 86, column 5. Can you please post the statement from that line? I'm guessing it's the 

testrail.apiClient().SendPost(apiArgsAsString, dataDictionaryObj);

line.

Next, I would put a debug breakpoint on that statement, and then run the function that you've been testing with. Once TC stops at that line, use the Locals panel and/or the Evaluate dialog to get the apiArgsAsString value and dataDictionaryObj (may have to change the latter into a string or json object). Paste those values and we can go from there. My guess is that apiArgsAsString is not a valid string.

Another place to check would be the client root endpoint is correct

var client = dotNET.Gurock_TestRail.APIClient.zctor("https://testrail.blahblah1234.com/testrail/");

[leandropoblet]

Hi @schef, @joseph_michaud and @Colin_McCrae,

Found the source of the problem thanks to schef's last reply and it has nothing to do with schef's scripts.

TestRail address was incorrect although that wasn't the only problem. Also TestRail has an awkward way to reference test cases in a test run. Using the method add_result_for_case passing runId and caseId it's a bit tricky as the caseId in this case is the test case number (C) not the Test (T) from the test run.

Yes as complicated as it sounds that's the source of the problem. So basically TestRail references a test case in a run using RunId + CaseId instead of RunId + TestId (which is generated for each test run).

Thank you all for the good effort and support as it would have been impossible to get it working without you. Much appreciated.

Cheers,
Leo

[leandropoblet]

Just FYI I'm folowing up on another threat regarding the get_case method as it returns Json but can't be accessed as such

[Colin_McCrae]

"So basically TestRail references a test case in a run using RunId + CaseId instead of RunId + TestId (which is generated for each test run)."

That's what I was alluding to in my "**BRAINWAVE**" post further up. TFS is much the same. A test case always carries it's own ID, which is permanent, but when part of a test run, it carries a Run ID for that run only. Same idea with TestRail. And it makes sense when you stop and think about it. But does make referencing things a little more awkward.

Glad you got it sorted out. 🙂

[schef]

Nice brainwave @Colin_McCrae that is indeed how testrail works. If you know the unique case id, and run id, you can get the test id, which is the specific instance of the test case in the test run. It's a bit confusing, but allows you to put unique case ids in your test data or scripts in testcomplete. You can even create new test runs via the testrail api.

@leandropoblet let us know about that return json if you are having new issues.

Glad this worked out, thanks everyone. Happy testing.