ReadyAPI Test for Azure DevOps manual test association

[sokol]

Hi

I have executed test in Azure Devops pipeline and Ready API Test for Azure Devops extension.

I have tried to connect my ReadyAPI test executed in pipeline in Azure Devops (using ReadyAPI) to my manual test cases to make more coverage result and see much more information such test case steps, description etc. 

Have you guys tried something like that? I have read a lot of information and can find only answer that it is not supported (e.g. Selenium test you can easily associate in VisualStudio to manual test cases prepared in AzureDevops, but I can't find info how to do it with ReadyAPI tests). 

I'm sure that someone maybe wrote some addon or maybe it is only a matter of configuration.

Please help, any good ideas or info from your experience are more than welcome.

Thanks in advance

Piotr

[khanchett]

If I am understanding your question correctly you are wanting to tie in your test plans to execute ReadyAPI test cases?  If this is the case I have figured out a way to do it with ReadyAPI and any other tests you can execute from a command line that isn't "Microsoft".

The short of it is that you need to use Visual Studio to achieve it with Microsoft's VS Test task.  The good news is I believe it would even work with the Community version.  It's very bare bones of what you would need to do in Visual Studio.  I have it currently working with ReadyAPI and Java Selenium via Maven calls.

I don't think Microsoft supports any of it except if you are using their products exactly the way they say to.  I've not found much help elsewhere.  My results are reported back for each test case and traced and seems to be working pretty smoothly.  

If this is what you are looking for feel free to reach out to me directly.  I'm not sure there is a private way to do that here.  I would be more than happy to share my code and explanation.

[sonya_m]

Thank you for helping @khanchett ! 

The SmartBear Community supports private messaging, however, if the info you are willing to share is OK share publicly, please do - other people will be able to use your approach in the future!

Hi @sokol, is this something you are looking to implement? Looks like khanchett  can help!

[khanchett · Accepted solution]

It's a rather involved process.  The gist of it is this:

* Install and setup Visual Studio for MSTest

* Write a MSTest test case that calls a command line to execute the ReadyAPI command line call.

  NOTE:  I've noticed anything printed to the console during the test run is shown as an attachment in the ADO test case results.  This can be helpful to relay info back to ADO.

* Get this working through VS, then move on to Azure DevOps.

* In Azure DevOps, Test Plans, create a test case and save. 

* Grab the test ID and back to VS, open Test Exporer, right click on the test case to tie to the ADO test case and associate the test id and save.

* In Azure DevOps create a separate CI-CD pipeline to run just test plan test cases.

* In your new Test Plan, select the test case created and click "Run with Options" in the drop down.

* In the next pop up you will select the middle option to run automated tests, connect it to the build # and the pipeline and Stage.

There are a lot of little things in-between to tweak.  This is the general format that I have working for a Java Selenium automation framework and for a ReadyAPI automation framework.  

Hope this helps.  To be clear, Microsoft doesn't clearly explain this in any way.  This is something I've come up with on my own and researching each step of the way in the process.  If you find something else that makes it easier please share!  

[sokol]

Hi

Thanks a lot @khanchett it is very helpful. 

Your approach provide the full creation and connection from VS to Azure and it could be kind of solution (I'm going to test it). 

One more question - do you know if it is possible to not execute readyAPI in VS command line but use existing ReadyApi Task from azure devops which exactly did every execution, results etc. but haven't linked to description in manual TC.

Regards

[khanchett]

The problem I ran into is that VSTest ties nicely back to the test cases with the results and everything else.  The ReadyAPI task or other tasks do not (they are basically a standalone, not connected to the test run). It is because in MSTest in Visual Studio that you are tying it to the specific test case that allows that connection through.  So anything to do outside of the Microsoft umbrella can be executed within the MSTest environment.

Also, the ReadyAPI task is limited in what it can do and I normally do the command line task in ADO instead for the needs I have.  I had asked in previous posts about it and was told the thing I was trying to do was limited and the command line was the way to go.

[Dustin0]

@khanchett 

wanted to say thank you for the initial suggestion.

I went to try and implement something like this for C#, but most of the Test projects would not allow the running of the executable.  For example an MSTest UnitTest Project would say "A Project with an Output Type of Class Library cannot be started directly"

I was able to launch Ready API within a new C# project and can utilize a Process.Start command in Main method within the class. This gives me the launch of the executable that I can probably update to be more dynamic later.   But struggled to get the command line runner working within Test projects. 

If you have some capacity to PM a bit this coming week I'd love to go over your setup for getting an executable or batch file called within TestRunner

this was my source within a standard C# project that worked

static void Main(string[] args)
{
Console.WriteLine("Starting Ready API!");
ProcessStartInfo startInfo = new ProcessStartInfo();
startInfo.FileName = "S:\\Program Files\\SmartBear\\ReadyAPI-3.9.0\\bin\\testrunner.bat";
startInfo.Arguments = "-sRegression -j -f${WORKSPACE} \"-RJUnit-Style HTML Report\" -FXML -Eqa C:\\Users\\TestUser\\Downloads\\MY-readyapi-project";
startInfo.WindowStyle = ProcessWindowStyle.Normal;
Process.Start(startInfo);
}

[khanchett · Accepted solution]

First thing is you need to setup your tests environment in Visual Studio.  I went with MSTest, seemed to be older, for my purposes it worked well with ADO.  There are other options you can use, you can research online and see what works best for you.  So I would start there.  If you are not familiar with what you choose, first step would be to create a basic test case to get comfortable.  Once you have that then what I did for the test case is basically set it up like this:

public void <MSTest Name here>()
{
     string parameter = " -P<Prametername>=2948";
     string commandline = "/c " + testrunner + testcase + environment + parameter + projectfile;
     System.Console.WriteLine("**Command Line: " + commandline); // Output to see values for debug

     System.Diagnostics.Process process = System.Diagnostics.Process.Start("CMD.exe", commandline);
     process.WaitForExit();
     int x = process.ExitCode;
     System.Console.WriteLine("~~~ " + testrunner);  // Output to see values for debug
     System.Console.WriteLine("Exit code: " + x); // Output to see values for debug
     Assert.IsTrue(x.Equals(0)); // Tells us if this is successful or not. 0=Pass
}

I setup all these variables to the values I needed to pass in and make life easier.  (Using the "Launch TestRunner" in ReadyAPI helps here). 

I would suggest you get this to work first and then worry about cleaning it up later.

Hopefully you find this helpful.  If you run into anything please don't hesitate to reach out to me privately and I can give you more detail. 😊

KHanchett