hello bqualters
That seems a little bit like re-inventing the wheel. From your description, it sounds a little complicated for just a little bit of information. Why not just run the suite as normal but include a few well placed log.info lines of groovy code into some groovy test steps for the test cases you would like information from? At the conclusion of the run, the script log will contain all the values from the test case that you would like to look at. If you run your project or suite or testcase from the testrunner.bat command line interface (smartbear has good information for that when googled... https://www.soapui.org/docs/test-automation/running-from-command-line/functional-tests/), it will have a log file created for posterity that anyone who cares about the test run can review. The log would have individual text files for full request/response data that shows what was executed also.
Below is example of Groovy Script test step that does assertions in code (lines with "assert")...
log.info 'Test Step "' + testRunner.runContext.currentStep.name + '" start...';
log.info "";
//def rawResponse = context.expand( '${HTTP Request GET KeyConfig#RawResponse}' ); // for ReadyAPI
def myRequestStep = testRunner.testCase.getTestStepByName('HTTP Request GET KeyConfig');
// for SoapUI
def rawResponse = new String(myRequestStep.testRequest.messageExchange.rawResponseData); // for SoapUI
log.info "rawResponse=>" + rawResponse.toString() + "<";
tokenizedResponse = rawResponse.tokenize();
httpCode = tokenizedResponse[1];
httpMsg = tokenizedResponse[2];
//tokenizedResponse.each {
// log.info "it=" + it;
//};
def actualResponseCodeStr = "${httpCode} ${httpMsg}";
def expectedResponseCodeStr = '200 OK';
assert expectedResponseCodeStr == actualResponseCodeStr, "Expected Response Code of '" + expectedResponseCodeStr + "', but actual Response Code is '" + actualResponseCodeStr + "'";
def response = context.expand( '${HTTP Request GET KeyConfig#Response}' );
log.info "response=>" + response.toString() + "<";
testRunner.testCase.testSteps['Properties'].setPropertyValue('apiResponse', response.toString());
log.info "";
log.info 'Test Step "' + testRunner.runContext.currentStep.name + '" done...';
More Groovy code with an assertion and JsonSlurper use...
import groovy.json.JsonSlurper;
log.info 'Test Step "' + testRunner.runContext.currentStep.name + '" start...';
log.info "";
def jsonSlurper = new JsonSlurper();
def apiResponse = context.expand( '${Properties#apiResponse}' ); // log.info 'apiResponse=' + apiResponse;
def jsonResponseObjActual = jsonSlurper.parseText(apiResponse); // log.info "jsonResponseObjActual=" + jsonResponseObjActual;
log.info "jsonResponseObjActual=" + jsonResponseObjActual;
log.info "element count=" + jsonResponseObjActual.size();
def expectedCount = 3; // just sanity check since returned data will not contain service level specific data
assert expectedCount == jsonResponseObjActual.size(), "The number of elements returned was expected to be '${expectedCount}' but it was actually '${jsonResponseObjActual.size()}'";
jsonResponseObjActual.each { name, value ->
log.info ' ' + name + ' ' + value;
// expectedValue = databaseExp.("${name}"); // log.info "expectedValue=" + expectedValue;
// assert value == expectedValue, "The value of '${name}' was expected to be '${expectedValue}' but it was actually '${value}'";
// expectedValue = jsonResponseObjExpected.("${name}");
// assert value == expectedValue, "The value of '${name}' was expected to be '${expectedValue}' but it was actually '${value}'";
};
log.info "";
log.info 'Test Step "' + testRunner.runContext.currentStep.name + '" done...';
Regards,
Todd
TNeuschwanger-Thanks for the response!
At this point I have figured out how to do what I wanted in terms of being able to loop through all the tests in the suite one at a time, then through each test step in each case, run dynamic data retrieval SQL's and use the values for checkpoints, BUT I am still trying to figure some things out - especially the use of assertions if they fail in the groovy script (not in the embedded groovy script of a step mind you - just in the Groovy Script test that acts as my driver). If the assertion fails the script errors out - that doesn't seem to make sense as I thought it would simply cause the test to fail - so I built my own code to do the check at this point. The assert seems to have made itself useless because of this fail possibility.
Also, using the messageExchange object seems to only be available if you are in a Script Assertion step within a test step - I don't know what I need to import in a regular Groovy script to have access to that so I worked my way around it. I get an error when I try to use messageExchange so...not sure what to do other than not use it here. I can use it in the embedded Assert Script but that defeats the purpose of having ONE SCRIPT for all test cases.
The point of what I am trying to do is make it so that all the guys on my team, WHO HAVE NEVER DONE API testing or used this tool or written Groovy scripts, can create test cases simply through an excel spreadsheet. I will (eventually) read from an Excel file what the desired test is to create and dynamically create it - assertions and all, dynamically through the driver script - much like I currently do with our functional testing in UFT.
I just started with SoapUI - in my free time for the last month or so - and found it cumbersome (the free version mind you as work is too cheap to pay for anything) to build a single test case manually - you have to get the wsdl, make all the header settings, set up properties, do data prep with the AUT to get to initial condition, then make your assertions for each step one by one within the tool, not to mention the ridiculously poor debugging tools and meaningless error messages that tell you the error is on line 12 when it's actually on line 378 etc. Also, am I missing capabilities like putting breakpoints and viewing runtime variable values while the script is in progress? If so, please advise as I am all ears and hoping it gets better. So anyway, I am writing a script that runs in whichever test suite you place it, and utilizes the various property levels to make as much of the testing dynamic with the focus being FUNCTIONAL testing.
My big problem right now is figuring out how to make sure the test steps use the property values I put in the api variable (parameter) places - for example, I think I mentioned above, each of my tests is to be written with a SQL statement that finds a valid piece of data that meets needed initial conditions - writes the result to a test case property, and that test case property is referenced in the parameter space for the API call - when the script runs - it fails - don't know why as log doesn't tell me. BUT, after the run, since the data has been retrieved and written to the testcase property being referenced, if I click Run manually to run it again - it works and uses the value the SQL retrieved. So..what's the deal there?
I will be the first to admit I am NOT a Java / Java Script / object oriented programming languages person. I am self taught at everything I do and as such - what I've learned MAY BE WRONG! So if you have some advice -I will take it.
Also, in regards to the logs that SoapUi provides - they suck. Maybe I'm missing something but they simply tell you whether or not the test step passed or failed - but lack the details as to WHY. For example, I created 5 test cases - ran them at the Test Suite level and generated the report. The only info given is the basics and the response package BUT there was nothing about which assertion failed even for the assertions I created with the built-in tool. If I have 5 or 6 different checks I want to know WHICH one failed IN THE REPORT LOG so I don't have to go open up the test steps individually and see which are green and which are red...what am I missing? Again - I have written my own checkpoints now and don't need the built in ones unless I want to use them so I can turn the step green or red and make it look pretty that way. Seems to me, if you grab the response and convert it to a string you can easily do almost ALL the checks that are "built-in" with very little effort using the Groovy commands - ie contains, count etc. The manual creation part is very cumbersome and time consuming.
Let me know what you think - especially on the parts where I have actual questions and not so much the complaints - I complain alot for a non-developer type person - I know that. But hey, with a lack of knowledge comes a willingness to blame others!
hello bqualters
That is a lot to unpack for a question you need answered... 🙂
"If the assertion fails the script errors out - that doesn't seem to make sense as I thought it would simply cause the test to fail"
If you would like the testcase to finish even though an assertion fails you can un-ticked the "Abort test if an error occurs" within the TestCase Options feature of every testcase.
Also, in regards to the logs that SoapUi provides - they suck. Maybe I'm missing something but they simply tell you whether or not the test step passed or failed - but lack the details as to WHY
You get as much out of the log information as you put into them... My testcases consist almost entirely of Groovy test steps...
As you can see from the sample Groovy I submitted before, you can put as much log information as you need to help debug your scripts. As a testcase runs, I can view the "script log" tab at the bottom of SoapUI screen to see all of the log statement content so I know exactly where my issue might be when a failure occurs. I often include a lot of log.info code when developing a test and comment it out or delete it after I see what is happening. If there is a groovy failure, I can usually review the "error log" tab for specific language/programmatic failure reasons. That might be a communication gap here... I use the script log as my canonical report which is different than an actual report that you can generate in SoapUI user interface.
I was drawn to SoapUI to begin with because of its ability to be flexible in meeting my needs. If it did not have a feature I needed, I could write my own in Groovy. I think I won't be of much help in your endeavor to create a monolithic script to do what you want. I AM a Java / Java Script / object oriented programming languages person and prefer my projects to be modular in nature instead of your desired "purpose of having ONE SCRIPT for all test cases." I can try to answer small specific questions though.
Regards,
Todd
Still not sure why this is happening-->
I can not seem to get the test steps to execute using the referenced variables (either as part of the request or part of the assertion) - for example, I run a SQL - it retrieves an OrderKey - which I write as a Property at the testcase level - then I sub in that value using the parameter in the API Request of the test step and instead of using the hardcoded value I supply ${#TestCase#OrderKey} in the parameter value column - this works if I trigger the execution of the step manually as it uses the value currently stored in the test case Property called OrderKey BUT--> when I run the script to get it to execute this step it fails even though the SQL has run and defined the value in the test case property (OrderKey) successfully - it fails to make the request and throws an error. If I manually execute it again at this point with the retrieved value from the SQL - it works. The same goes for the Assertions - if I reference a property - it fails when run through the script (which I removed the variablized "submission" value and hardcoded it) - but if I click the Request to run manually with same setup - it works.
Sooo...what's the deal?
hello bqualters
it is hard to visualize what you are doing... could you include some screen pictures of your test case and show the code/dialog where you are setting and getting properties? Are you using jdbc, property transfers, groovy script for the tasks you describe?
The community edition of SoapUI is a pretty mature product, so I don't think it could be a bug or many people would have encountered this issue. We will have to look to see what the test steps are in your test case.
Regards,
Todd
