I have an issue with some inconsistent test results. I will explain the scenario and then what is different between a manual execution and an automated execution.My scenario involves executing PUT / GET / POST and DELETE methods at a specific endpoint. I am trying the following:PUT - specifying a device IDPUT - without specifying a device IDGET - specifying a device IDGET - without specifying a device IDDELETE - specifying a device IDDELETE - without specifying a device IDPOST - create a new devicePOST - update an existing deviceThe results I expect are as follows:PUT - specifying a device ID - FAILPUT - without specifying a device ID - PASSGET - specifying a device ID - PASSGET - without specifying a device ID - PASSDELETE - specifying a device ID - PASSDELETE - without specifying a device ID - FAILPOST - create a new device - PASSPOST - update an existing device - PASSOn each test I have one or more assertions so, if the two tests above fail with the right HTTP response they actually pass my test if that makes sense.When I run the tests sequentially manually everything work as I expect. When I run the test suite everything also works as I expect. When I run the tests via a driver script written in Groovy I get the following failures, which I do not expect:GET - specifying a device ID - FAILPOST - update an existing device - FAILIn between some of the tests I am performing property transfers of the device ID. The full execution therefore looks like this:PUT - specifying a device ID - FAILPUT - without specifying a device ID - PASSDevice ID Property Transfer #1GET - specifying a device ID - PASSGET - without specifying a device ID - PASSDevice ID Property Transfer #2DELETE - specifying a device ID - PASSDELETE - without specifying a device ID - FAILPOST - create a new device - PASSDevice ID Property Transfer #3POST - update an existing device - PASSTo reiterate, when running these sequentially manually or at test suite level, this all works fine and the property transfers occur successfully.When running this via the driver script I am doing nothing elaborate other than running the tests by name within the test suite - i.e. nothing different to what pressing the green play arrow icon in the test suite does.I can see why the tests are failing but I do not understand the reason for the failure and I hoping someone can assist me.My assertion for the failed test step 'GET - specifying a device ID' is checking the response element irrespective of the namespace and looks like this:-------------------//*:Response[1]-------------------<Response xmlns="${#Project#prop_ServiceEndpoint}/accounts/${#Project#prop_Account_1}/devices/${GET - With Device ID#Id}"> <AccountID>${#Project#prop_Account_1}</AccountID> <Description/> <DeviceID>${GET - With Device ID#Id}</DeviceID> <Number>${#Project#prop_Number_1}</Number> <Order>*</Order> <PIN null="true"/> <Verified>true</Verified></Response>-------------------As you can see I have parameterised the namespace in the response. The part that is failing is the ${GET - With Device ID#Id} part in the namespace. It is seemingly not holding any value at the point the assertion is made. For example the actual namespace in the response is 'https://<url>/accounts/<account_number>/devices/123' (please not, I have replaced sensitive information with placeholders in angled brackets). But the part it tries to match against is 'https://<url>/accounts/<account_number>/devices/' - i.e. it does not have the device identifier. I have got similar tests in other endpoints and they all work fine with the Groovy script and parameters in the namespace but for some reason this one is misbehaving.The other unexpected test failure - 'POST - update an existing device' fails on an assertion which only checks the HTTP response code. If I run it manually it works and gets the right HTTP response, but if I use the Groovy script it comes back with a 500. When I execute it manually immediately after it fails, it works. I have checked the back-end databases and the information that should be present is there and so this is not a database or other server issue.In both of these unexpected failures I can see that the property transfer test step has executed successfully and the value has been passed into the right place but execution still fails. I have spent an inordinate amount of time on this and I cannot fathom what the problem is. Everything points to the Groovy script being incorrect but I am not doing anything other than looping through the test steps within the test case and executing them in sequential order - nothing different than what is being done when I manually execute them myself or run them at the test suite level.The assertion error I receive on the first failed test, is:XPathContains assertion failed for path [//*:Response[1]] : Exception:org.custommonkey.xmlunit.Diff[different] Expected namespace URI 'https://<url>/accounts/<account_number>/devices/' but was 'https://<url>/accounts/<account_number>/devices/978' - comparing <Response...> at /Response[1] to <Response...> at /Response[1]Please note, I have once again replaced sensitive information with placeholders in angled brackets.I had thought the issue might be one of timing since the Groovy script executes very fast indeed but I have put in delay steps of varying lengths and I consistently receive the same responses.Could anyone shed any light on this? Unfortunately I cannot post the Groovy script as my company would not allow it but it is not complicated. In essence the execution code is essentially as follows:for( testStep = 0; testStep < testCase.getTestStepCount(), testStep++ ){ testRunner.runTestStep( testCase.getTestStepAt( testStep ) );}Yet the results of this differ from manually executing the same test steps.Any help would be greatly appreciated.

Hi,Are you running the Groovy script using testrunner or in the GUI?In the example code you posted your using testRunner object which is used when running with testrunner utility. Regards,MarcusSmartBear Support

Hi, I unsure what you are referring to (I'm new to the tool). I am running the Groovy script as another test step. Please see the attachment, which display how I am running the Groovy script by clicking the green 'run' icon. I would assume this is running it via the GUI but I am not sure.

Could this be a specific problem with assertions where the failing test step contains a parameterised value. For instance, ${GET - With Device ID#Id}, which would resolve to the device ID that I am interested in. I have got another test which is failing that does not follow this pattern so this is something of a stab in the dark. I don't think for a minute that this is insurmountable but I currently don't know how to resolve this.

Hi, thank you for the reply. I am doing some customisation but only inasmuch as I am recording some verbose logging for each test step executed. I am recording the execution time of each test step executed in the test case along with the raw response and raw request, response data and request data and the result of the test (e.g. pass or fail). I am additionally creating text files for debugging purposes with the same content. For these reasons I don't believe I can use the solution you propose as I would be unable, insofar as I can tell, to get the level of granularity in the reporting/logging that I require. If you know differently I'd be really grateful if you could explain how I can still obtain then same information but using testRunner.testCase.testSuite.run(null,false) instead. Do you know if this method would resolve the unexpected failures I mentioned or is your reply more along the lines of best practice rather than a resolution to my issue? Regards.

Forum Discussion

GMSoapUI

Contributor

12 years ago

[Resolved] Manual test execution works, automation does not.

I have an issue with some inconsistent test results. I will explain the scenario and then what is different between a manual execution and an automated execution.

My scenario involves executing PUT / GET / POST and DELETE methods at a specific endpoint. I am trying the following:

PUT - specifying a device ID
PUT - without specifying a device ID
GET - specifying a device ID
GET - without specifying a device ID
DELETE - specifying a device ID
DELETE - without specifying a device ID
POST - create a new device
POST - update an existing device

The results I expect are as follows:

PUT - specifying a device ID - FAIL
PUT - without specifying a device ID - PASS
GET - specifying a device ID - PASS
GET - without specifying a device ID - PASS
DELETE - specifying a device ID - PASS
DELETE - without specifying a device ID - FAIL
POST - create a new device - PASS
POST - update an existing device - PASS

On each test I have one or more assertions so, if the two tests above fail with the right HTTP response they actually pass my test if that makes sense.

When I run the tests sequentially manually everything work as I expect. When I run the test suite everything also works as I expect. When I run the tests via a driver script written in Groovy I get the following failures, which I do not expect:

GET - specifying a device ID - FAIL
POST - update an existing device - FAIL

In between some of the tests I am performing property transfers of the device ID. The full execution therefore looks like this:

PUT - specifying a device ID - FAIL
PUT - without specifying a device ID - PASS
Device ID Property Transfer #1
GET - specifying a device ID - PASS
GET - without specifying a device ID - PASS
Device ID Property Transfer #2
DELETE - specifying a device ID - PASS
DELETE - without specifying a device ID - FAIL
POST - create a new device - PASS
Device ID Property Transfer #3
POST - update an existing device - PASS

To reiterate, when running these sequentially manually or at test suite level, this all works fine and the property transfers occur successfully.

When running this via the driver script I am doing nothing elaborate other than running the tests by name within the test suite - i.e. nothing different to what pressing the green play arrow icon in the test suite does.

I can see why the tests are failing but I do not understand the reason for the failure and I hoping someone can assist me.

My assertion for the failed test step 'GET - specifying a device ID' is checking the response element irrespective of the namespace and looks like this:

-------------------

//*:Response[1]

-------------------

<Response xmlns="${#Project#prop_ServiceEndpoint}/accounts/${#Project#prop_Account_1}/devices/${GET - With Device ID#Id}">
<AccountID>${#Project#prop_Account_1}</AccountID>
<Description/>
<DeviceID>${GET - With Device ID#Id}</DeviceID>
<Number>${#Project#prop_Number_1}</Number>
<Order>*</Order>
<PIN null="true"/>
<Verified>true</Verified>
</Response>

-------------------

As you can see I have parameterised the namespace in the response. The part that is failing is the ${GET - With Device ID#Id} part in the namespace. It is seemingly not holding any value at the point the assertion is made. For example the actual namespace in the response is 'https://<url>/accounts/<account_number>/devices/123' (please not, I have replaced sensitive information with placeholders in angled brackets). But the part it tries to match against is 'https://<url>/accounts/<account_number>/devices/' - i.e. it does not have the device identifier. I have got similar tests in other endpoints and they all work fine with the Groovy script and parameters in the namespace but for some reason this one is misbehaving.

The other unexpected test failure - 'POST - update an existing device' fails on an assertion which only checks the HTTP response code. If I run it manually it works and gets the right HTTP response, but if I use the Groovy script it comes back with a 500. When I execute it manually immediately after it fails, it works. I have checked the back-end databases and the information that should be present is there and so this is not a database or other server issue.

In both of these unexpected failures I can see that the property transfer test step has executed successfully and the value has been passed into the right place but execution still fails. I have spent an inordinate amount of time on this and I cannot fathom what the problem is. Everything points to the Groovy script being incorrect but I am not doing anything other than looping through the test steps within the test case and executing them in sequential order - nothing different than what is being done when I manually execute them myself or run them at the test suite level.

The assertion error I receive on the first failed test, is:

XPathContains assertion failed for path [//*:Response[1]] : Exception:org.custommonkey.xmlunit.Diff
[different] Expected namespace URI 'https://<url>/accounts/<account_number>/devices/' but was 'https://<url>/accounts/<account_number>/devices/978' - comparing <Response...> at /Response[1] to <Response...> at /Response[1]

Please note, I have once again replaced sensitive information with placeholders in angled brackets.

I had thought the issue might be one of timing since the Groovy script executes very fast indeed but I have put in delay steps of varying lengths and I consistently receive the same responses.

Could anyone shed any light on this? Unfortunately I cannot post the Groovy script as my company would not allow it but it is not complicated. In essence the execution code is essentially as follows:

for( testStep = 0; testStep < testCase.getTestStepCount(), testStep++ )
{
      testRunner.runTestStep( testCase.getTestStepAt( testStep ) );
}

Yet the results of this differ from manually executing the same test steps.

Any help would be greatly appreciated.

12 Replies

SmartBear_Suppo
Alumni
12 years ago
Hi,

Are you running the Groovy script using testrunner or in the GUI?

In the example code you posted your using testRunner object which is used when running with testrunner utility.

Regards,
Marcus
SmartBear Support
GMSoapUI
Contributor
12 years ago
Hi, I unsure what you are referring to (I'm new to the tool). I am running the Groovy script as another test step. Please see the attachment, which display how I am running the Groovy script by clicking the green 'run' icon. I would assume this is running it via the GUI but I am not sure.
GMSoapUI
Contributor
12 years ago
Could this be a specific problem with assertions where the failing test step contains a parameterised value. For instance, ${GET - With Device ID#Id}, which would resolve to the device ID that I am interested in. I have got another test which is failing that does not follow this pattern so this is something of a stab in the dark. I don't think for a minute that this is insurmountable but I currently don't know how to resolve this.

SmartBear_Suppo

Alumni

12 years ago

Hi,

Yes, you are running through GUI. I was referring to testrunner in <soapui install>\bin\testrunner.bat (.sh for non-windows). This is what is typically used for automating test in SoapUI since it is ran outside of the GUI.

To reiterate, when running these sequentially manually or at test suite level, this all works fine and the property transfers occur successfully.

When running this via the driver script I am doing nothing elaborate other than running the tests by name within the test suite - i.e. nothing different to what pressing the green play arrow icon in the test suite does.

If that is the case then you should use:

testRunner.testCase.testSuite.run(null,false)

That will run the entire test suite from groovy.
Is there some customization you are doing in groovy after each testStep? Because I do not see why you need to run them by name.

Regards,
Marcus
SmartBear Support

GMSoapUI
Contributor
12 years ago
Hi, thank you for the reply. I am doing some customisation but only inasmuch as I am recording some verbose logging for each test step executed. I am recording the execution time of each test step executed in the test case along with the raw response and raw request, response data and request data and the result of the test (e.g. pass or fail). I am additionally creating text files for debugging purposes with the same content. For these reasons I don't believe I can use the solution you propose as I would be unable, insofar as I can tell, to get the level of granularity in the reporting/logging that I require. If you know differently I'd be really grateful if you could explain how I can still obtain then same information but using
testRunner.testCase.testSuite.run(null,false)
instead. Do you know if this method would resolve the unexpected failures I mentioned or is your reply more along the lines of best practice rather than a resolution to my issue? Regards.
GMSoapUI
Contributor
12 years ago
I have just run a small test using
testRunner.testCase.testSuite.run(null,false)
and this works fine and the assertions pass as I would expect. It seems, therefore, that the issue is with
testRunner.runTestStep( testCase.getTestStepAt( testStep ) );
. Is this a known issue with this method of the testRunner onject in that parameter assertions fail? Is there a workaround with
testRunner.testCase.testSuite.run(null,false)
whereby I can still achieve the verbose logging by referencing the testStep by name or by its index? This is going to be a real problem for me if this cannot be resolved.

SmartBear_Suppo

Alumni

12 years ago

Hi,

I am recording the execution time of each test step executed in the test case along with the raw response and raw request, response data and request data and the result of the test (e.g. pass or fail).

You can use event handlers for this.
http://www.soapui.org/Scripting-Propert ... dlers.html

As an example to get raw request and response data, use event handler SubmitListener.afterSubmit.


String rawRequest = new String( submit.response.getRawRequestData() )
String rawResponse = new String( submit.response.getRawResponseData() )

Regards,
Marcus
SmartBear Support

GMSoapUI
Contributor
12 years ago
Thank you for the reply. I will investigate this and see if this will resolve the overall problem while still allowing me keep the verbose logging and text file output to hard disk. I will not be able to look at this untilo next week now but I will post back with how I get on.
SmartBear_Suppo
Alumni
12 years ago
Thanks for the update, we'll wait for your reply.

--
Regards

Erik
SmartBear Sweden

GMSoapUI

Contributor

12 years ago

I have tried a very small Groovy script incorporating the suggested workaround. Here is the code:

 def vntProject = testRunner.testCase.testSuite.project;

 // for loop for controlling test suite in use
 for( vntTestSuite in vntProject.testSuites )
 {
  //assign the test suite to a variable
  vntTestSuite = vntProject.getTestSuiteByName( vntTestSuite.key );

  // assign test suite name to a variable
  strTestSuiteName = vntTestSuite.name.toString();

  // ignore the Groovy test suite as we don't want to run this as this is the driver script - without this execution would get into an infinite loop
  if( strTestSuiteName.indexOf( "Groovy" ) == -1 )
  {
   // log the name of the test suite in use
   log.info( " > TEST SUITE NAME IS: " +  strTestSuiteName );
   log.info( "hello" );
   testRunner.testCase.testSuite.run( null, false );
   log.info( "world!" );
  }
 }

However the code executes to the point of writing "hello" into the Log Output pane but then hangs at:

testRunner.testCase.testSuite.run( null, false );

It never gets beyond this so never writes "world!" to the Log Output. No other log tabs contain any further information that helps me track the nature of the problem down.

Forum Discussion

[Resolved] Manual test execution works, automation does not.

12 Replies

Recent Discussions

Test result Pass when data source times out

Run Test Case properties not populating for Data Source based items

Save Groovy output to JUnit report

Related Content

Zephyr Cloud API - automated creation of execution

Disable device cloud extension through automation in test Execute 14.4

SaaS Platforms Like Salesforce – Automation: Feasible or Futile?