Hi Aaron,
I previously worked in a company where we had nearly 3000 data-driven tests running in over 40 test suites. The actual test cases in SoapUI were around 400 or so, we re-used the same test cases to test various scenarios i.e a test case is to login, getProfile, verifyExpected and actual, fill out report, clear relevent properties and run the same test again with different data-set, i.e login with superUser, admin or employee etc.
You can re-use the same tests with data-driven approach to test various combinations. The practice you have described seems pretty decent, though I would add reporting/tests outcome in there to know what scenario/data-set passed or failed.
In automation, the key thing is to re-use as much as possible. We don't have an option of step defs in SoapUI, though we can use the concept of step defs within our test cases.
Strongly agree with Radford in regards to ensure the test cases can be run independently. They should not depend on previous steps to pass or generate data. In future, if your regression pack beocme very large, it can be managed by splitting the XMLs in two or three and run them in paralell to save time and to ensure all tests will still pass as there is no dependency.
Hi,
> ensure the test cases can be run independently.
> In automation, the key thing is to re-use as much as possible.
Well, just my $0.02...
(Disclaimer: I am from TestComplete world which means functional testing and this might matter)
I think that the peculiarity here is that test cases may migrate into test steps when we proceed from simplest (unit) tests to more and more complex (end-to-end) ones.
For example, let's consider the lifecycle of some publication in some system.
Initially, an empty publication must be created. Then one or more topics can be added to it. Then one or more images can be added as well. Then other items like content, annotation, etc. can be added. Publication can be saved after any above step. Saved publication that is assumed to be ready, may need to pass one or more approval steps. Approved publication may be released to the public. Released publication may require additional registration.
All the above steps must be verified during development and initial testing and this verification can be implemented as independent tests that either use stubs (that imitate existing empty publication) or use the minimal steps set (e.g. create an empty publication and add an image to it).
However, when you proceed from simple verifications to more and more complex end-to-end tests, more and more of existing independent tests may (and must) be reused, thus, becoming to be test steps.
For example, when you test that a publication can be created, an image can be added to it and then publication can be saved, you are actually creating a wrapping test case that utilizes three existing ones as its steps. But those three steps (tests) are no more independent ones as they must act against the same publication.
Even more: in the real world, some actions may require a reasonable time to be executed which means that it may be a good idea to reuse results of the tests executed previously in order to speed-up the overall tests run time.
For example, if it 'costs' (in terms of time, resources, etc.) to create a new publication and fill it with the content, it may be not a bad idea when verifying that the created publication can be publiched, not to create a brand new publication, but reuse the one that was already created by tests executed previously and just publish it.
With all the above in mind, one might consider a framework (not sure if it is considered lightweight or not:) ), when tests store references to the created artifacts (publication name, its status, sections added, etc.) to some storage. And it is up to the subsequent tests to either reuse the existing results or repeat the already executed actions over and over again.
