Best practices for synchronizing coordinated tests?
In our test environment, we have test scenarios that involve
- take a snapshot of our system state (this can take several minutes)
- poke the system to cause a certain kind of change;
- WAIT for the change to propagate through the system (may take 5 to 10 minutes);
- take another snapshot of the system state (again, several minutes);
- compare the two snapshots to validate that the correct change occurred.
Little of this has been implemented in ReadyAPI (SoapUI Pro), it's mostly still manual tests.
What I would like to be able to do is define and implement each such test sequence as a coherent entity (so that the appropriate poking and comparing of snapshots are tied together in one grouping of some sort), but then run a bunch of them as a group, so that we only take one pair of snapshots, and only do the waiting once. It would be great if I could somehow just provide a list of the test scenarios to be included, so that I could run one or many, without having to modify the master project or whatever it is each time I want to run a different collection of tests.
So we'd want to be able to have some sort of master ?project? ?test suite? that would start this all off, and take the initial snapshot.
Then it would cause the various other scenarios to do their "poking" work, preferably (but not necessarily) running in parallel.
Then if things were running in parallel, some synchronization would happen, to wait until all the test scenarios reported that they were done with their poking.
Then we'd do the waiting for the system to trickle the changes through.
Then the master would take the second snapshot.
Then the master would allow each scenario to proceed with its own comparison of the snapshots to decide if that scenario had worked. Again, preferably but not necessarily in parallel.
Then somehow the results of all these test scenarios would be aggregated and reported.
So my main point is that I want to take some fairly independent test scenarios and aggregate them, but in this sort of chopped-up way. Secondarily, I would like to run things in parallel.
Does the ReadyAPI / Groovy environment give me a way to fire off separate projects in parallel, or separate test suites in parallel? Does it provide some sort synchronization mechanisms so that the master thread can know when all the children have completed their poking, and so that each child can know when it's time to do its snapshot comparisons?
Are there other ways of thinking about this that would be better?
Thank your for your patience in helping with this.
EDIT: re-worded to de-emphasize running things in parallel.