Ask a Question

Pro-Tip: Automation expediency vs. automating real world

Esteemed Contributor

Pro-Tip: Automation expediency vs. automating real world

I just wanted to share a scenario from my past as a test automater.  Occasionally, there are discussions on here where we weigh the balance of whether or not we should do something in our automation because it's "easier" to automate or if we should go with a "harder" solution because it's more real-world.  The short answer is "it depends".  But a longer answer has to take into consideration the risks involved if you're not fully automating real-world scenarios.  


Use Case:

I used to do automated testing for a company that did a point of sale application.  In our initial set of automated tests, we were using an administrator level user to perform all the tests.  Functionally speaking, that user could do whatever we needed to be done and so our automation worked.  It was easy, it was expedient, and we didn't need to maintain multiple levels of users in our test environment. However, there was a bug in the system that our automation did not catch.  A pretty big and nasty one.  One that showed up “in the field” at an actual customer site during their main business. Really bad.


Basically, when you logged in as an administrator you went to one set of screens that initialized some objects that were needed for the rest of the application to work. However, if you logged in as a cashier, those objects did not get initialized and every operation attempted by the cashier failed with major errors on the screen.  If our test automation had actually been using the correct role to test those functions, we would have caught that problematic code before it was shipped.


Making the choice:

Now, this is a different use case than making the decision of whether or not to use "SetText" or "Keys" or whether or not to actually find and click on a component versus calling the low level code behind it.  Each situation and scenario has it's risks and benefits.  Sometimes, it makes sense to do things the expedient way if the test case is not dependent upon the real-world stuff.  Sometimes, it is better to do the hard work of configuring a real-world scenario.  The key is to know the application you are testing, to have a good relationship with the developers, and to be tuned into what changes are being made within the application so you know how best to target your tests.  Knowing this stuff will help make those decisions easier. 


Robert Martin
[Hall of Fame]
Please consider giving a Kudo if I write good stuff

Why automate?  I do automated testing because there's only so much a human being can do and remain healthy.  Sleep is a requirement.  So, while people sleep, automation that I create does what I've described above in order to make sure that nothing gets past the final defense of the testing group.
I love good food, good books, good friends, and good fun.

Mysterious Gremlin Master
Vegas Thrill Rider
Extensions available

I run into this sort of dilemma every day. Always the close to an end user experience should the test steps be? And more difficult (and very subjective) to answer, how do we define a realistic user experience for this test? 🙂


Before I commit to automating something with TC...
1. Are the sequence of steps that make up this 'test case' likely be re-used by another 'test case' in the future?

2. Are the sequence of steps that make up this 'test case' central/vital to what is being tested in this 'test case' or is it just getting the environment into a particular needed state?

3. Do I have time for perfection or do I need to come back later on and make the test steps more realistic?

4. Is the part of the UI that I am automating with TC likely to be swapped out by a third party component (devexpress)? If that is the case, there is a risk I will have throw out the test that was tightly coupled to the old UI component and rewrite it.

Showing results for 
Search instead for 
Did you mean: