Pro-Tip: Automation expediency vs. automating real world

I just wanted to share a scenario from my past as a test automater.  Occasionally, there are discussions on here where we weigh the balance of whether or not we should do something in our automation because it's "easier" to automate or if we should go with a "harder" solution because it's more real-world.  The short answer is "it depends".  But a longer answer has to take into consideration the risks involved if you're not fully automating real-world scenarios.  

 

Use Case:

I used to do automated testing for a company that did a point of sale application.  In our initial set of automated tests, we were using an administrator level user to perform all the tests.  Functionally speaking, that user could do whatever we needed to be done and so our automation worked.  It was easy, it was expedient, and we didn't need to maintain multiple levels of users in our test environment. However, there was a bug in the system that our automation did not catch.  A pretty big and nasty one.  One that showed up “in the field” at an actual customer site during their main business. Really bad.

 

Basically, when you logged in as an administrator you went to one set of screens that initialized some objects that were needed for the rest of the application to work. However, if you logged in as a cashier, those objects did not get initialized and every operation attempted by the cashier failed with major errors on the screen.  If our test automation had actually been using the correct role to test those functions, we would have caught that problematic code before it was shipped.

 

Making the choice:

Now, this is a different use case than making the decision of whether or not to use "SetText" or "Keys" or whether or not to actually find and click on a component versus calling the low level code behind it.  Each situation and scenario has it's risks and benefits.  Sometimes, it makes sense to do things the expedient way if the test case is not dependent upon the real-world stuff.  Sometimes, it is better to do the hard work of configuring a real-world scenario.  The key is to know the application you are testing, to have a good relationship with the developers, and to be tuned into what changes are being made within the application so you know how best to target your tests.  Knowing this stuff will help make those decisions easier.