Forum Discussion
I am facing a similar issue in my current position of how to work in test automation in an agile environment.
The primary concerns I have are reusability and where to focus our efforts.
I have for some time been automating our test cases just for the sake of getting them done and appeasing my coworkers who are desperate to justify their necessity - but where does the meat go? I have yet to see any value in this type of automation. In fact I find writing test cases to be a complete waste of time and added fluff to the QA process. Writing out scenarios and breaking down requirements into meaningful action items to test I have found to be pretty useful but writing out all my steps for everything I test...waste of time.
Just as I drive to work every day I do not need to write the steps and document every turn I take to arrive at my destination. I do it quite well on my own - naturally - because I have done it for some time now and I am familiar with the steps involved to start my vehicle and arrive at work. Sure I occasionally look for alternate routes - or maybe even drive somewhere besides work - but the maps do not apply to subsequent trips and therefore do not see a purpose to keep them around.
I know how to drive my car - I know how to use google maps - I'm prepared for whatever lies ahead.
So automation can jump through a bunch of steps with different data sets, but so can I - and I learn more - catch more - do it faster without having it automated or wasting time on the fluffy redundant documentation of that process.
So I have typically been leaning towards Automation being valuable only in regression testing.
What do you all think about test cases and can you elaborate on the greatest value that automation has in the agile environment?
One thing of note on the "waste of time" part of documenting test cases. Let's use your example of driving to work.
For you, it's natural. Heck, when I drive to work, sometimes I kind of go, mentally, on auto-pilot. I know where all the turns are, where all the exits are, etc. I know EXACTLY what to do every time.
HOWEVER... if I have a friend that I want to have visit me at the office and they've NEVER been here before. Even worse, they have never been in this part of the country EVER. They want to get to my place of employment from my home town, I need to communicate how to get here. And, for that matter, I'd better be pretty clear on the directions because in the 90 minute drive I have, there are a LOT of turns, exits, roads, etc. Not to mention contingencies of "if it's snowing, forget that route and take this alternate route". Somehow, that communication needs to happen. Even if we go techno-modern and tell them "just use the GPS, here's the address"... well, SOMEONE needed to input all that data into the GPS system in the first place, right?
Consider test plans, analysis, and so on from this perspective. They are the means of communicating to someone else how to do what you just did. The possibility exists for you to get hit by a bus tonight and SOMEONE needs to know a) what you've already tested and b) what still needs to be tested and c) how to go about doing so because that SOMEONE has to ramp up pretty quickly. Even exploratory testing requires some sort of documentation to this effect. If I've spent 5 hours testing an application and someone comes up and says, "How far are you from being done?" what do you respond with? If you don't have a measurement point to say, "Well, out of 40 requirements, I've fully tested 29, partially tested 10, and have one yet to address", you're basically going to answer with "Um... I dunno".
In Agile, also, there's this idea of "the definition of done". You need some sort of metric to decide when done means done. If there's no analysis or measurement, then you can't define done accurately. Again, the level of documentation may be different from project to project or environment to environment, but somehow the definition needs to be made.
Let's go one step further.. back to driving to work. I get a call after half an hour, "Dude, I'm like TOTALLY lost. There was an accident on the route and I got detoured and have NO idea where to go from here." My answer is "What is the last road that you were on in the directions and where did you go from there?" If my friend doesn't have that information, I'm just as lost as they are and have no real way of getting them to my office.
Test plans, test cases, test scenarios, test scripts provide that necessary documentation to evaluate "OK, something unexpected happened. How did we get here so we can figure out what went wrong?" Again, exploratory testing SHOULD include documenting what you did. If there's no documentation, how do you report the reproduction steps of the bug?
Too often, Agile testing becomes "ad-hoc"... just do it, do it fast, and get it done. I find that to be very dangerous. I do think that sometimes the amount of documentation that is done as part of the testing process is excessive. But I think throwing out the documentation part of the process entirely is just as dangerous.
One more thing: with software increasingly becoming integrated into all levels of society, there are legal implications of what happens when software goes wrong. In those legal implications, there is regulation... PCI, Sarbanes-Oxley, HIPPA, etc. are just some examples. With those becomes a necessity of being able to answer an audit. Some regulator comes by and says, "Show me how you are HIPPA compliant in your storage and access of patient records." If you have no documentation of how your tests verified that compliance... well... ....
- Marsha_R8 years agoModerator
We don't document our exploratory testing much. We have a high level test plan that's developed by the group and attached to the JIRA ticket that has the story/task. The tests in TestComplete have enough comments and log entries in them that a user familiar with the software would understand them. The ticket also contains links to a log of a successful test run and a link to the test in svn.
We don't expect that a brand new person would be able to pick up where an experienced person left off. I started here when everything was documented step by step in Excel sheets and it really wasn't that much help until I had actual user training on the software.
- tristaanogre8 years agoEsteemed Contributor
Agreed, Marsha_R... sometimes the amount of documentation we do as testers IS excessive. But it will vary from shop to shop and situation to situation. That's the thing about Agile is that it has a lot of adaptability. It's driven by the needs of the team doing the work... and each team make up is just a LITTLE bit different. :)
If I have enough documentation and artifacts created to get my friend from my house to my office with all the contingencies covered, then it's enough. Don't do more than what is needed... but certainly, I would not consider it a waste of time if it IS needed.Cheers!
- Marsha_R8 years agoModerator
tristaanogre Yes, exactly!
I should also mention that we are audited by customers several times a year and we are also ISO certified and our documentation serves to get us through those audits with flying colors.
Do the documentation that you *need* but always figure out if it's actually needed. Make sure it will be useful before you spend a lot of time and energy on docs that no one will ever see. Maybe you can get some of that time back to do actual testing!
- Ryan_Moran8 years agoValued Contributor
Wow that's a long reply to my example of driving. Some of that I agree with but the idea as a whole I do not.
Here are my reasons:
"well, SOMEONE needed to input all that data into the GPS system in the first place, right"
Yes they do, and working in IT I expect you to know how to use google maps or GPS.
Likewise I expect you to know how to use a Windows application and click on buttons without me telling you that.
" I get a call after half an hour, "Dude, I'm like TOTALLY lost."
Ok, well the example of driving doesn't really fit here does it? I can walk over to their PC and see exactly where they are or communicate with them in numerous ways. I guess if we wanted to get "techo-modern" in this example then I could have them share their location on google + and tell them exactly where to go. Isn't technology great?
"How far are you from being done?" what do you respond with?"
Give them the same bull**bleep** answer I would give them if I did actually know exactly where I was?
I mean really checking off items in your cases does NOT tell you how far before you're done.
Like you said...there may be an accident on the way. Right?
"with software increasingly becoming integrated into all levels of society, there are legal implications of what happens when software goes wrong"
If you're relying on test cases for this. Good luck with that?
I think what you are describing here is the necessity of use cases where legal requirements exist.
Anyhow it's always good to hear the other side of the argument so thanks for supplying that.
Related Content
Recent Discussions
- 18 hours ago
- 18 hours ago
- 5 days ago