Agile / Scrum / DevOps/ Waterfall... : (Dis)likes, Experiences, pitfalls, best practices!?

I find the same... automation's best ROI is twofold, IMO

1) Regression.  You've already manually tested it once when it was developed.  If you have to re-do, manually, every test you've ever done, you'll NEVER get anything new tested because, each sprint/iteration you'll have to test more and more and more.  Adding functional regression automation to the development process is a HUGE help as it frees up the testers from having to do the manual regression work.  Have the regression tests running constantly... at least nightly... so that, as SOON as something breaks, you'll have a notification and it can be addressed immediately.  Be smart about your regression tests, also.  Sometimes something done manually never needs to be repeated.  Or, you are testing a module that is reused multiple times in the application.  If you've fully tested it in one place, you don't need to fully test it in another.  A simple "does it work" will suffice for each subsequent instance of that module.  And so on.  Also, there are a LOT of tests done in manual testing that are "edge" cases... things that a customer MIGHT run into.  Important test cases, for sure (I use the rule, "If I can come up with the scenario, odds are the customer will do the same thing") but regression is about risk.  If it's a relatively low risk scenario and you have other high risk things still to do, put it in a backlog and get to it "when you have the time."

2) Need for high accuracy and repeatability.  While this plays well into regression, one of the things that I ran into in the past was an intensely complicated workflow/state engine built into the application.  There was a need for high accuracy and rapid repeatability of the test.  If I found a bug, I would need to be able to rerun the test as many times as I needed with the same level of accuracy.  In my example, I wrote an automated test that did this for me that I ran on demand.  While it took some time investment up front to do so, it saved me a LOT of time each time I needed to run the test.  The automation was later integrated into the regression suite but, as an assistance to my manual testing, it was invaluable.

There was an article I read a while back that addressed the concept of "artifacts" created during the testing process.  This includes documented test plans, configuration scripts, and automated test code and scenarios.  Creating an automated test creates an artifact that needs to be stored, maintained, and referenced.  Does the overall process benefit from having such an artifact created and is there an ROI for the continued maintenance of the artifact?  If the answer to both those is yes, then the artifact should be created.  If the answer to either one is no, then don't bother.  

Ryan_MoranThanks for your reaction.

I think the credo "context is everything"also counts here.. It all depends on the potential risks involved in the system you are testing (whether it's manual or automatic: doesn't matter in this case).

Example: the (sub)systems which run (fly) an airplane will need the full bunch of documentation, and the 'complete package' of testscripts, worked out in detail. No compromises. Why? Because human lifes are potential at stake.

When your testing efforts are focused on a "regular" IT system in "typical" business, most of the times human lifes are not at stake, and therefore risks can be lower (as an example). Mix these context aspects with (lack of) available to time, resources, etc, and you have to make the decision if complete 100% complete and accurate testcripts need to be worked out in 100% detail... And this again relates to agile way of work: a lot of companies choose to work 'agile' because of some of the benefits of this software development method: it results in fast deliverables, with a high degree of interaction between end users and developers...

But, at the same time, Agile has it's cons as well.. One of them being lack of robustness, and this is caused by lacking of time to work out all the documentation and all the testscripts you are referring to as well...

Putting it back togeter in the example: I am 100% there is no airplane manufacturer in the world (whether it's the plane itself, or any software systems running in/on it), which even starts to think of working Agile for their development/manufacturing processes... the risks are just too high! To rely on the 'old, classical', documentation, decent testing, in each and every (sub)step of their development/manufacturing processess...

If the company you work for has choosen to work "Agile" you can conclude for yourself that quality assurance is not the major point of focus, in whatever their it systems need to do. Being a tester in an Agile culture has its challenges, you can figure it out yourself.....

Actually, I've never heard the argument for DDT as "do more cases, catch more bugs".  Instead, a good DDT structured framework is not necessarily based upon generating a lot of test cases but more on separating the code that executes against the AUT from the data that the code uses.  For example, I want to enter data into a form on screen.  The code for that form is pretty static and basic, no matter what the data is.  So, I abstract the code for  the form to make all data entry fields and selections parameterized variables that are sourced in data. 

Now, if I want to run a test case against that form, I simply create a data record for that form and boom.  One test case is done.  Want another test case?  Don't need to touch the code at all, just add another data record.

Now... if done well, this isn't just looping through the form and populating data.  That code that "drives" that form is called from other framework structures. So, I'm testing a use case, for example, that enters data in that form and then does some sort of operation based upon what was entered.  So, my data includes not just the data on the form but other steps and processes, button clicks, validations, etc., that drive the whole use case.

Build your framework properly, and when it comes to adding another test case, the amount of actual code you need to write is reduced and most of your test case construction is simply adding data records.  I had a framework at that company that did POS applications where I had code that applied payments, selected tickets, verified transaction journals, etc.... all modularized code.  Everytime I wanted to add a new test case, I didn't need to touch a line of code, I simply went into my database, added a few data records, and now my project has expanded by a test case.

Sure, the end result is more test cases... but the true aim of DDT is to speed up the ability to add a test case and even make it so that a non-programmer tester can add a new test case by the simple task of data entry.