GUI changes in regression test

We also experience a lot of GUI changes, but have been able to overcome with extensive framework to make tests in such a way that they do not rely on name mapping, object full names, etc.  In addition to that you want to design your tests in such a way that they perform a small set of tasks/tests, then return to the start of the application or a common point where the next set of tasks/tests would be capable of starting from. Between all of that error handling is key.

No test should produce an error that will stop all tests.



I know TestComplete is sold as a record and playback of tests without requiring any programming knowledge but I honestly do not see how anyone could produce quality automated tests without first building their own custom framework.



As Marsha said, you do want your tests to "fail" when there are program changes (that is the entire goal to detect changes between versions) but this should not stop subsequent tests from running.



As they say...Back to the drawing board :).

Name mapping is OK.



IF .... you are smart about it. If you just map objects and accept the standard mapping properties, you are likely to start hitting problems fairly quickly with anything but the most basic applications.



I use Project Variables to parameterise various properties common throughout the application (base part of URL, class names, etc etc). I also select my own properties and dependencies for object (the required child identifiers can be very useful) so I know they're more likely not to break. These can then easily be updated if things change. Also using extended find selectively, setting up your Alias part of the map in such a way that it closely resembles the screen structure (cut out all the parts of the site you don't really care about but are required for it to run) and I only map higher level parts of a site/app. The smaller, more dynamic, parts I tend to do the same as Ryan and use helper/finder functions during a run.



As long as you are careful and smart about how you use maps, and Aliases, they work fine. I very seldom have to make changes to mine and when I do they tend to be fairly minor.



You can also use them as tests if you map certain properties. For example, for a table I'll map the column headers and number of columns. Then all I need to do is check the object exists and I've verified that the columns are correct .... (and it'll flag up new ones appearing rather than just omissions).

It seems to me that there's a lack of communication between your test development group and the other programmers.  I agree that overnight tests need to be as robust as possible, but you should not be expected to guess at the new development changes that are happening.  Is there some way you can get access to the changes before they are actually released?  We work on the automated tests as the changes are being made, or soon after, and we are able to test in a development or test environment before the release to the outside world happens.  



This is still not foolproof because someone outside our group can change a program and we might not hear about it, but we find out about it pretty quickly when our test crashes.



And really, a test crashing because the program changed is a good thing, isn't it?  I can write you a test that will always pass and you won't ever have trouble with it at night but it will tell you absolutely nothing about the quality of your code. 

Not sure how much it helps, but I pretty much don't use any of the built in error handling. (I switch it all off in the project setting and also switch off the error stopping in my code through script.)



I build a LOT of error handling into my scripts. And I mean .... A LOT. An average script function for me can easily be 70% validation & error handling and only 30% actual test.



This handles validation of the user supplied data as well as validation and error handling around the controls used in the test. My test units are all user defined, modular and run and controlled through a central framework. So if one test (or test pack) fails, the next one can still run. Assuming the test flow is designed and split up in a sensible way of course. If you try and run one giant test for everything, that would be more of a problem.



Reason I put so much effort into it is, for me, one of the key things for any automated test is that it should never crash. (Which is probably impossible. But I try and get it to the point where it's very hard to crash at least.)



An overnight run against our application (kicked off by the nightly build process) can take 6-8 hours so automated tests are useless if it crashes 20 minutes into the run. I think I must have it about right as it hardly ever crashes. It may spew out a lot of errors if someone introduces a change which breaks things, but it won't crash - and you'll get meaningful error messages in the log files. (My log files are also custom built.)



But then, I've built my entire framework from the ground up on this basis. I suspect retro-fitting it if I hadn't might be trickier.

"And really, a test crashing because the program changed is a good thing, isn't it?"



Personally, I 100% disagree with this. Failing - yes. Crashing - no.



As soon as automated tests start crashing, that's when management (amongst others) start to question the validity of them. Such has been my experience anyway.



I'd much rather the test kept running and my logs gave me good information about what didn't work. If it's a large fundamental change, the logs may contain nothing but failures. But more often than not, it's not and only part of the suite fails. The rest will carry on and run as normal.



Error handling, recovery, error handling, recovery ..... it's like a mantra for building a new module! :)



And as Marsha says, it's handy if your dev people also let you know about potential breaking changes as early as possible. Some are better than others at this ....

Yes Nishnatha,

I search for every object by process, form, and property ("name", "Caption","text", etc.).

Most forms will not have controls with identical names, captions, or text and when they do (though I try hard to avoid it) I will search by fullname or other properties.



All of my data driven tests are run through a set of common routines or the "framework" if you will.



It takes a while to build something like this initially, but the maintenance involved at this point is next to nothing.

I'll speak specifically for my situation testing a Web based front end.  I worked closely with the development team to ensure they supplied a unique identifier ( in my case IdStr) for each UI element, this allows me to avoid object mapping  and dynamically interact with the element of interest.  As others have mentioned, building in error handling is essential no matter what the object access method is.



The bottom line is, if automation is important to your project team and the problem your having can be solved as I described, this shouldn't be a tough issue to solve.  The switch from object mapping to finding a Unique ID may take some time but will pay off in the long run. I would be happy to provide more details if need be.

 

 

Hi guys,

Thanks for guidence and sharing experince.

Ryan, what do you mean by "do not rely on name mapping" ? 

do you seach object by property at runtime?

"As Marsha said, you do want your tests to "fail" when there are program changes"



No. Marsha said it was a good thing if a test CRASHED due to a change! I said it was a good thing if it FAILED. Big difference.



I'm with Ryan. Use much the same methodology as him. Framework. Small test units/functions. Super-robust error handling and recovery.



I know Ryan uses helper functions to dynamically find and return objects during a run rather than name maps. I use a mixture of both - depending how dynamic a site/application is or isn't. Even if using mapped objects, new ones, changed ones or missing ones should still not cause a crash. Doesn't in my tests. Due to robust error handling and recovery ...

As per discussion what I gathered are

1)      Robust error handling are essential (I like Colin’s 70%-30% concept) 

2)      Tester –developer collaboration is very important

3)      Name mapping is error prone , better use finding object by ID

 

I’m confident on working on item 1 and 2 would like to know more on item 3

Currently my frame- work based on Name- mapping and feel it’s time to switch on to more reliable method.  Could you please share any information that would help.

I’m using jScript.