Implement scan option (or update namemapping) during test run

Hi Mathijs,

I really like this idea and we had been pondering how to correctly implement this mechanism.

How would TC know if an object is deleted only from one application screen and not from the entire application, for example?

MKozinets

good question Masha, thank you!

I think the tester has to have the end responsibility and TC would only suggest to update namemapping, based on the differences it finds during test run.

One way to display the differences would be to use marking on the places where TC would expect to find a control, but it is deleted. See the example here: Version 1 contains an input field "Bedrag", version 2 (which is released next build for example when using continious integration could be the next day). 

During test playback TC could compare both screens and link the info to the namemapping prior to testrun (sort of like a internal pre-run test validation). When it sees differences it should be made visible (like yellow marking), and TC would to suggestion to delete (or disable) the specific input field from both Namemapping AND Testscripts (both Keyword and Scripts).

Of course this is ideal situation for me, it should be discussed / evaluated by other TC users as well.

Hello, I think this is not something to do during tests runs : what would you do with results which differ from a build to another one? a missing object raises an error or not? you can't trust your test results in this way... and if a missing object doesn't raise an error during run, you may have a lot of false positives. However, this can be usefull as an analysing tool to run before the tests.

Montikore As discussed here, it all depends on the chosen approach of software development and releasing of updates. 

When using CI/ CD (Continues Integration/ Continues Development) the focus of test automation is on getting the bugs out of the software, driven by fast feedbacks on the testruns. Comparison to previous builds is not done, there is no time for that.

You can still trust your test results! But you have to keep in mind that testing is all done of the latest release and your findings have limited lifespan. Next day new software is delivered and you have to re-execute all the tests.

That's why you need to automate as much as possible (to keep up the speed), but use extensive scripting as little as possible.

Montikore you're way to deep in technical implementation.

Never ever is a tool going to decide if it's a bug. That's allways up to a human being to interpret.

TC should only recognize differences between latest deployment and the latest deployment -1 (second last deployment).

Specifically it should be able to:

- recognize deleted objects in the application: objects used in second last testrun but are removed from the latest version. It does not make sense to run tests on these since they would fail. (but that's logical because the applicaton has evolved). 

- recognize differences on the UI by marking these. So that tester can do his/hers work on the test automation framework. TC should be able to do that since already similar logic is built in to it (comparing screenshots). That would be ideal for assisting the tester in recognizing the differences in releases and keep the test framework up to date with developers/latest releases!

See this feature request also.

Montikore If the object is missing, then I do NOT want the test to pass. But I want TC to give me a signal that the application has changes / the specific object is deleted (or replaced somewhere else). In the latter case: I do not expect TC to give me the new location (since it's placed on a new position it is unknown area). although if that intelligence would be there it would definitely become a leading tool in te market. But I think we can only expect that to be recognized by tooling not within first couple of years.

TestComplete (or other tools as well) should be become more intelligent in handling application changes. Not saying that they should report changes as bugs, but changes should be recognized (and interpreted to some extend), and should be intelligently managed by the tool. In CI environments you (as the tester) dont have to time to inspect whole namemapping models to changes manually! By creating a snapshot of every new devilery TC should be able to "tell" the differences, just in an informative way. And then the tester decides (per change) if that needs updating the scripts and/or namemapping, or needs further (manual) investigation (which could eventually lead to a bug report).

As you can read in the feature requests I have created I want TC to become more intelligent in handling the changes. Of course it is not only the tool itself which handles application changes also the framework as designed by the tester should be capable of handling changes, but that is the task of the intelligence of the tester (his/hers skills).  Those two together would prevent reporting false positives as much as possible.

Basically, on a high level, I am saying this: TC (other tools have well) just do not have the capabilities to recognize and interpret application changes. These changes are becoming much more relevant in CI/ CD enviroments. I am looking for intelligence in tooling (the interpretation can be done by comparison on screenshots and models (namemapping in case of TC). Since TC has the data of all of these in it is potentially very much usable if intelligence is built into it!