Implement scan option (or update namemapping) during test run

Montikore Happy I got you convinced ;)

technically you are right: it's more of a "scan on changes" function, or an analysis step. Those could be implemented in the run phase, but I agree with you it may be better to separate them, and call them "Scan" (analysis) and Run as in separate stages.

 Actually for me the primary goal now is that this functionality is implemented (or set on a roadmap), in the first place!.

Happy I got you convinced :smileywink:

(Much of this discussed in the forum thread but to summarise my thoughts ....)

Conceptually, I kind of like the sound of a "scan" feature.

I would NEVER want an automation tool to start updating object maps etc without me checking the changes it wants to make first.

So having it update/guess missing object, on the fly, during a run, I would NEVER risk using.

Which brings me back to the main blocker from my perspective ....

How does it do a "scan" without doing a run? Many object will only be loaded and/or triggered as a result of other actions (which will happen during a test). So you pretty much have to do a test run in order to do the "scan" anyway?

I usually do this anyway. First run finds the broken bits and failures. I go through these, find the areas that failures are dues to changes (ie - not actual failed tests) and the ones that are due to bugs (not always obvious until you update for changes - these come out more often in run 2) and the subsequent run then finds the actual failures.

I just don't see how you do a scan without a run?

Except on really simple applications with very little complexity or variation in behaviour.

Having it scan and update Object/Alias names in script when you change the mapped object (I can avoid this being a problem most of the time already my using Aliases well/properly) would be handy though.

Colin_McCrae :

Conceptually, I kind of like the sound of a "scan" feature.

I think it would best be called "scan for changes" (or something like that). (minor detail).

I would NEVER want an automation tool to start updating object maps etc without me checking the changes it wants to make first.

That's ok, and I think it should be set configurable as how TC should update the model when changes are detected. I think the best setting depends on how testautomation has been implemented in the organization and how fast new software is being delivered , which results in how much time you have to create "your own" model (namemapping). In the past I have worked in non-agile environments and I was able to spend lots of time investigating the namemapping model, improve it, "play" with it, etc. It was nice to do but very time consuming. And, when you (as a tester) received a new version you would do all the work again because the namemapping failed. Lots of time were "eaten up" by "debugging" the namemapping and/or testscripts.

Now, I work in Agile environment and one of the biggest challenges for testers is keeping up with the speed. Everyday new delivery you cant work on drastic namemapping changes. Then I started to see: this can be done by improving the tool. Software/computers are extreme good (in theory) in comparing situations. If you run the tool every day on each released version it should be able to "see" the differences, and help the tester with it in some way.

By letting TestComplete do "more under the hood", releases more time for the tester to work on validation on a higher level of abstraction: like functional end-to-end testing, data driven testing validation, or other specific technical issues like API testing etc. 

Of course I won't expect this to be working flawlessly but it should be created with an "undo" for each change TC commits automatically or after confirmation by the tester.

How does it do a "scan" without doing a run? Many object will only be loaded and/or triggered as a result of other actions (which will happen during a test). So you pretty much have to do a test run in order to do the "scan" anyway?

You are right, it should do a "validation" or a "pre run scan", whatever you call it. In first instance I thought it would be implemented  the current Run command, but things have evolved and I think it is better to implement seperate "scan" function (which in facts functions somewhat like a "run" if you see it that way).

I just don't see how you do a scan without a run?

I agree with you as in it can't be scanned without running the testscripts.

mgroen2

I suppose I kind of already do a scan then?

I have my own framework. Results files. Log files. And I write/update to test suites on TFS. Everything is keyword and data driven to abstract the test code (done by few) from the tests (done by many).

So when I do a run, the first one is always with TFS links switched off. Things get updated/fixed to remove false positives. Second run (or third or more depending on the scale of the changes) is done with TFS links switched on and regarded as the final run. (Against that build version)

We also run "Agile" (or rather, our version of it - as others have said, interpretation of what agile is vary wildly!) but expecting test code to keep up with build code has never been expected, nor asked for. If anything, we tend to use TC for regression tests. So the suitable manual tests are extracted and converted and the regression pack grows.

The application I do most work on at the moment has a large team of devs. Uses several different languages and compilers, and changes quite frequently. There is no way we could keep the test code build up to date, on the same day, as all new application code. It simply isn't feasible.

Based on all the above, and the fact that I wrap all my stuff in enough error handling that a missing object will not crash the test suite (and it's modular so subsequent tests will recover and run), I get the "object not found" messages in my logs and act on them accordingly. At the end of the initial run.

The way I use the name map (high level), Alias Map (condensed high level) and Helper Functions (low level), object updates seldom take me long anyway. 99% of the time I only have to update the name map. The alias doesn't change. That only happens with VERY large structural changes. Which don't happen often.

Montikore

You're right about time. Some runs, especially large end to end ones, can take hours.

But again, with complexity tends to come dependency. And that is absolutely the case with the stuff I work with. No way you could "simply declare a path" to parts of it. Some parts simply will not be loaded or appear until you do Step A, then apply Action B, then merge it with Step C, and finally press Button D.

I can guarantee you 100%, there is no way you could trigger all the parts of my application without a whole bunch of pre-work. It's a desktop application, not a website. You can't simply jump to a URL in a desktop app ....

By the time you set up all that, you might as well just run the test!

Montikore, Colin_McCrae Thank you both for the useful comments.

I want to emphazise that the world around us tester is changing rapidly and agile and even CI/ CD affects the way we can do our work (as testers), whether we like it or not. (of course I would like to have much more time to build a profound testing framework including greate namemapping features and all), do some great engineering with it, etc... but can't do it in current situations.

Please don't expect that this feature request will solve all the problems in Agile/CI/CD driven projects.

No project is the same. So much variables, so much influencing factors... its's all about complexity of the AUT, chosen development approach(es), (internal) commitments, people, etc, etc, etc....

It's just that I think TestComplete should become better at supporting CI/CD driven projects, and I think it has real good potential considering the changes in software/development world in the (near) future.

I know that Tricentis (Tosca) is working on similar scan / update "namemapping model" on their to-be-released updated version of Tosca as well. Tosca is not a perfect product nor Tricentis is a perfect company far from that but by evaluating their product I saw something interesting, and  I translated this to TestComplete and I saw something was missing.