mgroen2's avatar
mgroen2
Super Contributor
9 years ago
Status:
Accepted for Discussion

Implement scan option (or update namemapping) during test run

As stated in this feature request, we work agile and the tester gets new applications versions (almost) on a daily basis.

The tester does not know which objects have been changed/deleted. And does not know which controls (object) are new.

 

This feature request is to enhance TestComplete with a feature:

It is to implement a 'scan' (or update namemapping feature) to scan the updated application so that the namemapping gets updated according to the updated application:

new objects are placed in the namemapping;

changes done in existing objects get updated in namemapping;

deleted objects are removed from namemapping.

 

After that the tester runs the script and it runs with no errors (because all changes are reflected in the namemapping prior to the test run), and also (but this is really a dream coming true), TestComplete marks the new but-not-yet-tested controls (in the Visualizer images for example). So that the tester can update the testscripts to new objects get tested as well. This is something like the current "Map object from screen" function, but with the difference that it scans the whole application (or screen based), and reflects the Name mapping repository automatically as well.

 

I realize this is a rather big feature request but I think allows Smartbear to become real market leader in test automation!!

 

  • mgroen2's avatar
    mgroen2
    Super Contributor

    Montikore Happy I got you convinced ;)

     

    technically you are right: it's more of a "scan on changes" function, or an analysis step. Those could be implemented in the run phase, but I agree with you it may be better to separate them, and call them "Scan" (analysis) and Run as in separate stages.

     

     Actually for me the primary goal now is that this functionality is implemented (or set on a roadmap), in the first place!.

  • (Much of this discussed in the forum thread but to summarise my thoughts ....)

     

    Conceptually, I kind of like the sound of a "scan" feature.

     

    I would NEVER want an automation tool to start updating object maps etc without me checking the changes it wants to make first.

     

    So having it update/guess missing object, on the fly, during a run, I would NEVER risk using.

     

    Which brings me back to the main blocker from my perspective ....

     

    How does it do a "scan" without doing a run? Many object will only be loaded and/or triggered as a result of other actions (which will happen during a test). So you pretty much have to do a test run in order to do the "scan" anyway?

     

    I usually do this anyway. First run finds the broken bits and failures. I go through these, find the areas that failures are dues to changes (ie - not actual failed tests) and the ones that are due to bugs (not always obvious until you update for changes - these come out more often in run 2) and the subsequent run then finds the actual failures.

     

    I just don't see how you do a scan without a run?

     

    Except on really simple applications with very little complexity or variation in behaviour.

     

    Having it scan and update Object/Alias names in script when you change the mapped object (I can avoid this being a problem most of the time already my using Aliases well/properly) would be handy though.

  • mgroen2's avatar
    mgroen2
    Super Contributor

    Colin_McCrae :

     

    Conceptually, I kind of like the sound of a "scan" feature.

    I think it would best be called "scan for changes" (or something like that). (minor detail).

     

    I would NEVER want an automation tool to start updating object maps etc without me checking the changes it wants to make first.

    That's ok, and I think it should be set configurable as how TC should update the model when changes are detected. I think the best setting depends on how testautomation has been implemented in the organization and how fast new software is being delivered , which results in how much time you have to create "your own" model (namemapping). In the past I have worked in non-agile environments and I was able to spend lots of time investigating the namemapping model, improve it, "play" with it, etc. It was nice to do but very time consuming. And, when you (as a tester) received a new version you would do all the work again because the namemapping failed. Lots of time were "eaten up" by "debugging" the namemapping and/or testscripts.

     

    Now, I work in Agile environment and one of the biggest challenges for testers is keeping up with the speed. Everyday new delivery you cant work on drastic namemapping changes. Then I started to see: this can be done by improving the tool. Software/computers are extreme good (in theory) in comparing situations. If you run the tool every day on each released version it should be able to "see" the differences, and help the tester with it in some way.

     

    By letting TestComplete do "more under the hood", releases more time for the tester to work on validation on a higher level of abstraction: like functional end-to-end testing, data driven testing validation, or other specific technical issues like API testing etc. 

     

    Of course I won't expect this to be working flawlessly but it should be created with an "undo" for each change TC commits automatically or after confirmation by the tester.

     

    How does it do a "scan" without doing a run? Many object will only be loaded and/or triggered as a result of other actions (which will happen during a test). So you pretty much have to do a test run in order to do the "scan" anyway?

    You are right, it should do a "validation" or a "pre run scan", whatever you call it. In first instance I thought it would be implemented  the current Run command, but things have evolved and I think it is better to implement seperate "scan" function (which in facts functions somewhat like a "run" if you see it that way).

     

    I just don't see how you do a scan without a run?

    I agree with you as in it can't be scanned without running the testscripts.

     

     

  • Montikore's avatar
    Montikore
    Occasional Contributor
    Regarding the scan without a run question, there is a time issue here. Even if time is not the most important point in automation, running a whole campaign can take hours. If you run all your tests, you have then to wait hours before updating your tests and run them again. With a scan option, you may declare the different screens of your application, with a path to it. Even with a lot a different screens, this doesn't seem to be lot of work as the path shouldn't evolve so often. The scan can then be performed in a few minutes for basic situations. As the goal here is to perform the tests asap, with reliable results, running the test campaign twice is not an option imo...
  • mgroen2

     

    I suppose I kind of already do a scan then?

     

    I have my own framework. Results files. Log files. And I write/update to test suites on TFS. Everything is keyword and data driven to abstract the test code (done by few) from the tests (done by many).

     

    So when I do a run, the first one is always with TFS links switched off. Things get updated/fixed to remove false positives. Second run (or third or more depending on the scale of the changes) is done with TFS links switched on and regarded as the final run. (Against that build version)

     

    We also run "Agile" (or rather, our version of it - as others have said, interpretation of what agile is vary wildly!) but expecting test code to keep up with build code has never been expected, nor asked for. If anything, we tend to use TC for regression tests. So the suitable manual tests are extracted and converted and the regression pack grows.

     

    The application I do most work on at the moment has a large team of devs. Uses several different languages and compilers, and changes quite frequently. There is no way we could keep the test code build up to date, on the same day, as all new application code. It simply isn't feasible.

     

    Based on all the above, and the fact that I wrap all my stuff in enough error handling that a missing object will not crash the test suite (and it's modular so subsequent tests will recover and run), I get the "object not found" messages in my logs and act on them accordingly. At the end of the initial run.

     

    The way I use the name map (high level), Alias Map (condensed high level) and Helper Functions (low level), object updates seldom take me long anyway. 99% of the time I only have to update the name map. The alias doesn't change. That only happens with VERY large structural changes. Which don't happen often.

     

    Montikore

     

    You're right about time. Some runs, especially large end to end ones, can take hours.

     

    But again, with complexity tends to come dependency. And that is absolutely the case with the stuff I work with. No way you could "simply declare a path" to parts of it. Some parts simply will not be loaded or appear until you do Step A, then apply Action B, then merge it with Step C, and finally press Button D.

     

    I can guarantee you 100%, there is no way you could trigger all the parts of my application without a whole bunch of pre-work. It's a desktop application, not a website. You can't simply jump to a URL in a desktop app ....

  • Montikore's avatar
    Montikore
    Occasional Contributor
    Colin_McCrae Indeed, applications can have a high level of complexity, that's why i said " in a few minutes for basic situations". When i'm speaking about a path to a sreen, i don't think about an URL, but more about something like a list of actions, without all the controls you have in a test. Even in complex situation like you mention, it can be really faster.
  • By the time you set up all that, you might as well just run the test!

  • mgroen2's avatar
    mgroen2
    Super Contributor

    MontikoreColin_McCrae Thank you both for the useful comments.

     

    I want to emphazise that the world around us tester is changing rapidly and agile and even CI/ CD affects the way we can do our work (as testers), whether we like it or not. (of course I would like to have much more time to build a profound testing framework including greate namemapping features and all), do some great engineering with it, etc... but can't do it in current situations.

     

    Please don't expect that this feature request will solve all the problems in Agile/CI/CD driven projects.

     

    No project is the same. So much variables, so much influencing factors... its's all about complexity of the AUT, chosen development approach(es), (internal) commitments, people, etc, etc, etc....

     

    It's just that I think TestComplete should become better at supporting CI/CD driven projects, and I think it has real good potential considering the changes in software/development world in the (near) future.

     

    I know that Tricentis (Tosca) is working on similar scan / update "namemapping model" on their to-be-released updated version of Tosca as well. Tosca is not a perfect product nor Tricentis is a perfect company far from that but by evaluating their product I saw something interesting, and  I translated this to TestComplete and I saw something was missing.