TestComplete Evaluation?

Hi,

1. All notes below assume clean, out-of-the-box products state.

2. The text below assumes following definitions:

  -- Unit testing checks if some functionality (operation) works or not. Checked functionality can either function or not and has no other state. For example, sun-roof in your car either can be opened or cannot;

  -- Functional testing checks if some task can be done or not. Task consists of one or more operations and it is possible that some operations may fail but not block the task itself. For example, sun-roof in your car may fail to open. This is the problem and it must be reported. But failing to open sun-roof does not block you from driving and you still can check if your car can get from point A to point B.

With the above in mind, the major architectural difference between TestComplete (QTP, Ranorex, ...) and Selenium (and any other unit-testing tool like Appium, MS Test, etc.) is that Selenium has assertion-based architecture. This means that the only possible visible way in Selenium to report a problem in a test is to raise an unhandled exception. Unhandled exception means that the test will stop. The above behavior means the following:

-- It is not possible in Selenium to clearly report a problem and continue to run the test. While this is fine for unit testing, this is not the best behavior for functional testing where, on the one hand, you must clearly report all problems, but on the other hand you must check if it is possible to get the final result even if some intermediate actions cannot be done in full. So, with Selenium, you either report the first problem and stop or you must implement some logging and analyze this log instead of using the test result reported by Selenium.

Other Selenium peculiarities:

-- Selenium does not have built-in log and you must implement it yourself;

-- Selenium cannot connect to (reuse) the already running browser. Among other inconveniences, this means that it is not possible (e.g. for debugging purposes) to start browser, put your tested application into required state, launch Selenium test, connect to the running application and proceed with your test or debugging. Instead, you must either wait until your test reaches the point that you need or create a reasonable amount of additional test code (that, actually, is not needed);

-- Selenium can work only with html content and cannot handle neither objects provided by the browser (e.g. file open/save/download dialogs) nor the content like Flash;

-- Selenium provides no means to find out with what exact control the code works at the moment. I.e. in your code you have just some programmatic object and it is not possible either to highlight it on the screen or review its identification properties. Quite often, during debugging, this means that you can only assume that you work with correct object, but it is not possible to prove or double-check this;

-- Selenium does not perform actual actions against web controls but emulates them via internal API calls. On the one hand, this has some advantage as it makes it possible for your test code to run if the screen gets locked with screensaver, on the other hand this means that the test does not behave exactly as a human user do;

-- It is not a trivial thing in Selenium to wait for a control during some timeout, check whether the control appeared or not and proceed accordingly without throwing an exception;

-- It is not a trivial thing in Selenium to work with control that is either not in the view or is hidden;

-- Selenium does not provide any infrastructure and you will have to implement yourself the support for, say, data driven testing, data exchange between tests, data personalization for different test environments, etc.

There is no real problem with everything mentioned above in TestComplete.

Obviously, TestComplete is not a silver bullet and quite often Selenium is not the worst option for the unit-testing, but it is my opinion that for the functional testing, TestComplete is much better option.