Discussion : Do you need to organize Test Left tests for execution ?

Hi,

 

I am trying to consider when and how TestLeft can be used in the testing process and have the concern that I would appreciate to hear your thoughts/experience about.

 

So:

-- As a runtime engine TestLeft uses unit-testing engine (MSTest or NUnit).

-- Unit testing is based on the concept that a small atomic action is verified and this action either works in its entirety without *any* issue or does have an issue. And if the action contains the issue it (action) is considered as *completely* not working. Which means that there is no need to continue with the action as soon as some issue is found (assertion approach).

-- The above paragraph quite logically results in approach that unit tests may be executed completely isolated, in a clean environment and with the outer world replaced with mocks.

-- Because unit test verifies just an atomic action in isolation, there is no need to preserve the result of this action.

 

The result of the above is that unit tests do not depend one from another and can be executed in random order.

 

On the contrary:

-- TestComplete and TestExecute are tools for the functional tests.

-- Functional tests are based on the concept that the test verifies if some task can be performed by the end-user.

-- Usually, the task consists from more than one action and the result of the previous action is used as an input for the subsequent action. Thus, the actions order matters and action's result must be preserved.

-- For the functional test it is acceptable that the certain action fails. The failed action must be reported as an issue but the failed action does not necessarily means that the task cannot be executed and usually is not a reason to stop test execution. (For example, the 'Save' command from the toolbar may not work, but if the 'Save' command from the main menu works the test's execution can proceed.)

-- Tests with the failed actions must be reported as 'problematic' but this is a matter of the additional investigation of whether or not the tested functionality (the task attempted to be performed) should be considered as unusable or not (i.e. usable with some known issues).

-- Tasks tested with the functional tests can be complex and composed from several time-consuming actions that produce complex data. This makes it impractical to mock creation of these data in the setup and clean them up in the cleanup sections of the action test.

 

The result of the above is that functional tests and actions within those tests are order-dependent. Besides that, different functional tests might be required to be executed for different test run configurations.

 

The above challenge in TestComplete usually was solved by creating a proper test items tree and enabling/executing its different branches which cannot be done for TestLeft tests.

 

Do you see the same challenge for your projects? Do you think that test items tree is something that is worth to be asked for for TestLeft or attributes for test methods work fine for you? Do you see some wrong assumptions in my points above? Any other notes or comments?