cancel
Showing results for 
Search instead for 
Did you mean: 

Discussion : Do you need to organize Test Left tests for execution ?

Community Hero

Discussion : Do you need to organize Test Left tests for execution ?

Hi,

 

I am trying to consider when and how TestLeft can be used in the testing process and have the concern that I would appreciate to hear your thoughts/experience about.

 

So:

-- As a runtime engine TestLeft uses unit-testing engine (MSTest or NUnit).

-- Unit testing is based on the concept that a small atomic action is verified and this action either works in its entirety without *any* issue or does have an issue. And if the action contains the issue it (action) is considered as *completely* not working. Which means that there is no need to continue with the action as soon as some issue is found (assertion approach).

-- The above paragraph quite logically results in approach that unit tests may be executed completely isolated, in a clean environment and with the outer world replaced with mocks.

-- Because unit test verifies just an atomic action in isolation, there is no need to preserve the result of this action.

 

The result of the above is that unit tests do not depend one from another and can be executed in random order.

 

On the contrary:

-- TestComplete and TestExecute are tools for the functional tests.

-- Functional tests are based on the concept that the test verifies if some task can be performed by the end-user.

-- Usually, the task consists from more than one action and the result of the previous action is used as an input for the subsequent action. Thus, the actions order matters and action's result must be preserved.

-- For the functional test it is acceptable that the certain action fails. The failed action must be reported as an issue but the failed action does not necessarily means that the task cannot be executed and usually is not a reason to stop test execution. (For example, the 'Save' command from the toolbar may not work, but if the 'Save' command from the main menu works the test's execution can proceed.)

-- Tests with the failed actions must be reported as 'problematic' but this is a matter of the additional investigation of whether or not the tested functionality (the task attempted to be performed) should be considered as unusable or not (i.e. usable with some known issues).

-- Tasks tested with the functional tests can be complex and composed from several time-consuming actions that produce complex data. This makes it impractical to mock creation of these data in the setup and clean them up in the cleanup sections of the action test.

 

The result of the above is that functional tests and actions within those tests are order-dependent. Besides that, different functional tests might be required to be executed for different test run configurations.

 

The above challenge in TestComplete usually was solved by creating a proper test items tree and enabling/executing its different branches which cannot be done for TestLeft tests.

 

Do you see the same challenge for your projects? Do you think that test items tree is something that is worth to be asked for for TestLeft or attributes for test methods work fine for you? Do you see some wrong assumptions in my points above? Any other notes or comments?

Regards,
Alex
[Community Expert Group]
____
[Community Expert Group] members are not employed by SmartBear Software but
are just volunteers who have some experience with the tools by SmartBear Software
and a desire to help others. Postings made by [Community Expert Group] members
may differ from the official policies of SmartBear Software and should be treated
as the own private opinion of their authors and under no circumstances as an
official answer from SmartBear Software.
[Community Expert Group] signature is used with permission by SmartBear Software.
http://smartbear.com/forums/f83/t86934/community-experts/
================================
2 REPLIES 2
Super Contributor

Re: Discussion : Do you need to organize Test Left tests for execution ?


@AlexKaras wrote:

...

Do you see the same challenge for your projects? Do you think that test items tree is something that is worth to be asked for for TestLeft or attributes for test methods work fine for you? Do you see some wrong assumptions in my points above? Any other notes or comments?


I haven't tried TestLeft yet but would assume the ability to run individual steps (items) and group steps should exist.

Community Hero

Re: Discussion : Do you need to organize Test Left tests for execution ?

@william_roe:

Hi William,

 

Actually, at the moment TestLeft provides nothing specific in the area of tests organization and is 100% based here on the functionality of MSTest. So TestLeft's behavior here is exactly the same as for CodedUI and MSTest itself (see its command line reference to get an idea about possible options) and pretty much the same as for Selenium, NUnit and any other unit-testing tool.

 

The good news for me is that you did not object against my initial clauses. Smiley Happy Some related discussion is here: https://community.smartbear.com/t5/Desktop-Testing/How-to-deal-with-suit-and-sub-suit-test-items/m-p...

 

Regards,
Alex
[Community Expert Group]
____
[Community Expert Group] members are not employed by SmartBear Software but
are just volunteers who have some experience with the tools by SmartBear Software
and a desire to help others. Postings made by [Community Expert Group] members
may differ from the official policies of SmartBear Software and should be treated
as the own private opinion of their authors and under no circumstances as an
official answer from SmartBear Software.
[Community Expert Group] signature is used with permission by SmartBear Software.
http://smartbear.com/forums/f83/t86934/community-experts/
================================