Minimizing 'false' failures
Here is the problem:
When I run my tests one by one on my local host machine they pass.
When I run my tests by projects on local machine - approx 30% of tests fail.
When I run my tests by projects remotely (Jenkins, Hyper-v) - approx 60% of tests fail.
Reasons of failures are definitely not bugs and they are amazingly different: starting from TE not able to click a button because it is invisible (while the screenshot show the button is there) to skipping some steps (I suppose that by log messages that are not present in the log - but either one (step succeed) or another (step failed) message should be there is the step was executed).
I added a lot of refreshes of object browser, timeouts to my tests. But nothing helped.
Reruning tests against one and the same build gives different results in failures.
Rerunning them locally one by one makes them pass.
Any ideas on how to reduce these 'false' fails?
The problem is that I have only one Test Complete license and I need it on my host machine... Checking properties with Test Execute is not available.
Any ideas on how to overcome that?
- If you have floating license you install multiple places and use one at a time
- Create functions with more Log messages in-order to print the object properties in the log pane
How can 'Exist' or 'Visible' be different?
You could have a field that is hidden from view which would then Exist and not be Visible. It could be another timing issue as well, the field Exists but has not yet become Visible to the user.
Because of that, I opt for just testing for Visible. Exists then comes along for free. If it's never Visible, I don't really care if it Exists, and if it is Visible, then it has to already Exist.
Sometimes cutting down on the requirements helps the test be more stable.
[Community Heroes] are not employed by SmartBear Software but
are just volunteers who have some experience with the tools by SmartBear Software
and a desire to help others. Posts made by [Community Heroes]
may differ from the official policies of SmartBear Software and should be treated
as the own private opinion of their authors and under no circumstances as an
official answer from SmartBear Software.
The [Community Hero] signature is used with permission by SmartBear Software.
If it's never Visible, I don't really care if it Exists, and if it is Visible, then it has to already Exist.
Unfortunately, for me it is not as straightforward as you described.
Sometimes (let's say, to quit the app between tests) I need to perform different actions based on what exact screen is visible, so the logic is: if this one is visible, click that button, if another one - click another and so on. I cannot rely on the expected app state based on the last step since the test can fail somewhere in the middle and TE will not be able to quit the app and proceed with tests. I cannot Terminate the app, since I have other processes to quit, which happens with regular quit of my app via UI actions.
To implement that first I started with Visible only, but I got error kinda 'You are trying to get a property Visible of an element that does not exist'. I did not want that error, because for me it is not actually an error, I expect that element can be absent on screen. And I am OK with that. I perform specific verification of element visibility if needed.
So the best working decision I was able to implement to meet all my possible scenarios and avoid unnecessary errors: first check if an element Exists, and only after that (if it Exists) I am checking if an element is visible.