TestComplete causing Feature Files and Scenarios linked CucumberStudio to be put in Trash
I've been battling this issue for over a year and am interested if others have run into it.
This would be for users of TestComplete and syncing/binding with CucumberStudio.
The issue is that fairly regularly, a couple of times a month, large numbers of Scenarios and Feature Files in CucumberStudio will end up in the Trash, counting down the days before being permanently deleted.
The identifying name for the person who trashed all these items is one of the automation engineers who was running tests in TestComplete that is linked to the project in CucumberStudio. It can be anywhere from a dozen to over one hundred Scenarios and Feature Files that show up in the Trash. There is no warning, dialog, or announcement that we have seen. The only way to mitigate lose of this large amount of work is to regularly look at the CucumberStudio Trash and question any large scale collection of items with QE Automation engineer names attached to it.
We have about a dozen automation engineers working on almost as many separate teams, all on different aspects of the same product. This means that all Test Cases (Scenarios), regression tests, etc. need to be available to all teams.
Manual Testers write up their tests in Gherkin structure in CucumberStudio, and the automation engineers automate the ones identified for automation.
Since CucumberStudio is cloud based, there is only one version of the Scenarios, however that is managed by having working folders and tags.
The automation code though can be better managed by different groups using Git Branches. We work in a Microsoft world, so it is ADO Git.
So far we have not been able to create a reproducible series of steps to create this issue. We have observed it over the year at least 30 times and seen general patterns.
In a defensive move we DO NOT make changes to Scenarios in TestComplete, and DO NOT need sync from TestComplete to CucumberStudio.
As mitigation for the problem, we ONLY make changes to Scenarios in CucumberStudio and rely on the sync to TestComplete to bring those changes into TestComplete.
We have also unchecked in TestComplete the "Synchronize feature files on project opening" and the "Synchronize feature files before test run" checkboxes in the TestComplete Project's CucumberStudio binding settings. We have kept the "Save BDD test results to CucumberStudio" checked, and feel a bit suspicious about that.
Even with all of these steps, and with automation engineers fully aware of this issue, and trying to guard against it, we still have times when large amounts of CucumberStudio Feature Files and Scenarios get put in the Trash behind the scenes.
The common elements are:
- 1) Automation Engineer is working in TestComplete that is linked to CucumberStudio. Probably running tests, at least small ones as they develop automation code. Note that they do a manual "Update from CucumberStudio" at least at the beginning of their day, if not other times.
- 2) Other QE teams are making changes in CucumberStudio in their teams as needed.
Since we have not found a reproducible way to make the issue show up, that's all I can say for sure.
I'd love to get a solution of course, and I have a ticket in with SmartBear to investigate how this might be happening and how to prevent it.
If any of you have this issue, I'd like to compare notes.
When asking for help from SmartBear on this, they have said I'm the first person to have mentioned this. Considering how many times it has happened and how otherwise benign or setup is, I find it hard to believe that we are so lucky to be the only ones observing this.
