Forum Discussion

gregkennedy001's avatar
gregkennedy001
New Contributor
2 months ago

Test Execution Limits in Zephyr Scale

We've recently moved to Zephyr Scale and whilst the features and functionality are as expected, what is a surprise is the low levels of execution limits. So low in fact that we're almost thinking we've bought the wrong product or our use case is not what ZS is intended for.

We are a small team (<10) who run all our CI/CD workflows through GitHub Actions (however, the same issue would occur with other orchestration tools such as Jenkins/Bamboo). 

Per CI/CD best practices, we run our unit tests (via Pytest) and integration/acceptance tests (via ZS) on every commit to our GitHub repos (i.e. multiple times per day). Multiplied across even a small development team, this will quickly reach hundreds, if not thousands of test runs per month.

We're now aware, after purchase, of the test execution limits. My question is, the limits so low (100 default on the 11+ user Jira tier or 600 with Zephyr Scale Automate) that we think we're misunderstanding the product entirely. 

I have a support case open with Smartbear asking similar, but we wanted to sanity check our assumptions and see if the community is seeing similar limitations?

Any help appreciated.

Zephyr Scale Cloud
  • MisterB's avatar
    MisterB
    Champion Level 3
    2 months ago

    Thanks for posting this question.  I'm not aware of any test execution limits with Zephyr Scale, and I can't see anything on the MarketPlace to suggest there's a limit.  There is an execution limit on the 'AI' or automation feature, but you buy those as an additional bolt-on.

    I'm interested to hear what you get back from support.

    • gregkennedy001's avatar
      gregkennedy001
      New Contributor
      2 months ago

      MisterBCorrect, there's no limit on the number of manual tests completed but any automated tests, run through Reflect by Smartbear, has execution limits at various tiers of usage.

      Even with the additional bolt-on of Zephyr Scale Automate the limit for our tier is 600 automated tests per month.

      This number is very small given that our CI/CD process, which is pretty standard, runs our all our integration and acceptance tests on every code Pull Request.

      We appreciate it takes resources and money to run these automated tests but compared to similar services like GitHub Actions, the limits are very low. 

      • MisterB's avatar
        MisterB
        Champion Level 3
        2 months ago

        Thanks for clarifying gregkennedy001.  I like the thought of using these tools (when an appropriate use case comes along) but I'm not a fan of the pricing structure and the usage limits you describe are no doubt going to be prohibitive.

  • gregkennedy001's avatar
    gregkennedy001
    New Contributor
    2 months ago

    Just for clarification, we are on Zephyr Scale Cloud purchased through Jira Marketplace