Forum Discussion

pb1's avatar
pb1
Contributor
6 years ago

Data Driven testing with dynamic/changing assertions

Hello fellow SoapUI users.

 

I am currently working on implementing data driven tests with a DataSource. The idea came up that we could cover a program flow chart with SoapUI by creating a "flow template" and then control the flow using the parameters from the data source. This means we need to use dynamic assertions, checking for content that one row produces which might not exist in other rows. Sadly I cannot find a way to dynamically enable / disable assertions and set their expectation value.

 

One idea is to have a groovy script that dynamically evaluates the data in the current row and asserts according to the expected outcome within groovy. This leads to a lot of groovy scripting which I would like to circumvent because the system will be passed to a successor with limited programming experience and I fear for the maintainability of the tests.

 

Another idea could be to simply include the expected assertion values in the DataSource and have ambiguous assertions that take the data to assert against from the row as well. This could however lead to an unmaintainable Excel-Sheet with too many columns and an unintuitive way to call assertions, this also does not allow asserting a dynamic amount of potentially generated parameters.

 

I hope you can provide me with further ideas or refer to a similar implementation.

 

Thanks in advance.

Assertions
Automation Environments
Data-Driven Testing
REST
  • pb1's avatar
    pb1
    6 years ago

    Depending on how the tests will be structured I can encapsulate and separate most of the batches, but the flow diagram I am provided with is very long and basically has a few "bottlenecks" that I can use to split the TestCases. On a first glance the worst flow has 12 decisions - each causing a change in the expectation - that I need to cover. After talking a bit more with my colleagues we think the best way to go about it is the bloated excel sheet because it offers a single point of reference, even if it means that our columns will be bloated. 

     

    Thank you very much for your input, I'll keep the topic open for more ideas but will implement the large excel-sheet for the time being.

  • groovyguy's avatar
    groovyguy
    Community Hero
    6 years ago

    I have looked down the path you are starting to work towards and it was exactly what you feared. You either have to write a groovy script that can handle all of the various cases, that can get inordinate and complex entirely too quickly. You can store the assertion in a data source and call it there, but that will increase the bulk of the excel sheet as you stated.

     

    The closest solution I could come up with that was "easier" but by no means less complex, is to write the assertions for every outcome on the test step. Then you can have a groovy test step that parses the DataSource data before the test step does, and enables/disables assertions on the actual test step as needed. 

     

    Does that help?

    • pb1's avatar
      pb1
      Contributor
      6 years ago

      Thanks for the fast response.

       

      In my case maintainability is the top priority, your idea does sound like the most suitable one because it allows the user to take a look at all the possible assertions that COULD be called if the groovy script enables them. I simply fear drastic changes in the API causing problems in the "toggle-logic" for the assertions which cannot be taken care of by my client.

       

      Building upon that idea it might be a solution to create a secondary data source to determine which assertions to enable/disable that can then be referenced in the primary DataSource. Something like the screenshots attached imply. In order to aid understanding of the flow-versions a graphical interpretation of the current path in the flow-chart may be in order. Again - maintainability is (sadly) my focus.

       

      Any further ideas or improvements?

      • groovyguy's avatar
        groovyguy
        Community Hero
        6 years ago

        I think using two data sources could work, assuming the two of them can be kept in sync if there are going to be multiple options. I do not see a way through to solve this prolem that does not require some level of maintainability, whether within groovy scripting or the entire logic flow. 

         

        As a rough guess, how many different batches of assertions do you think you have to handle?