Forum Discussion

Taz's avatar
Taz
Contributor
2 years ago

How to best assert unordered results?

One of the great features of ReadyAPI are the Smart Assertions.  They are really useful when dealing with large amounts of data and can be easily adjusted when there are some changes by using "Load from Transactions".  This saves a lot of time having to manually add each assertion.  However, there a a few issues that make it less than ideal.  1. No Search Facility to quickly find an Assertion. 2. No Facility to add Flows like If and Else or Loops. 3. No Conversion to Script.  4. And especially no facility to deal with unordered list results where the order can be different with each call.  5. Also no Update Feature that allows preserving properties and previous setups. Therefore we unfortunately can't really use this feature with most of our calls.  I would like to know if there are others that have similar problems to solve and if there is a recommended way to deal with this apart from writing complex scripts. Data-Driven Tests might be one way.

Assertions
Data-Driven Testing
Function Tests
REST
Scripting
Test Results
  • JoostDG's avatar
    JoostDG
    Frequent Contributor
    2 years ago

    Hi Taz . I am glad that you bring thus up again. There are many users frustrated with the "Smart Assertion" feature (me included).

    See an old post here : https://community.smartbear.com/t5/ReadyAPI-Questions/Issues-and-Feedback-about-Smart-Assertions/m-p/228752#M55442

    Bottom line: The current "Smart Assertions" feature does not accommodate the majority of the ReadyAPI user's needs and I for one am eagerly awaiting a SmartBear ReadyAPI update on this...

    No easy way to assert unordered results. Currently only complex scripting does my trick. Or, if you don't mind some loss in coverage, the assertion groups  can also help ("assertion 1 OR assertion 2" see https://support.smartbear.com/readyapi/docs/testing/assertions/index.html#assertion-group) 

  • nmrao's avatar
    nmrao
    Champion Level 3
    2 years ago

    We understand that the smart assertions falling short of doing certain thing and there were posts to get the ReadyAPI team's attention. Let them come back.

     

    Just curious about your use case. Would you please explain with your input sample and what is expected? Will see if there are other possibilities.

    • Taz's avatar
      Taz
      Contributor
      2 years ago

      Our APIs return large lists of data that contain other lists of data that are taken from a database.  The order is never guaranteed because of parallelism where data is pushed out as soon as they are available and also because the database itself returns data with no guarantee of order.  This means that the root data can be unsorted as well as the leaf data.  Right now to ensure that all data are asserted correctly it is necessary to do it in a script where we either can pre-order the results and evaluate them one by one or make conditional statements where we check the id and evaluate correspondingly.  Some fields are expected to change over time like version numbers, we therefore evaluate them against a single property that we can change with each new version.  Other fields do change at times during development and we therefore need a way to quickly adjust for that.  If Smart Assertions would recognise Lists they could do the ordering based on a specified key.  This would be ideal but unfortunately this feature doesn´t exist yet. They could be a little bit smarter in my opinion 😉