Issues and Feedback about Smart Assertions

[steve_rivers]

Some issues and feedback about Smart Assertions after a day or so of using them. 

I would like to provide some hopefully helpful feedback on the newly introduced Smart Assertion.

Please set the default name generated based on the type selected by the user. e.g. Smart Assertion - Metadata or Smart Assertion - Data

The smart assertion does not handle simple text responses in the way that message content assertions do. The following response cannot be parsed by the Smart Assertion. Sure it isn't following good API design practice but it is a valid HTTP response and it is valid JSON data.

HTTP/1.1 200 OK
Date: Mon, 19 Jul 2021 21:43:03 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 29
Connection: keep-alive
Server: Kestrel
Cache-Control: no-cache
Pragma: no-cache
Expires: -1

"Your name has been updated."

It doesn't seem all that smart to automatically include and enable in the metadata assertions data that is obviously going to change from response to response. Who thought that automatically adding response headers such as Date and Content-Length was a good ides? Date is guaranteed to change with every request. Content-Length is also likely to vary.  This is not going to be an enjoyable experience for the novice user of ReadyAPI - "Use our Smart Assertion on your response, make your testing easier, you'll get a fail the moment you re-run the test". 

[nmrao]

Nice that you have tried and valid inputs.
Hope Smartbear team listens to this feedback.

[sdeevers]

I know I'm a little late to the party on this, but ...

We use HTTP assertions on every test; header exists, header equals (with some regex thrown in the re as well) , status codes, etc.

Additionally 99% of our tests are data driven, so, those values could change on every request.

There are two operations in the Smart Assertion; "Received Data" and "Received Metadata."

After executing one specific test I performed the following

I turned off (deselected active) the "Received Data" as we have groovy code validating the response

I then turned off (deselected active) all "Received Metadata", then activated the specific HTTP data we need to validate. This was performed using conditions (equals, exists, matches regex, etc.) and updating the "valid value" column (some data driven info) as needed.

I then disabled all the "deprecated" assertions we previously used and saved the project (did not commit and push to GIT just yet --- smart move)

I then executed the test to see if everything still passed, which it did.

When I opened the Smart Assertion to see if all of my changes were still in place, the "Received Metadata" page was displayed and showed my changes were still in place (which I was very happy about).

When I changed to "Received Data" I received a popup showing all the content was being updated with the last response, which removed all the work I did above. Not a happy moment any more.

From my perspective once you have "saved" what you want to validate (Active box selection) for both options of the Smart Assertion, the Smart Assertion  should not repopulate unless you choose the "Populate from Received Data" or "Populate from Received Metadata" because everything you previously setup will be removed.

If new HTTP parameters are added in a new release of your code, one would want those to be available to configure but not have to update all previously set items as there is a very good chance one may miss something which was or was not previously configured. In previous releases, the HTTP assertions remained so if something new came along you could easily add the one, two, etc., HTTP assertions without having to reconfigure everything.

Please let me know if we are doing something incorrect when configuring the "new" Smart Assertion. Happy to do a demo for the support Staff !

Stu

[sdeevers]

Guess I was a little quick on the draw ... If you click Cancel after "reviewing" what was configured, what was originally configured will be displayed.

[TwoFang]

The one major issue i have with smart assist is the idea that writing a text trying to inform the user that data will be lost if you swap between the two types is just to me proof that this feature should never have been released in its current state

If it atleast would have been an annoying pop up confirm dialog so you are forced to read that your work will be deleted would be better and then can select cancel/continue 

Smart insert simply needs to be split into two asserts one for each tab unless it can be changed so that you can effortless move between the two types without losing data

[marjor]

Metadata and message content assertions should be completely separate assertions because switching the type destroys all the data you have added. How can you modify previously added assertions when all the values are lost? It is the same as not having a modify smart assertion feature at all. 

[sdeevers]

What we have started doing (yes its a "pain point") is to create 2 separate smart assertions, one for "Received Data" and one for "Received Metadata" and then naming them accordingly so users will know which smart assertion to update and / or check if an error is returned.

[marjor]

Yes, I ended up doing the same 👍

[JoostDG]

Any update on this Smartbear ? I see no improvements on this in the ReadyAPI v.3.20.0.

I agree with all of the above comments and am equally frustrated by the invisibility of WHAT is tested in the "smart" assertion and the sheer inflexibility. Updating them, for instance when new additional JSON response data is returned, is not possible. 

Also, when adding a request from the API section to the testcases via the "add to Test Case" right mouse click menu, the 1 of the default 2 assertions that ReadyAPI by default suggests is the "Valid HTTP Status Codes"... which is in the same ReadyAPI version set as DEPRECATED 🙄. (Side note: In my view these "default assertions" should be open to customization by us. Preferably also based on the swagger info (or other definitions) you used to import the API. For instance, even when it's clearly set in the swagger that a POST request should return a 201 ReadyAPI still suggest 200 😒).

I was more in favor of the above deprecated assertion as at least what assertions are present in my test step without the need for opening the smart assertion. It would be nice to have a quick overview of what is asserted against... In the end, it's the most vital thing of testing: What does your test do? Do I need to add any more checks? I wish I could see this in a glance.

I would not mind that the smart assertion of let's say 4 "Smart Assertion - Metadata" and "7 Smart Assertion - Data" would be translated to 11 individual assertions in the assertions section. Ideally each which the expected and actual value so testers can see what's going on.

[JoostDG]

*BUMP* 

 @sonya_m : Anyone from Smartbear care to reply?

Is there anything in upcoming release of ReadyAPI that deals with the frustrations from your user-base mentioned in this ticket?

The "solution" in this topic is obviously not a long term solution.

I just started a new project and would like to know whether I should invest in using the "smart assertion" or just leave it...