Forum Discussion

marcl's avatar
marcl
Occasional Contributor
10 years ago

Failing "message content assertions" in reports

Hello,   I've got some test suites which are run nightly (using Jenkins), and for whose I receive a report every morning. This report is generated using "testrunner.bat -j", which puts the data in ...
marcl's avatar
marcl
Occasional Contributor

Thanks for your suggestion, however I'm specifically asking about "message content" assertions, not "contains" assertions.

Replacing our "message content" by "contains" assertions is not an option, as we would have to create hundreds of new assertions for most of our test cases.

 

The problem is not to understand which assertion fails, but to know:

  • which check(s) in the assertion itself failed (in the example above, 1 check amongst 160 failed but we don't know which one)
  • which values were expected, and actually received
nmrao's avatar
nmrao
Champion Level 3
10 years ago

I just took that one as an example, and the same can be used for other assertions too.

I can understand that you have too many to rename. But you may better rename the assertions to avoid the issue posted by.

So you are free to choose what is priority/needed.

 

In fact you can use the use the values(expected) as well in the name it self which might be appropriate at times.

  • marcl's avatar
    marcl
    Occasional Contributor
    10 years ago

    I don't think I've made the problem clear enough. Knowing which assertion failed is easy.

    The problem is that a message content assertion actually contains multiple checks at once, and I can't know which check failed.

     

    For example if I've got a MCA called "Check that the updated values are correct", which check the following:

    • foo == 1
    • bar == 42
    • a == 33
    • b == "toto"

    And I get the following data when executing my test:

    • foo = 1
    • bar = 2
    • a = 33
    • b = "toto"

    In this case, the error message will say that "The assertion named 'Check that the updated values are correct' failed, failed/compared = 1/4". But I'll have no means to know that it's "bar" value which is wrong, nor which value was actually expected / received.

     

    So no, renaming the assertion will bring nothing to the table :-/

     

    • nmrao's avatar
      nmrao
      Champion Level 3
      10 years ago

      Got it what you mean by.

       

      Though that assertion has Error Message as last column in that table while defining the Message Content Assertion, that is not being show to the user in the reports. Is that expected or not, I have no clue, it can be confirmed by Smartbear team.

       

      But if the same error message is shown probably it would have been useful to the user, in my opinion. May be the same can be bring to Smartbear attention to get that addressed.

       

      If we take yours as an example, you have 4 checks in one assertion. It might be showing something as below if 2 of them failed.

       

      Message Content Assertion failed : failed/compared = 2/4

       

       

      Irrespective of that, you may have to rerun the test once the fixes are available, and make sure entire assertion is passed instead of partial pass.

       

      The alternative at the moment would be, if at all the details are needed until the Smartbear revert back saying or confirming the issue and updates available, to use separate xpath assertions. Of course, it defeats the MCA assertion type.

       

      By the way, I could try another alternative with the MCA(instead of xpath) with single property check(since multiple not working) and following are results(renamed MCA to below value) and at least the message is clear:

       

      [Is conversion rate equal to 65?] Message Content Assertion failed : failed/compared = 1/1
[Is conversion rate equal to 70?] Message Content Assertion failed : failed/compared = 1/1

      Thank you for your time for taking time to explain the problem.

      • kondasamy's avatar
        kondasamy
        Regular Contributor
        10 years ago

        From the first message in this thread, I see this information,

        "In my tests, I use a lot of "message content assertions". But when one of those fail, I can't seems to find the reason of the failure in the report."

         

        So, you mean you have used multiple MCA assertions and you could not able to point out which assertion has failed in the overall JUnit styled XML report generated. Correct?

         

        Also, I tried reproducing the behavior and could see expected outcome while running through Ready! API 1.4.1. Please find below the details,

        Jenkins would call below command,

        testrunner.bat -s"Directions API TestSuite" -c"Simple Tests" -a -j -fC:\Users\jayarak1\Desktop\Demo -R"TestCase Report" -E"Default environment" "C:\Don't Delete\Sample Project\Google Maps REST\Google-Maps-soapui-project - REST XML.xml"

        I got the report generated in XML format and see a detailed error note for MCA assertions,

        ----------------- Messages ------------------------------
[Match content of [bounds]] XPathContains assertion failed for path [declare namespace ns1='http://maps.googleapis.com/maps/api/directions/json';
//ns1:Response[1]/ns1:routes[1]/ns1:e[1]/ns1:bounds[1]] : Exception:org.custommonkey.xmlunit.Diff
[different] Expected text value '45.5101458' but was '45.5017123' - comparing <lat ...>45.5101458</lat> at /bounds[1]/northeast[1]/lat[1]/text()[1] to <lat ...>45.5017123</lat> at /bounds[1]/northeast[1]/lat[1]/text()[1]

[Check number of steps [12]] XPathContains comparison failed for path [declare namespace ns1='http://maps.googleapis.com/maps/api/directions/json';
count( //ns1:Response/ns1:routes/ns1:e/ns1:legs/ns1:e/ns1:steps/ns1:e)], expecting [13], actual was [16]

        Since, I have used -a switch, SoapUI helps me in exporting test case wise results. Please try rerunning the command with above specified options and let us know the results.

         

        Thanks,

        Samy