Forum Discussion

NFox's avatar
NFox
Occasional Contributor
11 years ago

Test Project Report for documentation purposes

Hello.

We have just purchased SoapUI Pro for documenting our test procedures for a product we will soon be launching.

As expected, we have a project, with several resources specified.
Against these resources we have a test suite for each system requirement.
Each test suite contains the relevant test cases and tests.
Each test case has it's assertions.

For documentation purposes, I need to generate a report which shows what we are testing, and the assertions for each test. (Not a test result report, as this will be a separate document - I just need to provide an outline of what is to be tested and how a pass/fail is determined)

I have generated the standard report showing Test name and details, but I need to also include the assertions for each too.

E.g.
TestSuite1 > TestCase1 > TestStep1 > AssertionsA, AssertionB, AssertionC
TestSuite1 > TestCase1 > TestStep2 > AssertionA, AssertionB
TestSuite1 > TestCase2 > TestStep1 > AssertionA, AssertionB, AssertionC, AssertionD
...
etc
...

Please could you advise me How I would go about generating this report.
Thanks in advance.

Kindest Regards.

8 Replies

  • SmartBear_Suppo's avatar
    SmartBear_Suppo
    SmartBear Alumni (Retired)
    Hi,

    You should be able to generate the report without running the test cases to list test cases, test suites, project name, etc. However, to get the assertions without running the test cases you will probably have to do custom reporting for that.


    http://www.soapui.org/Reporting/customizing.html



    Regards,
    Marcus
    SmartBear Support
  • NFox's avatar
    NFox
    Occasional Contributor
    Hello SmartBear Support.

    Thankyou for the pointer.

    I have been reading the article, plus others online over the weekend.

    From my understanding I need to create a report template to show the list of assertions from an assertion dataource.

    Are there any more in depth articles on:
    - Creating a datasource (hopefully from assertions for a test)
    - Creating/modifying a template using a custom datasource


    Kindest regards
  • NFox's avatar
    NFox
    Occasional Contributor
    Hello all,

    So it's been a while, and I am still no further forward with this, despite much trying.

    I have managed to create a clone of the default project report, and made some changes.
    I have added in new custom parameters, so that the person running the tests can input their name (code below) and added that to the footer of the page (also below), along with changing the company logo and size, page numbers within the footer, and a few other things.


    <!-- Tester Username Paramater -->
    <parameter name="TesterName" class="java.lang.String">
    <property name="interactive" value="true"/>
    <property name="label" value="Tester Name"/>
    <parameterDescription>Required to put the Testers Name on the Test Report</parameterDescription>
    <defaultValueExpression>"Enter Your Name"</defaultValueExpression>
    </parameter>



    <pageFooter>
    <band height="43">
    <printWhenExpression>!$P{FlowLayout}</printWhenExpression>
    <textField>
    <reportElement x="0" y="0" width="535" height="40" forecolor="#999999"/>
    <textElement textAlignment="Center" verticalAlignment="Middle"/>
    <textFieldExpression class="java.lang.String">
    "Created with SoapUI Pro on " + new Date() + " by " + $P{TesterName} + "\n" +
    "Page " + $V{PAGE_NUMBER} <!-- + " of " + $V{PAGE_NUMBER} -->
    </textFieldExpression>
    </textField>
    </band>
    </pageFooter>


    However, I am still no further forward with getting assertions included into a report, or even including a single customised sub-report into my new custom project report.

    The reporting system seems very flakey, as if a single character error is made, the report fails to load, and no information is given as to what is causing the failure.
    Are there any log files anywhere, which would show what part of the report are invalid??

    I have also downloaded Jaspersoft Studio, and loaded the default SoapUI reports from the "[SoapUI install location]/bin/reports" folder there, to try and get a better understanding of what is going on with the reports too.
    On a side note, if I load any report template in Jaspersoft Studio, and save it without making a single change, it then fails to run in SoapUI. Why is that??


    I am at a bit of a loss here as to how I am supposed to get a list of assertions for each test step to display in the report.
    My understanding is that I need to make an assertion SubReport, which simply shows the list of assertions for that test case, and run that as a subreport within the testStep SubReport, which runs within the main Project Report.

    However, due to several issues:
    - the fact that I cannot use Jaspersoft Studio to make the changes, as reports made in that application do not run in SoapUI
    - there seems to be no information given when a report is invalid/fails within SoapUI
    I am pretty stuck.


    I've just found the SoapUI Log and Error log tabs, which give me the following error:

    SoapUI Log
    Fri Mar 28 12:41:17 GMT 2014:ERROR:An error occurred [net.sf.jasperreports.engine.JRException: Error retrieving field value from bean : securityScanNames], see error log for details

    Error Log
    net.sf.jasperreports.engine.JRRuntimeException: net.sf.jasperreports.engine.JRException: Error retrieving field value from bean : securityScanNames
    at net.sf.jasperreports.engine.fill.JRFillSubreport.prepare(JRFillSubreport.java:711)
    at net.sf.jasperreports.engine.fill.JRFillElementContainer.prepareElements(JRFillElementContainer.java:329)
    at net.sf.jasperreports.engine.fill.JRFillBand.fill(JRFillBand.java:419)
    at net.sf.jasperreports.engine.fill.JRFillBand.fill(JRFillBand.java:378)
    at net.sf.jasperreports.engine.fill.JRVerticalFiller.fillColumnBand(JRVerticalFiller.java:2070)
    at net.sf.jasperreports.engine.fill.JRVerticalFiller.fillDetail(JRVerticalFiller.java:760)
    at net.sf.jasperreports.engine.fill.JRVerticalFiller.fillReportStart(JRVerticalFiller.java:270)
    at net.sf.jasperreports.engine.fill.JRVerticalFiller.fillReport(JRVerticalFiller.java:128)
    at net.sf.jasperreports.engine.fill.JRBaseFiller.fill(JRBaseFiller.java:946)
    at net.sf.jasperreports.engine.fill.JRBaseFiller.fill(JRBaseFiller.java:864)
    at net.sf.jasperreports.engine.fill.JRFiller.fillReport(JRFiller.java:84)
    at net.sf.jasperreports.engine.JasperFillManager.fillReport(JasperFillManager.java:624)
    at com.eviware.soapui.reporting.engine.jasper.GenerateJasperReport.createReport(SourceFile:467)
    at com.eviware.soapui.reporting.engine.jasper.GenerateJasperReport$ReportFillWorker.construct(SourceFile:428)
    at com.eviware.soapui.support.swing.SwingWorkerDelegator.construct(SwingWorkerDelegator.java:46)
    at com.eviware.soapui.support.swing.SwingWorker$2.run(SwingWorker.java:149)
    at java.lang.Thread.run(Unknown Source)
    Caused by: net.sf.jasperreports.engine.JRException: Error retrieving field value from bean : securityScanNames
    at net.sf.jasperreports.engine.data.JRAbstractBeanDataSource.getBeanProperty(JRAbstractBeanDataSource.java:123)
    at net.sf.jasperreports.engine.data.JRAbstractBeanDataSource.getFieldValue(JRAbstractBeanDataSource.java:96)
    at net.sf.jasperreports.engine.data.JRBeanCollectionDataSource.getFieldValue(JRBeanCollectionDataSource.java:100)
    at net.sf.jasperreports.engine.fill.JRFillDataset.setOldValues(JRFillDataset.java:821)
    at net.sf.jasperreports.engine.fill.JRFillDataset.next(JRFillDataset.java:785)
    at net.sf.jasperreports.engine.fill.JRBaseFiller.next(JRBaseFiller.java:1482)
    at net.sf.jasperreports.engine.fill.JRVerticalFiller.fillReport(JRVerticalFiller.java:126)
    at net.sf.jasperreports.engine.fill.JRBaseFiller.fill(JRBaseFiller.java:946)
    at net.sf.jasperreports.engine.fill.JRBaseFiller.fill(JRBaseFiller.java:864)
    at net.sf.jasperreports.engine.fill.JRFillSubreport.fillSubreport(JRFillSubreport.java:613)
    at net.sf.jasperreports.engine.fill.JRSubreportRunnable.run(JRSubreportRunnable.java:59)
    at net.sf.jasperreports.engine.fill.JRThreadSubreportRunner.run(JRThreadSubreportRunner.java:205)
    ... 1 more
    Caused by: java.lang.NoSuchMethodException: Unknown property 'securityScanNames'



    Is anyone able to shed some light on either problem, or point me to some software which will help with generating such reports?


    I would have thought that generating a report on what is to be tested within a project would be simple, and a core feature of the reporting process, for documentations purposes, team visibility, etc.



    Many Thanks.
  • NFox's avatar
    NFox
    Occasional Contributor
    version 4.0.2 does not exist any more apparently.
    http://community.jaspersoft.com/project ... r/releases

    I downloaded the latest iReport software 5.5.0 (which is weirdly under the 5.5.1 section), on the download page stated above.

    In - Tools > Options > General Tab > Compatibility Tab, you can set the reports type as 4.0.2. I tried later versions, but they do not work!

    You can then save reports into the reports folder as specified in the SoapUI Pro settings.
    Once you have saved them there, you can simply refresh the reports in SoapUI, and your new reports should show up there!

    Now for some editing...

    Anybody have any experience in using assertions as a datasource??


    Regards,
  • SmartBear_Suppo's avatar
    SmartBear_Suppo
    SmartBear Alumni (Retired)
    Hi,

    You can try customizing pdf report but you can get the list of assertion in SoapUI pro using script,
    http://www.soapui.org/Reporting/customizing.html

    http://www.soapui.org/Scripting-Properties/custom-event-handlers.html
    Add this to event handler TestRunListener.beforeRun

    for (suite in testRunner.testCase.testSuite.project.getTestSuiteList()){
    for (test1 in suite.getTestCaseList()){
    for (i in test1.getTestStepsOfType(com.eviware.soapui.impl.wsdl.teststeps.WsdlTestRequestStep )){
    for( assertion in i.assertionList )
    {
    log.info "Assertion [" + assertion.label + "] has status [" + assertion.status + "]"
    }
    }
    }
    }



    Thanks,
    Jeshtha
  • NFox's avatar
    NFox
    Occasional Contributor
    Hello and thankyou for getting back to me.

    I have been looking through the forum and online for more information on the reporting and data types available within the reports and cant seem to find anything of any real value anywhere.

    I have been looking at the following threads:
    search.php?st=0&sk=t&sd=d&sr=posts&keywords=custom+report&fid%5B%5D=2&start=45 (search)
    viewtopic.php?f=2&t=19478
    viewtopic.php?f=2&t=21021&p=47475&hilit=custom+report&sid=e7f4e8fad7d42a5e5fb9a7ee131d344f#p47475
    viewtopic.php?f=2&t=20865&p=47005&hilit=custom+report&sid=e7f4e8fad7d42a5e5fb9a7ee131d344f#p47005
    viewtopic.php?f=2&t=13629&p=33314&hilit=custom+report&sid=e7f4e8fad7d42a5e5fb9a7ee131d344f#p33314
    viewtopic.php?f=2&t=13027&p=31796&hilit=custom+report&sid=e7f4e8fad7d42a5e5fb9a7ee131d344f#p31796

    The only information I can seem to find are links pointing to either Jaspersoft's IReport, or alternatively the SoapUI customising report page here: http://www.soapui.org/Reporting/customizing.html

    Unfortunately, none of the above links provide information to anywhere near the depth required for SoapUI Pro users to generate customised reports.

    I am currently in the process of writing documentation for our company on how I generated customised reports, and would be happy to share this once I have it finished, but need some help on the paramaters side from SoapUI for it to be of any use.

    For my situation, I have created a duplicate of the default ProjectReport.
    This has a customised version of the ModelItemListReport as a Subreport to display all the TestSuites in the project.
    The ModelItemListReport subreport (SubRep1) then has it's own subreport (another customised version of ModelItemListReport, SubRep2) which is supposed to display all the TestCases within that TestSuite.
    Ultimately, SubRep2 will have a subreport showing the list of TestSteps (SubRep3), and that a SubReport showing all the Assertions (SubRep4).

    Once I get to that stage, I will have a complete set of reports which will be able to show all the information required.


    However, I cannot seem to find the information anywhere which I need to create the datasource for SubRep2, SubRep3 or SubRep4
    I have tried various things such as:
    $P{REPORT_DATA_SOURCE}
    $P{REPORT_DATA_SOURCE}.cloneDataSource()
    $P{REPORT_DATA_SOURCE#TestCases}
    $P{REPORT_DATA_SOURCE#TestSuiteTestCases}
    $P{REPORT_DATA_SOURCE.TestCases}
    $P{REPORT_DATA_SOURCE.TestSuiteTestCases}
    $P{REPORT_DATA_SOURCE}.{TestCases}
    $P{REPORT_DATA_SOURCE}.{TestSuiteTestCases}
    $P{REPORT_DATA_SOURCE}.$P{TestCases}
    $P{REPORT_DATA_SOURCE}.$P{TestSuiteTestCases}
    $P{TestCase}
    $P{TestCases}
    $P{TestSuiteTestCases}
    $F{TestCase}
    $F{TestCases}
    $F{TestSuiteTestCases}
    etc, but nothing seems to work.

    I just get the following error for the middle report:
    Mon Mar 31 18:23:11 BST 2014:ERROR:net.sf.jasperreports.engine.JRException: Error loading object from URL : subreport:ModelItemListReport2
    at java.io.ObjectInputStream$PeekInputStream.readFully(Unknown Source)
    which I am guessing means that the datasource does not exist.
    (On a side note, if I pass in a field name I know exists, $F{name}, I get the following error:
    Caused by: org.codehaus.groovy.runtime.typehandling.GroovyCastException: Cannot cast object 'SRSB015 - Register_Create_WithRegistrationData_wMinHeader' with class 'java.lang.String' to class 'net.sf.jasperreports.engine.JRDataSource')


    I feel like I am very close, but am missing a crucial piece of information to enable me to get the array of TestCases for a TestSuite as a suitable datasource to use in SubRep2.
    I also anticipate the same issue for SubRep3 and SubRep4 for TestSteps and Assertions respectively.

    Please please please could someone get back to me and let me know the field names for the objects SoapUI makes available in a report. This is becoming a big issue for me, and probably many other people too.


    Primarily, I need to know the following for a ProjectReport object:
    All objects/classes and sub classes available to a report.

    I have discovered the following, but need a complete list:
    - $P{ReportMetrics}
    - $P{TestProperties}
    - $P{ProjectRequirements}
    - $P{ProjectTestSuites}
    - - $P{Type}
    - - $F{name}
    - - $F{description}
    - $P{ProjectMockServices}
    - - $P{Type}
    - - $F{name}
    - - $F{description}
    - $P{ProjectResultMetrics}
    - $P{ProjectTestSuiteResults}
    - - $F{testSuiteName}
    - - $F{status}
    - - $F{startTime}
    - - $F{timeTaken}
    - - $F{reason}
    - - $F{testProperties}
    - - $F{testRunnerResults}
    - - - $F{testSuiteName}
    - - - $F{status}
    - - - $F{startTime}
    - - - $F{timeTaken}
    - - - $F{reason}
    - - - $F{testProperties}
    - - - $F{testStepResults}
    - - - $F{failedTestSteps}
    - $P{ProjectCoverage}


    Many thanks