TestSuiteRunner.fail() only works in gui, not in testrunner

Using soapui 4.5.2.

I implemented a TestSuiteRunListener and put it in a jar inside the bin\ext folder. I also defined a listeners.xml file and put it inside the bin\listeners folder.

When I call the TestSuiteRunner.fail("myMessage") or TestCaseRunner.fail("myMessage") method in the any of the hook methods and run a testsuite inside the gui it fails with the supplied message as expected.
However, when I run the testsuite with the supplied testrunner.bat it seems to ingnore the failure and I get the following output:

[exec] SoapUI 4.5.2 TestCaseRunner Summary
[exec] -----------------------------
[exec] Time Taken: 390ms
[exec] Total TestSuites: 1
[exec] Total TestCases: 1 (0 failed)
[exec] Total TestSteps: 1
[exec] Total Request Assertions: 1
[exec] Total Failed Assertions: 0
[exec] Total Exported Results: 0
[exec] soapUI 4.5.2 TestCase Runner

It also does not put an error in the output folder.