Negative testing

Hi,



By default, test items are set up to stop when errors occur in them. Create test items for your tests and make sure that their Stop on error option is set to 'Test Item'. This will make TC stop any failed test item and start the next one.

We do not want to stop TestComplete. All tests should run through.





We evaluate the results of tests on the end from the logfiles:





<Prp Name="status" type="I" value="2"/>

"Value=2" indicates that an error has occurred.





The idea is rather in the direction of a new column for tc test-suite that says:

if error occurs ignore, else if NOT an error occurs then report as an error.





the alternative way: one have to negate the result in the code.





e.g.

procedure myBuggyFunction;

begin

  // ... o my god an error occured!

 

  aqObject.CompareProperty(DiffMem, cmpLessOrEqual, MemDiff_SoftPlayer, true, lmError);

end;

this would report me an Error in the log-file.

but we know, that dies test fails because the bug is not debugged.









i have to reform above code into this code to transfer my idea into reality:

procedure myBuggyFunction;

begin

  // ... o my god an error occured!

 

  ERRORLEVEL := lmError;

  if IgnoreTestResult then

     ERRORLEVEL := lmWarning;





  aqObject.CompareProperty(DiffMem, cmpLessOrEqual, MemDiff_SoftPlayer, true, ERRORLEVEL);

end;

... and this additional code i have to write in any test!

The "Stop on Error" option on the test items, when set to "Test item" will not stop TestComplete, all it will do is stop just that one test item.  So, all your tests will still run, just those items that generate errors will stop on reporting the error.



As for your code, rather than using the trick you did (which is good, actually), have you considered using the try/except/finally with the "raise" function call?  If an error or exception occurs, the "except" block will catch it where you can then call Log.Warning.



For that matter, on a more universal call, you can set in your project properties the "Stop on error" global setting to false.  See the screenshot attached.

yes, you are right. the test-suite is not stopped when an error occured, but look at my attachment.





it produces an red-coloured- error-message.

and this is correct, because the bug is not fixed.





we want that the test runs however,

and the test should not report a bug, if the bug is not fixed.





this is in other words: an inverse testing.





good is bad

bad is good.





if the bug is fixed, we want to switch into the normal testing-mode for this test.

ok. this is now my concept to handle errors:

procedure LogError(FunctionName: string; Error: Boolean);

var 

  Inverse: Boolean;

begin

  Inverse := IsInBlackList(FunctionName);  // list of non-fixed-bug-tests





  if Error then

  begin

    if not Inverse then

      log.Message('Error: '+FunctionName);

  end

  else begin

    if Inverse then

      log.Message('Error: '+FunctionName);

  end;

end;





procedure Buggy;

const 

  FUNCTIONNAME = 'Buggy';

var 

  Error;





begin

  // instead of: aqObject.CompareProperty(1, cmpEqual, 0, true, lmError);  

 

  Error := aqObject.CompareProperty(1, cmpEqual, 0, true, lmWarning);  

  LogError(FUNCTIONNAME, Error);    

end;

is there a function that delivers me the name of the function self? 

  LogError( FunctionName(self)  , Error);    

only for the sake of completeness:

procedure LogCompareResult(FunctionName: string; CompareResult: Boolean);

var 

  Inverse: Boolean;

begin

  Inverse := IsInBlackList(FunctionName);





  if CompareResult then

  begin

    if Inverse then

      log.Message('CompareResult: '+FunctionName);

  end

  else begin

    if not Inverse then

      log.Message('CompareResult: '+FunctionName);

  end;

end;





procedure BadBoy;

const FUNCTIONNAME = 'BadBoy';

var CompareResult;

begin

  CompareResult := aqObject.CompareProperty(1, cmpEqual, 0, true);  

  LogCompareResult(FUNCTIONNAME, CompareResult);    

end;





procedure BadBoy_Fixed;

const FUNCTIONNAME = 'BadBoy_Fixed';

var CompareResult;

begin

  CompareResult := aqObject.CompareProperty(1, cmpEqual, 1, true);  

  LogCompareResult(FUNCTIONNAME, CompareResult);    

end;





procedure GoodBoy;

const FUNCTIONNAME = 'GoodBoy';

var CompareResult;

begin

  CompareResult := aqObject.CompareProperty(1, cmpEqual, 1, true);  

  LogCompareResult(FUNCTIONNAME, CompareResult);    

end;





procedure Test01;

begin  

  BadBoy;

  BadBoy_Fixed;

  GoodBoy;

end;

BlackList

BadBoy

BadBoy_Fixed





Result after call Test01:


The property value "1" does not equal the baseline value "0". 11:10:29 Normal

The property value "1" equals the baseline value "1". 11:10:40 Normal

CompareResult: BadBoy_Fixed 11:10:50 Normal

The property value "1" equals the baseline value "1". 11:10:50 Normal

Hi,



As an option, you can handle the OnLogError event in your project. When an error occurs in your project, this event is fired. In its hanler, you can write something like this:

...

LogParams.Locked := true; // we don't want the error in the log

Log.Message('Oops... the bug is not fixed. Stopping the current test item', LogParams.Str);

Runner.Stop(true);

...

This will block the error, post its text to the log (in the Additional info section of the message), stop the current test item and continue with the next one.

Hi Jared, events are a cool thing. as it is described on your linked page an event must be directly connection to one special script. is there no GLOBAL on-log-error-handler for the test-suite? ... although... i create a script instead of the test-suite. so my open request about "how to get functionname" could be answered by listening to the log-out-stream... thanks.

The event handler is linked to a script only in that it is the script that is executed when the even fires. That event, though, will fire for any Log.Error calls throughout the test project.

Hi Robert,




wow, your sentence blows my understanding of the english language. : D 

so I want to cut these sentences into several small units, as I understand them:





"The event handler is linked to a script."

(understand)





"The event fires only for the functions in this script."

(theses: if i use function out of this script eg calling a function in an other unit: the error would not pass throuh my event-handler? So i can not catch them?)





"That event, though, will fire for any log.error calls throughout the test project."

(This record seems to refute my theses)





I think I need a little more deal with these events until I understand them in all their deep ...





thank you.