Forum Discussion

SlickRick's avatar
SlickRick
Frequent Contributor
11 months ago

Flagging Known Bugs

So i wrote some UI tests for some feature which have known bugs.

Obviously those tests are currently failing because indeed the feature is broken.

However in my azure pipeline i would those tests to be flagged differently than other failing test so that i can know which one are really new failing tests.

The most logical way would be to show them as warning instead or errors.

Do you have an idea on how i could achieve this?

Currently im trying to use a @KnowIssue tag to identify those test, but not sure how to really consider that.

Also i though of renaming my test with a KnowIssue_ prefix and have the following OnStopTestCase event:

 

function TestFail_OnStopTestCase(Sender, StopTestCaseParams) {

    // Check if the test case failed
    if (StopTestCaseParams.StatusAsText == "Error") {
        // Check if the test case is tagged with @KnownIssue
        if (Project.TestItems.Current.Name.startsWith("KnownIssue_")) {
            // Log a warning instead of a failure
            Log.Warning("Test case '" + StopTestCaseParams.Name + "' failed but is marked as a known issue.");

        }
    }
}

Sadly this only add a warning on top of the current error and therefore the test still "fails with error"

Any ideas are welcome :)

Desktop Testing
  • MW_Didata's avatar
    MW_Didata
    11 months ago

    I like to post a warning on top and disable the actions inside the Keywordtest, this way you can see what tests are disabld by the warning, and then just remove the warning an re-enable when it gets fixed: 

    If you use a ticket system you could post the ticket number in the warning.

  • MW_Didata's avatar
    MW_Didata
    Regular Contributor
    11 months ago

    I like to post a warning on top and disable the actions inside the Keywordtest, this way you can see what tests are disabld by the warning, and then just remove the warning an re-enable when it gets fixed: 

    If you use a ticket system you could post the ticket number in the warning.

    • SlickRick's avatar
      SlickRick
      Frequent Contributor
      11 months ago

      I actually like this idea, simple and traceable.

      The main idea was to avoid forgetting to reenable the test when the test is fixed which can easily forgotten by the tester/dev if it is only disabled.

      By having a warning, at least the log still shows this warning and therefore less chance of forgetting it.

      This proposal does it, so ill accept it as a solution

  • Marsha_R's avatar
    Marsha_R
    Moderator
    11 months ago

    I would consider commenting out those tests until a fix is available. Are they adding any value at this point?

    • SlickRick's avatar
      SlickRick
      Frequent Contributor
      11 months ago

      I do feel like it would add value if i would be capable of flagging known issues the way i mentioned in my post.

      This way some team members can code certain tests and if it is a bug, we can still see the pipeline being 'not considered' final until all features are fully working and all known bugs are fixed.

      By disabling them, you need to rely on the developers to not forget to re-enable.

      Of course not that of a big deal, but i do think there is value to be able to tag certain tests as failing with a known bug and be able to differentiate between a test which is failing but reported and waiting to be fixed vs a test failing due to a new problem

  • rraghvani's avatar
    rraghvani
    Champion Level 3
    11 months ago

    I see no benefit in running failed tests, just exclude them from your test run. Or, if you're running an execution plan, and these failed tests are causing the run to stop, then adjust your project playback settings,

     

    • SlickRick's avatar
      SlickRick
      Frequent Contributor
      11 months ago

      But the purpose is not to exclude them from the test run, but to show them differently.

      A bit like TDD, where the product is not considered complete until all tests are fully passing.

      The idea here to differentiate between a new failing test (not known) vs a test that fails due to a bug which was already discovered but waiting to be fixed.

      Anyways, it seems like I'm the only one seeing value in this :)

      • rraghvani's avatar
        rraghvani
        Champion Level 3
        11 months ago

        I have a bunch of tests which are dependant on one another, and they need to be executed in order, using the execution plan. If a test fails, it will usually cause the other tests to fail, even though I have On Error set to Continue Running. However, if the remaining tests pass, I won't accept that as a pass. Until the issue has been fixed and the test that failed, actually passes. Certain situations may give a false positive or negative result. 

        It's best to keep the automation as simple as possible. 

  • Marsha_R's avatar
    Marsha_R
    Moderator
    11 months ago

    I think it's on the tester to reenable it when the fix comes back. 

    • SlickRick's avatar
      SlickRick
      Frequent Contributor
      11 months ago

      i do agree 100%. But i still like to have some sort of 'trace' that the test need to be enabled at some point. The warning does that.