Publish TC results to Visual Studio Online

I can add a little more to this.

Obviously, TFS is not static. As your project evolves, things advance in TFS.

But at the same time, you don't want to be constantly updating your tests with hundreds of new identifiers every time you run it.

In our use case, the regression test suite is copied forward each time a new sprint is started. A sprint is a "plan" in TFS land.

So our structure is:

Project > Plan > Suite > Test

My config file contains the following configuration options and is read once at the start of the run:

• TFS Active - Boolean flag to say whether the run is using TFS. We don't always want to.
• Authorisation - Authorisation token allowing the connection via the API. TFS has a couple of options for how you Auth.
• Project - The ID (or name) of the project to use.
• Plan - The ID (or name) of the plan to use.
• API version - The currently installed version of the API. This gets bolted to the end of requests to the API. Not required, but TFS states it's best practice to do it. So I do.

These bits do require updating, but infrequently. And they are single entries in config. So it's manageable.

Then within my test "packs" (I use my a data + keyword driven framework), there are references to:

• Suite - The Suite NAME to use. Name is used rather than ID as it doesn't change when the whole lot is copied forward in the next sprint (Plan in TFS land - see above). Given when a new suite comes into use. Can change multiple times throughout one of my test "packs" if need be. But usually only set once at the start as tests within a "pack" tend to be logically grouped in much the same way as they are within TFS.
• Test - The ID of the test Item within the Suite. Given along with a "start" flag. Ended using an "end" flag (as multiple steps within a TC test "pack" tend to form a single TFS test. They are seldom single step entities in my setup).

These entries should not require updating. The only time they do is if someone (annoyingly) changes the name of a test Suite.

When it starts a suite, it searches the current Plan for a suite of that name. It then finds all the test Items (as TFS refers to them) within the suite. Each test Item contains a reference to the original test ID. You need to scan through them to link the Item to the original Test, and hence to the entry on my test "pack". You can't use the Item ID as it changes every time you copy the Suite forwards and you would have to update every single individual test. Not practical. Obviously. Suites are referenced by name for similar reasons. The name doesn't change when you copy it forward, but the ID does.

Technically, you should then create a Run in TFS. With test Items in it. You should then pass/fail the Items in the Run. When you complete the Run, the associated Suite should updates IT'S Items in line with the pass/fail's in the Run. But that's where the RESTful API is (or was last time I checked) broken. That link doesn't work. And Suites do not update in line with Runs as they should.

So, instead, I update each test Item in the Suite directly as it completes. As a stopgap. It works fine. Just makes grouping for reporting a little less intuitive as you don't have everything gathered in a single run. It's basically the same as a manual tester right clicking on a test within a Suite setting it's pass/fail status.

Hope this lot helps as understanding the data structures used, and how to use them in such a way that I don't have to update hundreds of identifiers every time I want to do a run, was probably the trickiest bit of using the API to figure out!