TE: configure test sequence in external file
I'd like to control TE's test run by an external file. In the file, the testing engineer selects from the available tests to execute the ones he Needs to. Has anybody done this before? My quest...
Forum Discussion
congrats, this sounds really good. I also use my own Framework, it also has been a LOT of work and is working fine. As an example, I'm using aliases only to identify dialogs, the rest is addressed using captions. Multiple instances of a Dialog are supported, too.
Using external test control will give the tester more control on what is to be tested when using TE.
I'll consider using Excel as a config file, too. It will be better structured than the simple text file I wanted to use at first.
The appropriate Location for the config file will be Project.ConfigPath. This is what I was asking for in this thread.
Yep. I use locators which map to the Alias of a control in mine too. The Alias names follow the onscreen labels so the test guys can work 99% of them out (a few are badly labelled or nor labelled at all on screen) without having to ask me.
I take them in with pipe separators, parse them into an Alias reference, and then do some check on the object to make sure it's a valid reference before it gets used.
So a single line ref on Excel (dashes being cell separators for this example), for me, would be something like:
Y - <BLANK> - <Function_Name> - <Optional Marker(s) and/or dependency(s)> - <Parameter - Locator> - <Parameter or Expected Result> - <Parameter or Expected Result> - .... etc - it can take as many markers, dependencies, parameters and expected results as required.
The "Y" column is a Yes/No indicator to run the test or not. Will be over-written with pass/fail/error when run.
The <BLANK> column will be filled a timestamp if the row is run.
The function name will be used with all the parameter data to build a call to a function.
After that, everything is optional depending on what the function needs. But each optional part has it's own cell. Putting all the parameters in one cell could get VERY cramped! Each cell has a prefix which tells the framework what it is. P: is a parameter. E: is an expected result. D: is a dependency. M: is a marker. It reads the cell. Separates the prefix from the rest of the content and uses it to determine what to do with the content.
A blank cell tells it it's got all the data for that row.
Markers are noted and stored.
Dependencies check the indicated marker results.
Parameters get added to the string, along with the function name, which is built into the function call made using an "Eval" statement. The object locator is just an input parameter. Albeit a special one which I parse into an object Alies reference and validate.
Expected results are stored in a disposable array and checked as required within the called function.
Obviously, all this has a ton of checking and validation round it. It's part of the framework so it absolutely cannot crash. And, generally speaking, it doesn't. Certain conditions (webserver with the site under test going down for example) will cause it to output a boatload of fails, but it won't crash. :)
nice!
Her is how I'm doing it:
I use ODT to organize test runs. ODT is structured hierarchically, so it enables me to set up really complex tests.
The ODT test Definition tree is absolutely parallel to the log tree, giving me a Maximum of clarity what happened where.
I'm so sad that ODT was cancelled, as only few guys were able to really make use of it!
Have a look on the screenshot: You see a test Definition hierarchy. Test data and reference data can be defined on the desired Level and reused anywhere below. I created two Standard data types for Dialog definitions and for field definitions.
Initialization, testing and finalization can also be set up at the appropriate Level. This enables me to execute single sections, in many cases even to execute single test items.
A powerful library gives me easy Access to the test data:
As an example, in the context of the test Routine BDE.Item1.Item1, I just code
Td = PVA_0.ODT.TestDatenArr(This, 0)
to Access the test data Array of the item. To get the test data of superordinate Levels, I'd have to fill in a 1 or 2 instead of the Zero.
Once having Access, I can
- have TC get/set the text, Click the button or item for test data definitions by simply handing in the context of the control (the Frame or row object where it is actually located);
- open a Dialog by menu or by button for Dialog data definitions.
This works nearly perfect.
What I'm doing now is to make the test run more controllable from outside by configuration. I want to be able to just run BDE.Item5, for example, regardless of whether the ODT items are enabled or not.
Having never used ODT, a suspect parts of this are lost on me!
But it sounds good. And I like how it matches closely to the log. But with ODT in play, I have no idea how I would separate all this out in some sort of external control file .....
For me, the main Advantage of the Approach is the tree structure. You can also emulate this in Excel, but it is not as beautiful there.
A second Advantage is the ability to use generic subtesting for Standard test purposes. This will be very hard to get based on Excel..
About applying the control worksheet:
Of course the data is in the ODT tree (or in any store with ist key in the ODT tree), not in the Excel workbook.
So what has to be done is to execute ODT items on demand.
I Limit the granularity of control to Level 3, meaning Test.Section.Sequence, as the Items can not in every case be executed independently from each other, whereas Sections and Sequences can.
For this purpose, there are two mechanisms: ODT elements can be enabled/disabled or filtered.
There is a Scripting Interface to the ODT tree (enabling) and the ODT classes (filtering). I tried it out: to do filtring, You have to Change properties of the ODT class. Unfortunately, this does not work because of a Software bug: the Change is done, but the result is not written back to the ODT.xml file.
I tried writing the ODT elements "checked" property this way to simulate enabling. This works, I do not have to Change the XML myself.
