Very slow test startup

Hello Charles,



I would suggest if you or your team could optimize your project in such a way that a project will only contain maybe 50 scripts (tests) instead of 700. The reason why starting up the tests takes too much time is that during pre-compilation time (time after clicking the "Run" button), your 700 script's source code will be compiled. Can you also take a look on how many LOCs are there in each script? This would also be a factor during compilation time.



Regards -- Paul

Consider also doing some redesign to, instead of having 700 distinct tests in script code or keyword tests, building a framework of sorts to execute the tests.  The code base for such a framework will be considerably smaller and, therefore, start up faster while allowing a greater flexibility to add more tests in the future.

Thank you guys both for your responses.



Paul, does this compilation happen only once during the test run or is it before every test?  I am using COM automation to drive TestExecute.  I only open the project suite before the first test then TFS takes over as far as telling it which scripts to run.  The delay occurs between every test in the run though, so I'm curious if it's trying to compile the entire project before every test...?



Our scripts vary widely in length depending on the test, but just looking at the metrics...  we have 770 script units with 47,454 lines of code so we're averaging about 59 lines per script.



Is the number of scripts per PROJECT (not project SUITE) the main issue?  Because I could technically split the scripts into separate projects within the same project suite, but that will be a huge undertaking so I want to make sure it will make a significant difference before I attempt it.



Robert, we are using TFS as our primary "framework" for writing tests.  Our QA testers write test cases in TFS, then we script them (using one script unit per case to keep them separate) and associate the script back to the test case.  When playing the tests back, I have written a test "harness" that drives TestExecute to execute each test similar to a unit test.  We have written a library of functions that are used repeatedly, but we have to use TFS because that is what our department uses and it allows us to keep a history for each test case.



Are there any plans to improve performance in larger projects like this?  We purchased 5 licenses of TestComplete Enterprise for this very reason, because we were told that there is no limit to the number of script units a project may contain.  While this is technically true, a performance limit of 50 scripts per project for testing a huge application like we are is a major cause of concern right now.

Hello again Charles, looks like your execution process is the bottleneck for the test runs. Per my understanding (and please do correct me if my assumption is incorrect), TFS relaunches the project, say project 1 up to 700, in each run. By concept, the steps followed is the following (if right):

1. Launch TestComplete via COM


2. Open Project.


3. Run Script 1


4. Close Project


5. Repeat steps 2 to 4 until all the queued scripts have been processed.

Bottomline is, compilation happens in each run. (And that's only my assumption). If this is not the case then do let us know and we'll try to pin point the bottleneck.



BTW, can you check the antivirus settings of your test machine? It might be scanning your scripts during run time.



Regards -- Paul

Hi Paul,

Thanks again for your continued input here...



I've actually designed the test harness so that it does not re-open the project for each test.  So for each test run, it opens the project suite before the first test runs, executes all tests within the TFS test plan that I'm running, then closes the project suite upon completion.  



I am fairly certain it has at least something to do with the size/number of scripts in the project.  The slow test startup also occurs when I run an individual test locally on my machine from within TestComplete, although it isn't near as bad.  It is also interesting that the problem doesn't seem to occur if I execute a test directly from the script rather than through the project's test items.  Each test has an associated test item in the project, so there are also ~700 test items.  I need to execute from the test items though because each test has individual timeout values that I need to utilize.



We have an entire lab of computers, some with antivirus software and some without, but the problem seems to occur despite this.

  

 

Hmmm... I'm out of ideas here. It's just my gut feel that what's causing the slowness in the 700++ items so going back to original suggestion is to re-write the suite in such a way that it's not too "bulky"

I think you're correct, Paul.  I appreciate you trying to help out here though.  



Robert, aren't ya'll using TFS also?  I'm trying to think of a way to maintain our existing test case tracking while simplifying the TestComplete project structure.  Already just getting TestComplete fully "integrated" with TFS has involved quite a bit of coding on my part and I'm willing to do more if I can come up with a way to get the best of both worlds...



Also, does anyone know if I split 700+ script units over multiple projects in the same project suite if it will improve startup performance?  

We are using TFS as well.  However, the framework I've created allows you to drop in whatever CSV file set you want to run a particular set of tests.  We have not implemented it yet, but it would mean creating a custom script task in the TFS build environment to copy from a sub-directory the desired set of CSV files before executing the test.  One test item in the project, one set of flexible libraried code, potentially infinite number of test combinations.



The only thing this does not do is tie a test back to a test case in TFS.  I don't think, for me, that is a critical path item but YMMV

I'm thinking something along those same lines...  Such as developing our tests from one main project like we are currently doing, then when executing our test runs, programmatically adding the scripts that are going to run into a new project and executing from there.  I think something like this could be done which would allow us to retain our current tracking history by case but still utilize the power of TestComplete.



Just haven't had time to delve into the details yet...

Using a table driven framework means you don't need to programmatically add scripts.  The tests are executed based upon formatted data records in an external source.  I use CSV files in my framework (less overhead of needing a third-party app to edit, easy to compare in TFS source control, light-weight, etc) and a set of case/switch statements to read in what sort of step is being executed and grabbing data to use within the step.  The code stays very static when adding a new test, so long as the existing code already supports the steps you're executing.