It's been a while since I've run into problems like this. However, IIRC, part of it has to do with how much information you're writing out to your test log. If you're calling Log.Message for every little thing or if you're not using Log.LockEvents in your code, I think that a good bit of what you might be dealing with is that TC has to store a lot of the logging information in memory. Take a look at optimizing your utilization of the logging, reduce the number of messages, and make sure that Log.LockEvents is in place to prevent too much information being written out.