One of the things that can blow up the memory of an automated test suite is the amount of data being written to log files. A lot of that is kept in memory until it is written out to file. So, if you are logging a lot of informational messages (Log.Message) that are simply that, informational, and are not serving any actual validation purpose, you might want to consider suppressing that logging. Also, locking down events can help with that as well.
Also, depending upon the script language you're using, there may be garbage collecting in the way of undisposed of objects that you might want to double check on. Some languages do this automatically... others you need to be a bit more deliberate in freeing up objects.
Just some things to consider regarding your test suite.