Okay, just making sure I understand which OS and memory configuration we're troubleshooting there. Can you open Task Manager on windows and watch memory increase as it goes? Running hundreds of tests should not necessarily cause this sort of problem. One thing that comes to mind is to ensure you've set your memory optimizations per this article.
I would set every test case's settings to a Max Results of 0 and to discard successful test steps results to save memory.
Another piece you can try is to put in a groovy script test step / assertion that does the following code. This will force ReadyAPI to do a garbage collection. This has helped me in some instances when ReadyAPI starts using a lot of memory.
Unfortunately, without knowing what your tests encompass or the size of data in the request and/or response, it is difficult to troubleshoot. One option might be to open a support ticket, especially if the software continues to crash with that error.