java.lang.OutOfMemoryError: Java heap space error?

How much RAM does the machine running your tests have?

NewAutoTester: Is this error coming from the Linux or Windows machine?

On your Linux machine, setting the memory values to 4096 means you are allowing ReadyAPI to use ALL of your available RAM. It will not be able to do that, since the OS and other applications will need to use some RAM as well.

On the Windows machine, I'd say you could safely set that value to 12288 instead of 4096 to see if it fares any better. That would be 75% of your 16GB of RAM which might work better for you. My testing machine has 32GB of RAM and I set my memory to 24576. 

The Java heap space error that you are facing actually indicates that your JVM is running out of memory to allocate for objects in the heap. Which means, the Java Virtual Machine (JVM) running is actually running out of memory allocated to the Java heap. Even though you have already increased the JVM options (-Xms4096m -Xmx4096m), this type of issue may come up if the application is holding on to objects longer than necessary.. or suppose if the workload by itself requires more memory than the configured. This is because, always, JVM by default does not utilize the full container memory unless we instruct to.

On reviewing your stack trace, I get that the error occurs while processing transactions in parallel streams and during project loading in SoapUI (CompositeProjects.loadCompositeProjectFile). This suggests that a large amount of data is being processed or loaded into memory at once. 

To overcome this, there are a few areas to consider,

Suppose, if the issue continues even after increasing the heap size then it should be mainly because of the objects that will be retained unnecessarily in memory. This is actually an indication or a suggestion for a possible memory leak. In such scenarios, it is advisable to enable heap dump generation on OOM (XX:+HeapDumpOnOutOfMemoryError) and analyze the dump using tools like HeapHero to identify which objects are consuming the most memory.

The next possibility could be when the application attempts to load very large files or collections into memory all at once. Yes, this action will actually overwhelm the available heap. This should be the logic behind any application. If any application tries to load large files all at once, definitely the available heap will overwhelm resulting in the “Java Heap Space” error .
If you notice that the garbage collection is running frequently and not reclaiming space then it is a sign that the objects are still strongly referenced. So you can do an analysis for both heap and thread dumps together and get an understanding.

In short, with the recommended steps you can first check your heap size configuration and increase the -Xmx value if needed. If the problem remains the same, then definitely you will have to analyze heap dumps to understand memory usage, with which  you can change your code to process large data sets more efficiently.

You can check out this blog How to Solve OutOfMemoryError: Java heap space to understand more about this Java Heap Space error.