It has nothing to do with TC functionality, it has to do with physical limitations. Your machine may, simply, not have enough memory available to be able to keep the log in memory if it gets too large.
Artem's suggestion is the best, really. If your Log file is too large, you can do two things. First, prioritize your log and determine what messages, events, warnings, errors, etc., can be skipped and which ones are vital to your test results. You may be logging a lot of messages that are just "noise" and add no value to determining if a test passed or failed.
Do you really need to know, during a regular script run, whether each button was clicked, each key pressed, each window opened, etc? Or do you just need to know whether the test passed or failed and, if it failed, why? logging those other actions are good when debugging automation code, but for a regular run that you're just using to validate/verify/check a feature, they are not, strictly speaking, necessary.
Secondly, consider breaking your project up into multiple projects so that each project will, effectively, have a smaller log.
The only other thing, which is more expensive, is to expand the hardware capabilities of your environment.