Well... my first question is... what is making your log 5Gb in size? That's a MASSIVE log file... So, to be honest, probably before we start addressing how you can get your log file opened, a better question would be, what needs to be done to optimize the log file size.
A couple of tips and tricks for optimizing your logging:
1) At the beginning of your test run, make a call to Log.LockEvents(). This will prevent TestComplete from adding a log entry for every mouse click, key press, etc., that the tests do. These events are great for debugging... but for a regular test run, they tend to clutter things up. Additionally, LockEvents has a default of "20" as a parameter which means, for any warning or error, the log will display the last 20 events leading up to that... you probably don't need more than that.
2) If you don't need it, don't log it. The more log entries you write, the bigger your log file. So, don't log every little thing you do, only log meaningful results. For example, if you are testing whether or not a form is saved, don't log "Entered this data", "Clicked the save button", "Closed the window", "Form is saved properly". All you really need is that last entry... log just that, but then log when things go wrong. Essentially, if the information doesn't add value to the result set, don't log it.
3) Avoid logging pictures if unnecessary. Image files take up a HUGE chunk of log space, especially if you're using something like PNG or BMP. Again, only take a picture if it adds value in the result set. If you need to compare text on the screen, use an object comparison or a simple string compare and log if it fails.
Now... as to how to open your log file... I think some of the restrictions are based simply upon your machine specifications. If you don't have enough RAM, it can't open the file. TestComplete should be able to open most anything but it may take a while to open and process 5GB of data. I think, honestly, your best bet is to optimize your logging and try your run again.
