Hi all,
I did some investigation myself, and I can give you an update now. First of all: the positive news: I got it working! :) :)
I was triggered specifically by this part in the error message:
" x264 encoder error: height not divisible by 2 "
the situation is that I am running TestComplete in a VM, which was not running on full screen, appearantly at a resolution which is not divided by 2.
What I did to resolve it, was running the VM in fullscreen mode, which lead to a screenresolution dividabe by 2 (even numbers).
That did the job. So, now everything is working as expected.
Thanks all for thinking with me.