So i found a resolution a min after posting it:
Works under win10
1. Open "Regedit" and navigate to "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers\Configuration\XXXXXXXX\00"
2. Change the "PrimSurfSize.cx" and "PrimSurfSize.cy" to your desired values of resolution. Note that it must be changed in "demicial"
3. Then go to the second "00" and change "DwmClipBox.bottom" and "DwmClipBox.right", "PrimSurfSize.cx" and "PrimSurfSize.cy", "ActiveSize.cy" and "ActiveSize.cx"
Maybe the documentation from TestComplete should include this. For 100$ :D just joking :D
regards
Kai