Hi,
The generic requirement for all applications that have UI and interact with end-user is to have an active desktop session.
To fit this requirement (and note that it is not TestComplete-specific, but is valid for Selenium and any other GUI application):
a) There must be active, not locked desktop session on the test computer; and
b) Jenkins agent must be started not as a service (because there is no active user session in this case even if you allow service to interact with the desktop in its settings), but as a regular application within this session.
See https://support.smartbear.com/testcomplete/docs/testing-with/running/via-rdp/index.html and https://support.smartbear.com/testcomplete/docs/testing-with/running/on-locked-computers.html for more details.
P.S. Also, Smartbear provides Jenkins plug-in for TestComplete - https://support.smartbear.com/downloads/testcomplete/jenkins-plugin/ (https://support.smartbear.com/downloads/testcomplete/)