I'm trying to setup tfs with git repository, getting error: unable to determine the workspaces

If you're using git within TFS, you will need to download and install the git client in TestComplete.  The error message you're getting appears to be coming from an attempt to use the TFVC plug in.  Go to Tools | Options | Source Control and make sure you have the "Git Plugin" selected.