Hi,
It looks like your window is not treated as unexpected because of the way it is created or because of the way your test behaves. A modal window will be treated as unexpected only if your test tries to interact with the GUI objects it blocks (for example, if it tries to click a button which is currently not accessible). If your test doesn't do this, TC won't be able to process your window as unexpected.
Other possibilities are:
1. You window suspends the application's main thread and doesn't resume it until you close it. For example, crash dialogs behave this way.
2. Your window causes a lock-up. You test calls a method which invokes it, and this method waits for the window to close, but TC is unable to do this because it waits for the called method's result.
3. Your window cannot be processed by standard functionality of TC because it works in an unexpected way (doesn't react on WM_CLOSE, doesn't close by Escape, etc.).
For #3, you can handle the
OnUnexpectedWindow event and implement some custom logic in its handler (for example, find and click the needed button in your window).
For #2, call the method after which this window appears
asynchronously and process it as a regular window. Note that you can do this (without the asynchronous call) if you actually expect this window to appear. If it appears after a specific action with your application, just wait for it after this action and click the needed button in it.
As for #1, this situation is rare, and it usually doesn't apply to regular GUI testing scenarios unless your application crashes. This doesn't look like your case.