Unable to access the checkbox object

I'm not a VB guru so forgive bad code syntax.  Also, your first screenshot showing the Standard view of the preoprties cuts off at WndStylesEx.  Other than the Extended options in the second shot, I'm assuming there are no other properties?

That said, instead of using Window("Button", "", 1) to identify it, you can do a FindChild call.  It would look something like this

If mainDialog.FindChild(["ObjectType", "ObjectIdentifier"], ["Checkbox", "0"], 1).wState = cbUnchecked Then
        mainDialog.FindChild(["ObjectType", "ObjectIdentifier"], ["Checkbox", "0"], 1).wState = cbChecked
End If

Some articles for your reading pleasure:

https://support.smartbear.com/testcomplete/docs/reference/test-objects/members/common-for-all/findchild-method.html

And, I would suggest you investigate using NameMapping for object identification.  Please review

https://support.smartbear.com/testcomplete/docs/testing-with/object-identification/name-mapping/overview.html