OCR in UI Test Automation: Extending Coverage Where Traditional Identification Breaks Down
Automated UI testing increasingly operates in environments where traditional object identification is not reliable. Modern applications frequently render text and controls using custom graphics, canvases, charts, and dynamically generated visuals that do not expose accessible properties or stable locators. As a result, automation tools that rely solely on object hierarchies and properties can struggle to validate what is actually presented to the user. Optical Character Recognition (OCR) addresses this gap by enabling automation to extract and interpret text directly from what is rendered on screen. Instead of depending on the underlying implementation of a control, OCR works at the visual layer, analyzing pixels and patterns to recognize characters and convert them into machine readable text. This capability allows automated tests to validate user visible content in situations where traditional approaches fall short. Why OCR Matters in Real World UI Testing In many business critical applications, text is not always exposed through standard UI controls. Common examples include: Charts and dashboards rendered using custom drawing libraries Canvas based interfaces and rich graphical components Embedded documents such as PDFs or reports Custom buttons, labels, or alerts built without standard accessibility metadata In these scenarios, the risk is not just test fragility it is blind spots. If automation cannot confirm what text is displayed, teams are forced back to manual validation for critical user facing information. OCR enables tests to verify visible content regardless of how it is implemented. By converting visual text into actionable data, OCR allows teams to assert that values, labels, messages, and statuses shown to users are correct, even when object level access is unavailable. This makes OCR especially valuable for validating end-to-end business workflows where correctness depends on what users actually see, not just what the application internally represents. OCR as Part of TestComplete’s Object Recognition Strategy TestComplete incorporates OCR as part of its broader approach to handling complex and non standard user interfaces. OCR is available directly within the platform and can be applied to many different types of application testing without requiring separate tools or configurations. When TestComplete encounters unsupported or custom controls, OCR can be used to: Recognize text from a specified screen region Extract and compare visible text against expected values Locate UI elements based on displayed text rather than coordinates Interact with visual elements by identifying their text content OCR actions can be recorded automatically during test creation when traditional object recognition is not possible. Teams can also explicitly define OCR based checkpoints to validate messages, labels, and dynamic values that appear during test execution. By allowing interactions to be driven by recognized text instead of fixed screen positions, OCR based tests tend to be more resilient to layout changes and UI adjustments. See OCR in Action A short demonstration shows how OCR is applied in real testing scenarios, including recognizing text in custom or unsupported controls, validating user visible messages, and driving interactions based on on screen text rather than fixed coordinates. The demo focuses on practical use cases where traditional object identification is not available. Expanding Automation Coverage Without Increasing Fragility One of the persistent challenges in UI automation is balancing coverage with maintenance. Scripts that rely on brittle locators or coordinates often fail when visual layouts change, even if the underlying functionality remains correct. OCR helps mitigate this issue by anchoring tests to user visible content rather than implementation details. This is particularly useful for: Validating alerts or error messages drawn directly on the UI Verifying values inside charts or graphical widgets Testing applications with frequent visual refinements but stable business logic By enabling validation at the visual layer, OCR reduces the need for workarounds or manual testing in areas that were previously difficult to automate. The result is broader coverage with fewer fragile dependencies. OCR as a Bridge Between User Experience and Automation OCR is not intended to replace traditional object based testing. Instead, it complements it by extending automation into areas where conventional techniques are insufficient. Within TestComplete, OCR functions as a bridge between how users experience an application and how automated tests validate it. When automation can read and verify the same information a human user relies on, test results better reflect real world behavior and risk. As applications continue to evolve toward richer and more visually driven interfaces, OCR plays a key role in ensuring automated testing remains aligned with actual user experience not just underlying code structure.Hover Button...
When using the TestComplete Desktop module for record‑based automation testing, there are cases where a button is not visible and therefore cannot be clicked. This happens when the button only appears on mouse hover. When the mouse is manually moved to that position, the button becomes visible, but during the recorded automated test the button does not appear and the test cannot proceed. What is the best way to hover over a button that is not visible during the test and then click it? <<= Look !!!
Testcomplete issue with Java17 ( Javax Swing applications)
My desktop Javax Swing applications were recently upgraded from Java 8 to Java 17, and since then I’ve been experiencing an intermittent issue where the main parent object is not recognized, causing repeated failures. Has anyone else encountered this when using TestComplete with Java 17?I am using TestComplete version is 15.76.5.7 (x64).
Test Complete does not locate my Desktop app
Hi everyone, when I run every test, Test Complete does not minimize, so it does not locate my desktop app, this behavior happens 7 or 8 times, then it does minimize and locate my desktop app and the test runs. I want to mention that (on my properties project, on Playback) I do have the option: Minimize Test Complete clicked, but even though it takes so much time to run correctly, so I have to wait 7,8 or sometimes even 10 times for Test Complete to minimize and run my tests. Is there any solution for this issue? Thanks in advance Best regardsObject Recognition Challenges? Help Us Understand Your Use Cases
Hello TestComplete Community! We’re working on a new Visual Object Detection (VOD) capability powered by AI, designed to make it easier to test applications where traditional object recognition struggles. This feature aims to help you reliably identify visual elements that are hard to detect today. To make sure we’re building this in the most useful and relevant way, we’d love to learn more about your current challenges and use cases related to visual or hard-to-recognize objects. Your real-world scenarios will help shape how we design and prioritize VOD. Feel free to share your thoughts by replying directly to this post, or send me a private message if you prefer. Thank you in advance for your feedback we truly appreciate it! Temil Sanchez Product ManagerBeating SAP Testing Bottlenecks with TestComplete
Testing SAP is hard for all the familiar reasons, complex UIs, transports that tweak screens, sensitive data, and heavy audit needs. Below are the common bottlenecks and how TestComplete helps you cut through them. Fragile object locators in SAP GUI The bottleneck: SAP GUI controls can be tricky to identify reliably minor UI changes or different dynpro states can break scripts. How TestComplete helps: It provides native support for SAP GUI for Windows with extended objects (buttons, edit fields, grids, etc.), so you work with properties and methods not coordinates. Pair that with Name Mapping (a central, alias-based object repository) to make tests readable and resilient. 2) UI drift after transports equals flaky tests The bottleneck: After a support pack or transport, object properties change and tests fail even though the flow still works. How TestComplete helps: Self healing tests automatically look for close matches when an object isn’t found, reducing “false red” failures and maintenance. 3) “Hard” screens, canvas elements, or remote sessions The bottleneck: Custom controls or canvases don’t expose stable object trees. How TestComplete helps: Use AI-powered OCR (and the OCR Action in Keyword Tests) to find text on screen and create easy validation as a fallback when classic object IDs aren’t reliable. 4) Test data sprawl (pricing, partners, plants…) The bottleneck: You need many variants to cover conditions, taxes, partners, plants, and languages without hand cloning tests. How TestComplete helps: Built-in data driven testing lets you drive one test with rows from Excel/CSV/DB, multiplying coverage while keeping scripts lean. 5) Audit evidence for SOX/GxP The bottleneck: Auditors want traceable, reviewable evidence: who ran what, where it clicked, and what was on screen. How TestComplete helps: Test Visualizer captures step-by-step screenshots during record/playback; Video Recorder can capture full-run videos; detailed logs tie everything together. These are ideal for defect triage and audits. 6) CI/CD traceability (and repeatability) The bottleneck: Manual runs don’t scale; teams need runs linked to commits/builds. How TestComplete helps: Use the Jenkins plugin to trigger suites in jobs or Pipelines and view results in Jenkins, creating a clean chain of custody for each build. Final thought SAP is always changing, your tests shouldn’t break every time it does. TestComplete’s native SAP GUI support, Name Mapping, self-healing, OCR fallback, and data-driven runs help you keep testing stable and audit friendly with less maintenance. The following demo illustrates these features in practice, automating the creation of a purchase requisition within SAP while maintaining stability across UI changes. Demo: Automating Purchase Requisition Creation in SAP with TestCompleteQT - Error No such interface supported
Has anyone been able to test modern QT applications 6.7 or 6.8 and get it to work. I cannot even get the simple example applications from QT to function. According to support currently 6.7 and 6.8 are supported, but neither seem to work. Just using the object browser shows error for most items, and none of the QT related methods are even displaying or accessible. See attached image. Any one have a solution to this? **I tried to upload a small zip file that contains the QT sample app, but the forum prevents it from being uploaded.
