OCR in UI Test Automation: Extending Coverage Where Traditional Identification Breaks Down
Automated UI testing increasingly operates in environments where traditional object identification is not reliable. Modern applications frequently render text and controls using custom graphics, canvases, charts, and dynamically generated visuals that do not expose accessible properties or stable locators. As a result, automation tools that rely solely on object hierarchies and properties can struggle to validate what is actually presented to the user. Optical Character Recognition (OCR) addresses this gap by enabling automation to extract and interpret text directly from what is rendered on screen. Instead of depending on the underlying implementation of a control, OCR works at the visual layer, analyzing pixels and patterns to recognize characters and convert them into machine readable text. This capability allows automated tests to validate user visible content in situations where traditional approaches fall short. Why OCR Matters in Real World UI Testing In many business critical applications, text is not always exposed through standard UI controls. Common examples include: Charts and dashboards rendered using custom drawing libraries Canvas based interfaces and rich graphical components Embedded documents such as PDFs or reports Custom buttons, labels, or alerts built without standard accessibility metadata In these scenarios, the risk is not just test fragility it is blind spots. If automation cannot confirm what text is displayed, teams are forced back to manual validation for critical user facing information. OCR enables tests to verify visible content regardless of how it is implemented. By converting visual text into actionable data, OCR allows teams to assert that values, labels, messages, and statuses shown to users are correct, even when object level access is unavailable. This makes OCR especially valuable for validating end-to-end business workflows where correctness depends on what users actually see, not just what the application internally represents. OCR as Part of TestComplete’s Object Recognition Strategy TestComplete incorporates OCR as part of its broader approach to handling complex and non standard user interfaces. OCR is available directly within the platform and can be applied to many different types of application testing without requiring separate tools or configurations. When TestComplete encounters unsupported or custom controls, OCR can be used to: Recognize text from a specified screen region Extract and compare visible text against expected values Locate UI elements based on displayed text rather than coordinates Interact with visual elements by identifying their text content OCR actions can be recorded automatically during test creation when traditional object recognition is not possible. Teams can also explicitly define OCR based checkpoints to validate messages, labels, and dynamic values that appear during test execution. By allowing interactions to be driven by recognized text instead of fixed screen positions, OCR based tests tend to be more resilient to layout changes and UI adjustments. See OCR in Action A short demonstration shows how OCR is applied in real testing scenarios, including recognizing text in custom or unsupported controls, validating user visible messages, and driving interactions based on on screen text rather than fixed coordinates. The demo focuses on practical use cases where traditional object identification is not available. Expanding Automation Coverage Without Increasing Fragility One of the persistent challenges in UI automation is balancing coverage with maintenance. Scripts that rely on brittle locators or coordinates often fail when visual layouts change, even if the underlying functionality remains correct. OCR helps mitigate this issue by anchoring tests to user visible content rather than implementation details. This is particularly useful for: Validating alerts or error messages drawn directly on the UI Verifying values inside charts or graphical widgets Testing applications with frequent visual refinements but stable business logic By enabling validation at the visual layer, OCR reduces the need for workarounds or manual testing in areas that were previously difficult to automate. The result is broader coverage with fewer fragile dependencies. OCR as a Bridge Between User Experience and Automation OCR is not intended to replace traditional object based testing. Instead, it complements it by extending automation into areas where conventional techniques are insufficient. Within TestComplete, OCR functions as a bridge between how users experience an application and how automated tests validate it. When automation can read and verify the same information a human user relies on, test results better reflect real world behavior and risk. As applications continue to evolve toward richer and more visually driven interfaces, OCR plays a key role in ensuring automated testing remains aligned with actual user experience not just underlying code structure.Stop Skimming PDFs, Start Automating PDF Testing
On the surface, PDFs look simple, but testing them is a whole different story. Invoices, contracts, statements, compliance reports… they’re often the last thing that lands in a customer’s hands. That also means even the smallest issue, like a missing field or a misplaced decimal, can turn into something big. The challenge is that PDFs aren’t like web pages or apps where you can easily inspect elements. They’re containers packed with content, layout, images, and data from different systems. When you add in dynamic content that changes for every customer, formatting that has to stay perfect, and the regulatory risks in industries like finance or healthcare, you start to see why manual testing just doesn’t cut it. It’s slow, inconsistent, and doesn’t scale. This is where automation becomes essential. With automation, you can make sure data is always accurate, layouts stay consistent, and testing scales across thousands of documents without slowing down your team. Instead of spending hours opening PDFs by hand, QA can focus on higher-value work while still knowing that every report or statement going out the door is right. That’s exactly where TestComplete comes in. It’s built to handle the tough parts of PDF testing so you don’t have to. You can validate content down to the last character, run visual checks to keep layouts consistent, and plug it all straight into your CI/CD pipeline. The result is faster releases, fewer headaches, and a lot more confidence that the documents your customers see are exactly as they should be.Beating SAP Testing Bottlenecks with TestComplete
Testing SAP is hard for all the familiar reasons, complex UIs, transports that tweak screens, sensitive data, and heavy audit needs. Below are the common bottlenecks and how TestComplete helps you cut through them. Fragile object locators in SAP GUI The bottleneck: SAP GUI controls can be tricky to identify reliably minor UI changes or different dynpro states can break scripts. How TestComplete helps: It provides native support for SAP GUI for Windows with extended objects (buttons, edit fields, grids, etc.), so you work with properties and methods not coordinates. Pair that with Name Mapping (a central, alias-based object repository) to make tests readable and resilient. 2) UI drift after transports equals flaky tests The bottleneck: After a support pack or transport, object properties change and tests fail even though the flow still works. How TestComplete helps: Self healing tests automatically look for close matches when an object isn’t found, reducing “false red” failures and maintenance. 3) “Hard” screens, canvas elements, or remote sessions The bottleneck: Custom controls or canvases don’t expose stable object trees. How TestComplete helps: Use AI-powered OCR (and the OCR Action in Keyword Tests) to find text on screen and create easy validation as a fallback when classic object IDs aren’t reliable. 4) Test data sprawl (pricing, partners, plants…) The bottleneck: You need many variants to cover conditions, taxes, partners, plants, and languages without hand cloning tests. How TestComplete helps: Built-in data driven testing lets you drive one test with rows from Excel/CSV/DB, multiplying coverage while keeping scripts lean. 5) Audit evidence for SOX/GxP The bottleneck: Auditors want traceable, reviewable evidence: who ran what, where it clicked, and what was on screen. How TestComplete helps: Test Visualizer captures step-by-step screenshots during record/playback; Video Recorder can capture full-run videos; detailed logs tie everything together. These are ideal for defect triage and audits. 6) CI/CD traceability (and repeatability) The bottleneck: Manual runs don’t scale; teams need runs linked to commits/builds. How TestComplete helps: Use the Jenkins plugin to trigger suites in jobs or Pipelines and view results in Jenkins, creating a clean chain of custody for each build. Final thought SAP is always changing, your tests shouldn’t break every time it does. TestComplete’s native SAP GUI support, Name Mapping, self-healing, OCR fallback, and data-driven runs help you keep testing stable and audit friendly with less maintenance. The following demo illustrates these features in practice, automating the creation of a purchase requisition within SAP while maintaining stability across UI changes.
TestComplete detects the table control itself, but none of the internal row/cell objects are exposed
I have a table-like control similar to the one in the screenshot. Using "Add Object" in TestComplete, I can capture the main table object. (object: Aliases.Icad.dlgAECStylesManager.Item.SysListView32) However, when I explore it in the Object Browser, the table exposes only one child object — the header. There are no row objects, no cell objects, and no data nodes visible under the table. Because of this, TestComplete cannot interact with the table using the normal object model (such as row/cell access, FindChild, ClickCell, etc.). My goal is to automate editing based on the row index. For example: Update the “Name” column for the row where Index = 101 (change it to "Testing1"). Has anyone encountered this type of control or found a reliable solution for automating tables that do not expose row/cell objects? Any suggestions would be greatly appreciated. Thanks!From Confused to Confident: Master XPath for Web Testing
Introduction Learn how to confidently build, refine, and troubleshoot XPath expressions. This practical, example-driven guide covers TestComplete HTML and CSS basics, XPath syntax, and browser validation techniques, helping you create stable and maintainable automated web tests. While TestComplete can generate XPath or CSS selectors automatically, generated selectors can break if page layouts change, so knowing how to tweak XPath manually ensures your tests remain reliable even on pages that change frequently. HTML Basics W3Schools Reference HTML stands for HyperText Markup Language. It is the standard markup language for creating web pages. An HTML document can be represented as a Document Object Model (DOM), which is a tree of nested elements. Each element may have attribute name–value pairs that provide additional information about that element like its ID or class. An HTML element typically includes a start tag, content, and an end tag, for example: <tagname>Content goes here...</tagname> Save this simple example as a file named "index1.html" and open it in a browser, or paste it in W3Schools Tryit Editor left pane section. The <img> tag’s src may point to any valid image file or a valid URL. <!DOCTYPE html> <html> <head> <title>Page Title</title> </head> <body> <h1>This is a Heading</h1> <p>This is a paragraph.</p> <a href="https://www.w3schools.com">This is a link</a> <p>This is another paragraph.</p> <img src="w3schools.jpg" alt="W3Schools logo" width="104" height="142"> </body> </html> CSS Basics W3Schools Reference CSS stands for Cascading Style Sheets and defines how HTML elements are displayed color, size, spacing, and layout. CSS rules consist of a selector and a declaration block, which contains property–value pairs, for example: h1 { color: blue; font-size: 24px; } Save this simple example as a file named "index2.html" and open it in a browser, or paste it in W3Schools Tryit Editor left pane section. <!DOCTYPE html> <html> <head> <style> body { background-color: yellow; } p { color: red; margin-left: 100px; } </style> <title>Page Title</title> </head> <body> <h1>This is a Heading</h1> <p>This is a paragraph.</p> <a href="https://www.w3schools.com">This is a link</a> <p>This is another paragraph.</p> <img src="w3schools.jpg" alt="W3Schools logo" width="104" height="142"> </body> </html> CSS selectors are different from XPath selectors, and TestComplete can use both depending on the browser engine. XPath Basics W3Schools Reference XPath stands for XML Path Language. It uses a path-like syntax to identify and navigate nodes in an XML or HTML document. Now that we’ve reviewed how web pages are structured, let’s see how XPath helps locate elements within them. In web testing, XPath is commonly used to locate elements within the HTML DOM. Important points to keep in mind: HTML <tags> and element names aren’t case-sensitive are normalized to lowercase, but attribute values are and usually enclosed in quotes. XPath works the same in HTML as in XML and navigates elements using relationships: Ancestors, Descendants, Parent, Children, and Siblings. Absolute Path: Starts from the root with a single slash "/", for example: "/bookstore". Absolute XPath is brittle because page layouts often change. Relative Path: Starts with double slashes "//" and searches anywhere in the document, for example: "//book". Consider this XPath Expression: "//div/a[contains(@class, 'button')]/@href". Let’s break it down step by step: 1. // = Search all descendants in the document 2. div = Select any "<div>" element 3. / = Navigate to direct child 4. a = Select the "<a>" child element 5. [] = Apply a condition 6. contains(@class, 'button') = Select "<a>" elements whose "class" attribute value contains "button". The contains() is very useful for partial matches. 7. / = Navigate to the next level 8. @href = Returns the value of the "href" attribute. In Plain English: Find all "<a>" links inside "<div>" elements with "button" in the class attribute and return their "href" values. Understanding XPath expressions helps handle dynamic pages, where IDs, classes, or structures may change across builds or browsers. You could also use wildcards and parameters. Testing XPath in Chrome & SmartBear Sample Shop Browser Developer Tools (DevTools) among other things are used to debug the HTML DOM and examine individual elements. Press "F12" to open DevTools. When DevTools is docked to the right (the default), the page layout may adjust dynamically. To maintain the original layout, and for this exercise switch the DevTools dock position to the bottom. SmartStore’s dynamic structure may vary and actual results may differ if the site layout changes. The Console panel in DevTools lets you execute XPath expressions directly in the browser to validate their syntax and confirm that they correctly identify elements within the DOM. The $x("...") is Chrome-specific console command function and is also not part of TestComplete. Type your XPath inside the command. Step-by-step example On the SmartStore Page click on the header "Help & Services", then right-click on the menu item "About Us" and select "Inspect". The Elements panel in DevTools will expand and highlight the corresponding HTML DOM element. Perform the following steps in the Console Panel: Type $x("") and notice the console displays "undefined". Type $x("//a") and notice the console displays "(109)" meaning there are 109 matching <a> elements, stored in an array. Type the full XPath $x("//a[span[text()='About Us']]") and notice the status changes to (2) signaling two matches were found. Note: If your target text is not inside a <span>, adjust to $x("//a[text()='About Us']"). Press Enter to execute the XPath expression and then expand the results to hover your mouse over the first entry "0: a.dropdown-item.menu-link", the element in the SmartStore page gets highlighted. Similarly, in the Elements panel, hovering over the element highlights it in the SmartStore page. Right-click on "0: a.dropdown-item.menu-link" and select "Open in Elements panel". This navigates directly to the HTML DOM element in the Elements panel. Scroll the SmartStore page to the bottom, and in the Elements panel, right-click on the element and select "Scroll into view". The SmartStore page will scroll back up and the element on the page will be highlighted. Right-click on the element and select "Copy" > "Copy XPath", "Copy full XPath", "Copy selector", or "Copy Element". You can paste into any editor for review. The "Copy XPath" generates an absolute path, while the "Copy full XPath" is fully qualified from root, both can break if the DOM structure changes. Repeat the steps for the second match "1: a.menu-link" to reinforce your understanding. Refining XPath for precision Option 1: By adding attribute filters with name–value pairs: view the element details and update the XPath $x("//a[span[text()='About Us'] and @class='dropdown-item menu-link']") Option 2: A more robust and preferable approach is to reference an ancestor in the XPath. Copy both elements' full XPath and paste them in a text editor for visual comparison. /html/body/div/div[2]/header/div[1]/div/nav/nav[2]/div/div/div/a[5] /html/body/div/div[2]/footer/div[2]/div/div/div[3]/nav/div/div/ul/li[1]/a The full XPath is a brittle locator that breaks with layout changes. Find the top ancestor and refine the XPath, start from a logical ancestor like “//header//a…” or “//footer//a…” and continue with descendant selectors, not direct children. The double slashes (//) mean any descendant level, not necessarily a direct child. Don’t worry if your XPath doesn’t work the first time—this is normal! $x("//header//a[span[text()='About Us']]") $x("//footer//a[span[text()='About Us']]") When you copy the full XPath from the browser and try to run it, it may not work. Example of a common mistake that will result in such error: "Uncaught SyntaxError: missing ) after argument list". Why: The double quotes (") inside the XPath conflict with the double quotes wrapping the string in JavaScript or TestComplete scripts. Use single quotes (') inside your XPath string to prevent conflicts with the outer double quotes used in the scripting environment. Incorrect: $x("//*[@id="header"]/div[1]/div/nav/nav[2]/div/div/div/a[5]/span") Correct: $x("//*[@id='header']/div[1]/div/nav/nav[2]/div/div/div/a[5]/span") Common Mistakes & Troubleshooting If your XPath expression doesn’t yield expected results, refresh the page or ensure you’re on the correct frame, and check for dynamic rendering (elements loaded after page load). Also try waiting for the element or using relative paths. Check whether the element is inside an iframe — in that case, switch context before applying XPath. Conclusion XPath can initially seem confusing, but with patience and practice, you’ll confidently tackle a wide range of web element identification challenges. Mastering XPath will also help you debug TestComplete object recognition issues more effectively. TestComplete’s flexibility, combined with a solid understanding of XPath, allows you to create robust, maintainable, and reliable automated tests. XPath skills also translate to CSS selectors and other object identification methods, improving cross-technology automation. Start experimenting with simple examples today, you’ll be surprised how quickly manually manipulating XPath becomes your favorite action!Object Brower : annoying behavior
Hi everyone ! I have a very annoying issue. Sometimes, the object browser load the DOM in the same manner than "legacy features". The normal behavior is this : Custom Angular components are properly detected, like "AppSignRoot" in this example. But sometimes, the tree look like this : Custom components are not properly detected. Here, the "AppSignRoot" node is "Panel(4)". If I hit the refresh button, everything gets back to normal like the first screenshot. But, calling the Refresh() method on the page object or browser object (in my script and / or test item) does nothing. The DOM is randomly not detected properly. And I'm out of solutions. Any idea ?Object Recognition Challenges? Help Us Understand Your Use Cases
Hello TestComplete Community! We’re working on a new Visual Object Detection (VOD) capability powered by AI, designed to make it easier to test applications where traditional object recognition struggles. This feature aims to help you reliably identify visual elements that are hard to detect today. To make sure we’re building this in the most useful and relevant way, we’d love to learn more about your current challenges and use cases related to visual or hard-to-recognize objects. Your real-world scenarios will help shape how we design and prioritize VOD. Feel free to share your thoughts by replying directly to this post, or send me a private message if you prefer. Thank you in advance for your feedback we truly appreciate it! Temil Sanchez Product Manager
