Tricentis Vision AI Explained: Smarter, Visual Test Automation

What is Tosca Vision AI?

Tosca Vision AI combines computer vision and AI technology to enable faster, more accurate, and more adaptable testing processes. Whether you're working on web, mobile, or desktop applications, Tosca Vision AI leverages AI to recognize visual elements in the application interface and perform automated tests, mimicking human-like interactions.

Tosca Vision AI uses AI-driven algorithms to identify dynamic UI elements such as buttons, text fields, and images, based on their appearance rather than static attributes. This approach not only enhances test coverage but also allows for efficient testing of applications with dynamic or frequently changing interfaces.

Core Capabilities of Vision AI 

1. Image-Based Element Recognition:

Vision AI identifies UI elements based on how they appear on the screen, and it identifies elements even when the properties 

2. Cross-Platform Support:

It enables testing across desktop, web, mobile, virtual, and remote environments, including SAP GUI, and VDI systems.

3. AI-Powered Resilience:

Uses machine learning models to adapt to changes in UI layout, styling, or content, reducing test maintenance. Vision AI can still recognize element even if UI changes but the element looks visually similar.

4. Codeless Test Creation:

Maintains Tosca's model-based, codeless test automation approach. Testers can simply point and click to capture visual elements.

5. Self-Healing Capabilities:

If an element is not found in its original location or form, Vision AI can intelligently locate and interact with the most likely match.

Setting up Vision AI

Tricentis Vision AI is part of the Tricentis Tosca standard installation. To enable Vision AI, we need to select Tricentis Vision AI option during the Tosca commander installation.

You can manage Vision AI users via the cloud-based Tricentis User Administration.

Tricentis User Administration in the cloud offers the same functionality as the server-based Tricentis User Administration.

Scan Controls using Vision AI

In Tosca Commander, Go to Modules->Scan->Application, right-click the application to scan and select Vision AI from the context menu. Click Scan, select controls and save the modules.

Tricentis Tosca starts the Vision AI Agent automatically when we choose Vision AI option from context menu.

The Vision AI Agent doesn't close automatically when you close Tricentis Tosca. To close it, go to your system tray, right-click the   Vision AI Agent icon and select Exit.

Vision AI recognizes several different types of controls. The steering mechanism varies from type to type. For optimal steering, you can adjust the control type to best match your actual control.

ControlType, Lebel and TreatAs are the mostly used technical properties to identify the controls using vision AI:

- For the ControlType property, select a new control type from the drop-down list. XScan displays only the control types that Vision AI detected for this specific control.

- For the TreatAs property, select the control type that you want to use (ex: dropdown, button, Input, List etc)

- For the Label property, change the label of a control to match the actual label.

Apart from standard table steering parameters for TBox tables, Vision AI provides additional table steering parameters that enable you to detect and steer tables more efficiently.

To adjust table parameters, in the Identify by Properties pane, you can modify the following additional parameters:

- Method

- ColumnThreshold and RowThreshold

- MergeEmpty

- HeaderColumn and HeaderRow

- HeadersContain

- DecisiveColumns

- ColumnMinGap and RowMinGap

Create User-Identified Controls

You can teach Vision AI to work with controls that are not recognized correctly, for example if a table is not recognized as a table. To do so, use the feature User Identified Controls (UIDCs). This feature enables Vision AI to recognize a control correctly using surrounding controls as anchor points. Vision AI uses these anchors as reference points to identify the control.

To create a new UIDC:

- Go to Modules->Scan->Application and select Vision AI in the application window.

- Select Advanced view and right-click the main item and select ‘Create User Identified Control’ from the context menu.

- Vision AI opens the Vision AI assistant window and the preview of your application.

- Draw a box around the control that you want to identify.

- Click on the controls that you want to use as anchors. Anchors teach Vision AI to recognize a UIDC correctly. You can select anchors that are inside and outside of the UIDC. These controls appear in the Vision AI assistant window in the Anchor points section.

- If you select only one control as an anchor, Vision AI always returns the UIDC with the same size and position in relation to the anchor control. In case the anchor control in the application under test is bigger or smaller, the UIDC might not be recognized correctly. Tricentis recommends using text labels as anchors, that appear only once on the screen. For variable sized controls, use more than one anchor at each side or corner of the control. Specifying anchors on the left and right will allow the control to expand horizontally, and anchors on the top and bottom will allow the control to expand vertically. Selecting anchors inside the control is also allowed.

- Now save it and rescan the application. This control is now recognized correctly based on the parameters that you specified in the Vision AI assistant window.

The UIDCs info is stored on your company's Nexus container. To see all UIDCs created in your company, go to https://vision-ai.app.tricentis.com/ and click on Configuration and select Controls from the left navigation pane. 

Self-healing

Self-healing is a feature in Tosca that enables test automation to adapt to changes in the application under test, preventing test failures and reducing the need for manual intervention. When a control (like a button, text box, etc.) changes, Tosca's self-healing capabilities try to find a similar control in the system under test, allowing the test to continue execution.

Vision AI allows you to enable self-healing of every control that can't be found when you run your Execution List.

Enable self-healing of Vision AI Test Cases

- Create a Recovery Scenario in your Vision AI Testcase.

- In the Properties tab of your Recovery Scenario, set the Retry Level to TestStepValue.

- Use the Module Vision AI Self-Healing to create a Test Step in the Recovery Scenario. You can find this Module at Standard modules->TBox XEngines->Vision AI.

- Add the following test configuration parameters (TCPs) to the Testcase:

- OnDialogFailure with the value Recover.

- TestStepValueRetries with the value 1.

- Run the Testcase from an Execution List.

Vision AI attempts to apply self-healing to any control that can't be found.

Conclusion

Tosca Vision AI offers an advanced, intelligent solution to test applications that are difficult to access through traditional testing methods. Its ability to adapt to dynamic interfaces, detect visual changes, and automate cross-platform tests makes it a powerful asset for any organization looking to streamline their testing processes while improving overall quality. Vision AI allows testers to define actions based on what they see rather than its properties. This is helpful for business users or testers without deep technical skills.