Working with Match Levels

Traditional pixel-to-pixel comparisons are incredibly flaky since an exact match is needed for a test to pass. Other tools offer thresholds that indicate how many pixels need to be different for the test to fail but this is still unreliable and arbitrary. Traditional pixel-to-pixel comparisons are also very noisy and it can be hard to spot what exactly on the page is a visual bug vs. something that's just shifted slightly.

Applitools provides several match levels that allows you to control how the Applitools Visual AI will compare baseline and checkpoint images.

Exact Match Level

Basic pixel-to-pixel comparison, this should only be used for testing purposes and not in production.

Strict (default)

The default match level that uses AI to mimic the human eyes

Ignore Colors

Same as Strict but will not fail tests if colors change.

Layout

Compares the overall structure and relative positioning of elements on the page; great for dynamic content.

When to use Match Levels

In this guide, we'll go over each match level and show you an example of when you should use each one. We'll also cover match level regions which allow you to combine multiple match levels in a single test for different components of your page.

Strict Match Level

The Strict match level takes traditional pixel-to-pixel comparisons to the next level. It analyzes the differences between baseline and checkpoint images and will only fail the test if the differences found are discernable to the human eyes.

Strict should be used for the majority of your tests as it's the most reliable method of capturing visual bugs.

Example - Strict Match Level

Here is a page that we'd like to visually validate:

Let's say our page was updated and a 3rd input box was added to the form. Here's what the visual comparison looks like with the Exact pixel-to-pixel match level:

Here's what the visual comparison looks like with the Strict Visual AI Match Level:

As we can see, Strict blocks out the noise and zeroes in on what really changed on the page allowing you to easily analyze your test results and decide if there's a visual bug or an expected change.

Setting Match Level in the SDKs

Learn how to set the match level for your SDK in our documentation.

Layout Match Level

The Layout match level is used for testing dynamic content where text or images change frequently but the structural layout of the page remains the same. This match level identifies the structural changes in the layout regardless of the content. Therefore, the tool will not detect differences if, for instance, the news headline or list of products changes. This is a perfect match level for pages with a lot of variable content such as news portals, blogs, or e-commerce sites where product listings are changing regularly.

Example - Layout Match Level

Below is an example of a page where we have dynamic content, specifically, the balances of the accounts can change on each visit:

Here is what the comparison looks like when using the Strict Match Level:

As we can see, there's a ton of differences because the balances at the top changed and the dates and amounts in the table at the bottom changed as well. Assuming our data is dynamic and changes on each visit, our test will fail every time if we use the Strict Match Level.

Here is what our test result looks like with the Layout Match Level:

As we can see, this test passes and no differences have been flagged. The Applitools Layout Match Level ignored all of the dynamic data changing and prevented our test from failing unnecessarily.

When to Use Ignore Colors Match Level

The Ignore Colors match level behaves exactly the same as the Strict match level except any color changes will be ignored. This match level can be used whenever you have a component that changes color dynamically. For instance maybe you have an icon that changes color based on the time of day or day of the week.

Example - Ignore Colors Match Level

Let's look at an example where our navigation bar changes colors. Here's the comparison using the Strict algorithm:

Here is what that same comparison looks like when using the Ignore Colors Match Level:

Combining Match Levels

The best practice that we recommend when working with Match Levels is the following:

1. Use the Strict match level as the global match level
2. Add Layout and/or Ignore Colors regions for specific components on the page

Strict is the best match level for ensuring maximum coverage. Therefore, we highly recommend taking advantage of this and using Strict in conjunction with Layout and Ignore Colors regions.

To highlight this, take our example earlier where we used the Layout match level globally to handle the dyanmic content on our page:

We saw how no differences were thrown since the Layout match level handles dynamic data as expected. However, since we're not using Strict, other visual bugs could pass through our test:

Here we can see that no differences were flagged even though we had a visual bug in our sidebar on the left as well as the background color of our navigation bar changed. If we simply changed back to the Strict match level, we would get many differences for the dynamic data as we saw earlier.

The Solution:

• We need the Layout match level for the dynamic data
• Strict match level to capture unexpected visual bugs that are separate from the dynamic data
• Ignore Colors match level for our navbar that changes colors

We can accomplish this by using the Strict match level globally, wrapping Layout match level regions around our dynamic data and wrapping an Ignore Colors match level region around our nav bar. Here is the result:

As shown above, we successfully flagged the visual bug in the side bar since we used the Strict match level globally. We also handled the color changes and the dynamic data by using Ignore Color and Layout regions for those sections.

Adding Match Level Regions

Learn how to add match level regions to your tests in our documentation.

Conclusion

Applitools match levels offer a distinct approach to testing which leverages Applitools' intelligent visual comparison capabilities. Understanding when to use, and in what combination, can greatly enhance the power and accuracy of your visual regression tests. Thus, giving you robust and precise test results to ensure the visual integrity of your application.