Introduction 📚
Visual regression testing is a crucial aspect of software development that ensures the user interface remains consistent across different versions of an application. However, if you are new to this field, the terminology and jargon used in visual regression testing may be overwhelming. That's where this visual regression testing glossary comes in handy! In this article, we'll define and explain the most commonly used terms in visual regression testing so you can quickly become a pro.
Visual Regression Testing 👁️
Visual regression testing(commonly referred to as VRT, Visual testing or Visual snapshot testing) is a technique used to compare screenshots of an application's user interface to detect any visual changes that may have occurred between different versions of the application. This technique ensures that the user interface remains consistent after the code changes. Usually, VRT runs as the last part of your CI/CD pipeline closing the whole test suit and ensuring that your Frontend looks precisely like it should.
Shots 📸
In visual regression testing, a shot(also referred to as a visual snapshot) refers to a screenshot of an application's UI. Shots are used to compare the user interface between different versions of the application to detect any visual changes that may have occurred. Shot is a generally accepted unit of measure for visual regression testing SaaS.
Baseline images 🖼️ 🖼️
Baseline images(a source of truth or baseline snapshots) are reference screenshots of an application's user interface captured during the initial testing phase. These images are used as a reference point for future tests to detect any visual changes that may have occurred between different versions of the application.
Usually, baseline images are manually approved by people responsible for maintaining the VRT. However, on the first run of your chosen testing suite, you must approve all your images - those images will now become baselines.
Visual regression testing flow 🖼️ 🔄
Visual regression testing flow refers to capturing baseline images, running tests to detect any visual changes, comparing the results to the baseline images, and updating the baseline images in cases with intended changes. This process is repeated for each code change to ensure that the user interface remains consistent.
Visual difference ⛰️ ↔️🏔️
Visual difference refers to the changes between two shots of an application's user interface. These changes can be detected using the visual regression testing techniques mentioned above. Although not all visual differences are Visual Bugs, sometimes the changes are intended, and a baseline update is required.
Visual bug 🖼️ 🪲
A visual bug is a defect in an application's user interface detected and caught by VRT. Here are a couple of good examples of visual bugs:
Text overlapping or not aligning correctly with other elements on the page
Images not displaying correctly or appearing distorted
Visual bugs are the exact reason we are running VRT. We aim to catch those bugs early in our release pipeline and ensure that end customers never face a breaking UI.
Flaky visual tests ❄️
The term "flaky" refers to tests that produce inconsistent results. For example, in visual regression testing, flaky visual tests are tests that sometimes pass and sometimes fail, even though the application's UI has not changed. Various factors, such as network issues, browser inconsistencies, or timing issues, can cause flaky visual tests. Therefore, it is essential to identify and fix flaky visual tests to ensure that your visual regression testing results are consistent and reliable.
Luckily modern VRT tools provide a range of tools and practices to battle flaky tests and make the VRT easily one of the most valuable parts of testing on CI/CD.