The Ultimate Guide to Visual Regression Testing: 5 Tools to Try

Introduction 👀

Are you looking for the best visual regression testing tools to ensure your website or application looks and functions as intended? Look no further than our Ultimate Guide to Visual Regression Testing. In this article, we'll explore the top five tools you should try to streamline your testing process and catch any design or functionality issues before they impact your users. We will explore both free and paid options and go through the following tools and their respective use cases:

1. Playwright snapshots (Open Source)

2. Loki (Open Source)

3. Chromatic (SaaS)

4. Percy (Saas)

5. Lost Pixel (Open Souce + SaaS)

Playwright snapshots 🎭

Playwright is an open-source Node.js library for automating web browsers. Microsoft developed it and designed it to work with all major browsers, including Chrome, Firefox, and Safari. Playwright allows developers to write automated tests that interact with web pages, simulate user actions, and capture screenshots or videos of the browser window.

Playwright is a tool not explicitly designed for visual testing but offered utilities to streamline the whole visual testing process. Anytime in your E2E test run, you can await expect(page).toHaveScreenshot(); Ensure that your page looks exactly like you expect it to look at any given point in time.

Playwright offers various utilities to reduce the tests' flakiness and ensures you can configure the thresholds for maximum tolerable pixel differences in your tests. It comes in handy when your app relies on non-deterministic rendering and can have minor visual differences not distinguishable to the human eye and thus not relevant as negative cases.

Being an open-source tool, Playwright relies on your running the testing & baseline update process on your premises - either on CI or locally on a developer machine. This is both pros & cons, as you are missing a user interface to inspect the results of the visual tests, but at the same time, you are in complete control of your difficulties and don't need to use a third-party service to manage your visual tests. Being able to render practically anything, Playwright is a great tool to test your marketing pages, Storybook components or anything that can be caused in the browser. However, this would almost always require additional setup and tinkering.

Loki.js 👁️

Loki.js is an open-source visual regression testing tool for your Storybook components. It offers a low-maintenance workflow to test your Storybook components across different devices (React native supported). Although as robust as Playwright for the Storybook test, Loki.js is limited in power when trying your application pages or running a visual regression analysis after interacting with your front end. To achieve that, you would need to do some extra setup and probably add everything you want to test in Storybook.

Loki.js integrates perfectly with any of the CI providers, which makes it super easy to incorporate into your existing testing pipeline. It possesses the same limitation as the Playwright here, though, as you don't have an integrated user interface to work with the test results, and it usually boils down to comparing the images locally with a tool like Kaleidoscope. Although this workflow is robust and is used by many companies, the main struggle comes from blocking the developer machines to run the tests locally and producing the new snapshots, thus making it very costly to maintain. This also hinders collaboration between designers/engineers as the test results can be scattered across different user machines/ ci runs.

Chromatic 🌈

Chromatic is a SaaS visual regression testing tool designed to work with Storybook, a popular open-source tool for building UI components. Chromatic integrates with your existing Git workflow to automatically test and review UI changes in your components, ensuring they look and function as intended.

As we are delving more into cloud solutions, Chromatic is a clear step up in most aspects of the visual regression testing flow. It is easy to set up and integrates well with your Git flow, meaning you can be sure you don't accidentally release breaking visuals to your customers. Finally, it offers a UI that fosters collaboration and makes it very easy to review the test results.

The main limitation here is that being a Storybook tool, Chromatic excels at providing utilities for working with Storybook but does not make it easy to run visual tests on your pages, automated tests and other use cases where they span beyond Storybook.

Percy 🦔

Percy is a SaaS visual regression testing tool that allows you to automate your visual testing process. It integrates with your existing testing framework and captures screenshots of your web application or website to detect visual changes.

Percy offers a user-friendly interface to manage your visual tests and provides a collaborative environment for designers and developers to work together on UI changes. It also provides features like version control, commenting, and collaboration tools to streamline your testing process.

Being one of the oldest tools on the market, Percy covers a lot of use cases. Therefore, he can easily integrate both into your marketing pages, Storybook, and other visual testing use cases. In addition, it is owned by BrowserStack, so it logically offers to test the magnitude of different operating systems, devices and viewports. Percy being held by a large testing firm spawns some cons, though - it is one of the most expensive tools on the market. In addition, it is usually bulkier in setup & maintenance compared to other tools mentioned in this article.

Lost Pixel 🖼️

Lost Pixel is a combination of both open-source and SaaS offerings. It uses a lost-pixel open-source engine that resembles Loki.js in the easiness of setup and maintenance but covers all of the use cases for visual testing(SStorybook, application pages, E2E test) referred to as Holistic Visual Regression Testing and offered Lost Pixel Platform, which integrates into your existing Git workflow and enables collaboration through UI for reviewing the test results.

It's seldom a case that you want to just unit test your components, Lost Pixel becomes a single source of truth for your visual regression testing strategy. The Holistic Visual Regression Testing concept allows you to combine the results of your visual tests into a single run and achieve even more confidence when developing your UIs. For example, imagine a CI run that runs your Storybook tests, then covers a dedicated set of marketing pages that use those components, finishes with interaction tests, and snapshots the results into a single place. Luckily it is an easy setup with Lost Pixel.

A potential drawback of Lost Pixel & Lost Pixel Platform is its native integration with GitHub only(no support of GitLab or other git providers).

Summary 📊

All five tools have their pros and cons. Playwright and Loki are open-source tools that offer utilities to streamline the visual testing process, but they have limitations. Chromatic and Percy are SaaS tools that are easy to set up and provide user-friendly interfaces, but they are limited to testing Storybook components and can be expensive.

Lost Pixel combines open-source and SaaS offerings that provide a Holistic Visual Regression Testing approach. It integrates with Git workflow to test E2E, marketing pages, and Storybook components, making it the best option for engineers striving to achieve the best results with little maintenance effort.

Summary table 📖