Puppeteer vs Playwright vs Selenium vs Cypress
The first thing you should start creating your framework with is choosing a browser interaction engine. It is the engine that will become the cornerstone of your tests in the future. Therefore, the choice of engine is the foundation of the house that you build (and then this foundation cannot be remade by third-party libraries).
Chapter 1. Choosing an Engine
At the moment, there are 4 popular ways to work with the browser: Selenium, Puppeteer, Cypress and Playwright. Below is a comparison table of these engines.
|
Developer |
|
Microsoft |
||
|
Project started |
2004 |
2012 |
2015 |
2020 |
|
Which browsers support |
Chrome, Firefo, Edge, Internet Explorer, Opera, Safari. |
Chrome-family browsers (Electron, new Edge). |
Chrome-family browsers (Electron, new Edge), Firefox. |
Chrome-family browsers (Electron, new Edge), Firefox, WebKit. |
|
Code |
All open source |
All open source |
There are paid tutorials |
All open source |
|
Programming language |
Java,Python,C#,Ruby,JavaScript. |
Java,Python,C#,Ruby,JavaScript. |
JavaScript,TypeScript. |
Java,Python,C#,JavaScript,TypeScript. |
|
Explicit network control |
No |
No |
Represents a context-wide network hook to stub and simulate network requests. |
Represents a context-wide network hook to stub and simulate network requests. |
|
Headless mode for browsers |
Not all drivers support headless mode. |
Supports headless mode. |
Supports headless mode. |
Supports headless mode. |
|
OS |
Need Mac for Safari and Windows for IE (Edge). |
Work on 3 platforms (Windows, Mac, Linux). |
Work on 3 platforms (Windows, Mac, Linux). |
Work on 3 platforms (Windows, Mac, Linux). |
|
Principle of operation |
In Selenium WebDriver you have three processes: |
What is good about Puppeteer?It runs a headless browser and uses the DevTools protocol, so tests are faster and more stable compared to Selenium. |
The code you write in your Cypress test scripts doesn't run outside of the browser like it does in WebDriver. The browser launches it. In fact, it executes your test code and the code of the application you are testing. |
Tests written with Playwright run in a clean slate sandbox called browser contexts. |
|
Average execution time (in seconds) |
3.66 |
2.22 |
10.35 |
3.19 |
|
Standard deviation (in seconds) |
0.65 |
0.41 |
2.67 |
1.23 |
Chapter 2: How will we launch?
|
OS |
Need Mac for Safari and Windows for IE (Edge). |
Work on 3 platforms (Windows, Mac, Linux). |
Work on 3 platforms (Windows, Mac, Linux). |
Work on 3 platforms (Windows, Mac, Linux). |
The first way to run tests is to create an agent and run tests on it. You yourself create an agent (machine) with the desired OS and run tests on it. The agent can be a local computer.
|
Agents |
We have to create two agents, one is Windows, the other is Mac (if we want to use Safari and IE browsers) |
You can create only one agent (Windows, macOS, Linux). |
You can create only one agent (Windows, macOS, Linux). |
You can create only one agent (Windows, macOS, Linux). |
|---|---|---|---|---|
|
Running in parallel on different browsers. |
We have to use the Selenium grid if we have different browsers. |
Just run. |
Just run. |
Just run. |
The second way: cloud service. Connecting to a third-party server with browsers and using them to run autotests.
|
What are there? |
A lot of servers support Selenium. |
There are definitely BrowserStack and Sauce Labs and TestingBot. |
There are definitely BrowserStack and Sauce Labs and TestingBot. |
There are definitely Sauce Labs and TestingBot (in theory, it supports remote connection. |
|---|---|---|---|---|
|
Do I need to pay? |
Yes |
Yes |
Yes |
Yes |
Summary:
-
If testing on IE or genuine Safari is very important to you, Selenium is the only one for you - it's a centuries-old, time-tested product. If you have chosen Selenium, then I advise you to use cloud servers. Otherwise, all the work to update WebDriver and the browser will fall on your shoulders.
-
If cross-browser compatibility is important to you, but you refused to support IE (it will cease to be supported in 2021-2022), and you also agree to the WebKit engine instead of Safari, then feel free to take Playwright. This is a fairly young product, but rapidly developing. It's better, of course, to keep your own agent because it will be cheap (you can buy a headless Linux agent). Browser updates go inside the Playwright package. In fact, these browsers may not be installed on the machine, they run from the Playwright box.
- If cross-browser compatibility is not important to you and you can run tests on one browser, then take Puppeteer. It is the fastest, and also chromium, at the moment, the most stable browser.
- If it is important for you to do Network mocking in tests, then use Cypress, there are more ready-made scripts than on Playwright. But! This solution is only suitable for those who are friends with JS.
Chapter 3: What's Next?
Select Test Runners.
The next steps after choosing an Engine should be the step of choosing Test Runners. A framework that will organize your tests. Test runners depend on the programming language you are writing in. For JS, you can compare the 3 most popular ones: Jest, Mocha and Jasmine.
In automation, I did not notice much difference in the use of one or another runner, but I still note one point:
Two popular contenders are Jest and Mocha. Jest is an open source testing framework developed by Facebook. Built into the popular create-react-app package, it's faster and more convenient for writing idiomatic JavaScript tests. It has built-in simulation and assertion capabilities when running tests in parallel, which provides a smoother and faster test run. One of the unique features of Jest is that it provides snapshot testing for complete control over the user interface. Mocha provides developers with a basic testing environment with options such as assertion, mocking, and spy libraries. It is one of the most flexible JavaScript testing libraries. A small disadvantage of Mocha is the need for additional configuration. Chai, the most popular open source assertion library, is used with Mocha
Epilogue
Wrapper
The third stage of preparation for automation should be the choice of a framework - a wrapper for the engine. For example WebdriverIO (for Selenium) or CodeceptJS (for Selenium or Puppeteer). The choice of wrapper should also be made by weighing the pros and cons. This is a topic for the next article.
Among other things, there are also visual frameworks. The most popular are Percy and Applitools. They will add a beautiful screenshot comparison to your application if you need it.
The Cypress and Playwright engines can be used without additional wrappers, they have excellent descriptions of the functions.
Selenium and Puppeteer should always be wrapped so as not to reinvent the wheel.