The key to test automation is ensuring that tests work well and provide helpful information. Test automation reports let teams check how their tests are performing, spot issues quickly, and make good choices for better results. Playwright, a powerful tool for browser automation, comes with several built-in reporters that help organize and clearly show the results of automated tests.

This blog will guide you through Playwright’s reporting features, starting with why test automation reports are important for your work. We’ll explore the different built-in Playwright reporters, like List, Line, Dot, HTML, JSON, and JUnit reporters, and how each one can meet different needs for viewing test results. We’ll also show how to set up multiple reports at once and how to use other reporting tools like Allure and Monocart.

Importance of Test Automation Reports

Automated test reports are crucial for understanding automated test results and improving the testing process. In Playwright, these reports give useful information about whether tests worked or didn’t, helping teams find problems early and fix them. Here’s why automated test reports are important in Playwright:

• Easy-to-understand Results: Reports show whether each test worked or didn’t, giving a clear idea of how well the application is doing. This is much faster than checking each test by hand, saving time and work.
• Detailed Test Information: Reports not only show if tests passed or failed but also give a full picture of how all the tests went. This helps everyone quickly see how good and stable the application is. They point out which tests didn’t work, which ones were skipped, and which ones did work, making it simpler to follow progress and quality.
• Debugging and Troubleshooting: When a test doesn’t work, Playwright’s reports give you clear details like error messages, step-by-step paths of what went wrong, and pictures of the screen at the time. This helps you find and fix the main problem quickly, making the whole debugging process smoother.
• Checking Test Coverage and Progress: Reports help you see how much of your app is being tested by your automated tests. They show which parts of the app have been checked and which ones still need testing. This makes sure all parts of the app are tested properly and helps you find any missing tests.
• Monitoring Performance Over Time: Regular test reports help you monitor how well the app is doing in different tests. By looking at the results over time, teams can notice patterns, like more frequent failures, and take action to fix potential problems before they become bigger issues.
• Data-Driven Decisions: Detailed test reports help teams decide what to focus on. Whether they’re looking at the effects of changes or choosing new features to test, these reports give them the data they need to make smart, data-based decisions.
• Sharing Information: Playwright lets you create test reports in different formats like HTML, JSON, and JUnit. This makes it simple to share the results with your team, stakeholders, or clients.
• Time and Cost Efficiency: Automated reports mean less need for manual checks of test results. They also help identify common problem areas, so teams can concentrate on important issues, saving time and lowering the overall cost of testing.
• Post-Test Review: Detailed reports help teams see patterns in what works and what doesn’t. This helps them find ways to make their test automation better, which leads to better testing and higher software quality.
• Meeting Rules and Standards: In some fields, keeping detailed test records is important for following rules. Playwright’s feature to create detailed, adjustable reports ensures that all needed documents are always current.

Built-In Playwright Reporters

• Playwright Test has several built-in reporters that can be used for different purposes, and it also supports creating custom reporters. The simplest way to use these built-in reporters is by using the –reporter option in the command line. For more advanced options and customization, reporters can also be configured directly in the project’s configuration file. Let’s take a closer look at both methods.

List Reporter

• The List Reporter is the standard tool for showing test results in Playwright. It gives a clear report of your tests in the terminal. It shows the name of each test and whether it passed, failed, or was skipped.

• Passed Test: Shows a green checkmark (✓) and the test’s name.
• Failed Test: Shows a red cross (✕), the test’s name, and the reason for the failure.
• Skipped Test: Shows a dash (-) and the test’s name.

Command Line: npx playwright test –reporter=list

Playwright Config:

• Instead of specifying the reporter in the command line, we can configure it under the reporter property in the playwright.config.js file.

const { defineConfig, devices } = require('@playwright/test');

module.exports = defineConfig({
  reporter: 'list',
});

Generated List Report

The List report below displays the results of 4 passed tests, 2 failed tests, and 3 skipped tests.

Line Reporter

• The Line reporter is simpler than the list reporter. It only shows the most recent test and mentions any failures as they happen. This is helpful for big test groups because it keeps you updated without filling the screen with every single test.

Command Line: npx playwright test –reporter=line

Playwright Config:

• Instead of specifying the reporter in the command line, we can configure it under the reporter property in the playwright.config.js file.

const { defineConfig, devices } = require('@playwright/test');

module.exports = defineConfig({
  reporter: 'line',
});

Generated Line Report

• The Line report below displays the results of 1 passed test, 1 failed test, and 1 skipped test.

Dot Reporter

• The Dot Reporter is a simple and short way to see the results of tests. It shows the status of each test with just one letter. This makes it a great choice if you want a quick overview without a lot of extra information.
• For every test that runs, one letter is shown to tell you its status:

Character	Description
.	Passed
F	Failed
×	Failed or timed out – and will be retried
±	Passed on retry (flaky)
T	Timed out
°	Skipped

Command Line: npx playwright test –reporter=dot

Playwright Config:

• Instead of specifying the reporter in the command line, we can configure it under the reporter property in the playwright.config.js file.

const { defineConfig, devices } = require('@playwright/test');

module.exports = defineConfig({
  reporter: 'dot',
});

Generated Dot Report

• The Dot report below displays the results of 5 passed tests, 1 failed test, and 3 skipped tests.

HTML Reporter

• When running automated tests, having a detailed report helps you understand the results and resolve issues efficiently. Playwright has an HTML Reporter tool that creates a full test report in a special folder. You can open this report in your web browser to check the results step by step.

Command Line: npx playwright test –reporter=html

• This will make a folder called playwright-report in the folder you’re working in, with the report files inside. You can open the index.html file to see the report.

Playwright Config:

• Instead of specifying the reporter in the command line, we can configure it under the reporter property in the playwright.config.js file.

const { defineConfig, devices } = require('@playwright/test');

module.exports = defineConfig({
  reporter: 'html',
});

• By default, if any tests fail, Playwright’s HTML report will automatically open in your browser. You can change this behavior by using the “open” option in the Playwright configuration. The options available are:
• always: The report will open after every test run.
• never: The report will not open automatically.
• on-failure (default): The report will open only if a test fails.

const { defineConfig, devices } = require('@playwright/test');

module.exports = defineConfig({
  reporter: [
    ['html', { open: 'never' }]
  ],
});

You can view the HTML report as a webpage by selecting a host and port in the Playwright settings. After running the tests, open the report in your browser using the host and port you chose.

const { defineConfig, devices } = require('@playwright/test');

module.exports = defineConfig({
  reporter: [
    ['html', { host: '0.0.0.0', port: 9223 }]
  ],
});

The report is automatically saved in the “playwright-report” folder inside the folder where the program is running. If you want to save it somewhere else, you can use the “outputFolder” option.

const { defineConfig, devices } = require('@playwright/test');

module.exports = defineConfig({
  reporter: [
    ['html', { outputFolder: 'my-report' }]
  ],
});

Generated HTML Report

• Run the test, and the results will be displayed in the terminal. Below is an example output showing 2 tests passed and 1 test skipped.

• To open the HTML report of the last test run, use the command below.

Command: npx playwright show-report

• If you have saved the report in a custom folder, such as my-report, use this command instead.

Command: npx playwright show-report my-report

• Executing these commands will open the report in your browser, displaying the results of the last test run.

JSON Reporter

• The JSON Reporter in Playwright provides test results in a structured JSON format, saving an object containing all the details about the test run.

Command Line: npx playwright test –reporter=json

Playwright Config:

• Instead of specifying the reporter in the command line, we can configure it under the reporter property in the playwright.config.js file.

const { defineConfig, devices } = require('@playwright/test');

module.exports = defineConfig({
  reporter: 'json',
});

Generated JSON Report

• When you run this command, it will display the JSON in the terminal. The JSON report contains a structured object, with each test’s details, including the name, status (passed, failed, skipped), execution time, and any errors encountered during the run.

You can save the JSON output to a file by specifying the file name. This can be done in the configuration file as shown below.

const { defineConfig, devices } = require('@playwright/test');

module.exports = defineConfig({
  reporter: [
    ['json', { outputFile: 'results.json' }]
  ],
});

Now the JSON output is stored in the specified file like below.

JUnit Reporter

• The JUnit Reporter in Playwright generates test results in the JUnit XML format.

Command Line: npx playwright test –reporter=junit

Playwright Config:

• Instead of specifying the reporter in the command line, we can configure it under the reporter property in the playwright.config.js file.

const { defineConfig, devices } = require('@playwright/test');

module.exports = defineConfig({
  reporter: 'junit',
});

Generated JUnit Report

• Running this command will generate the JUnit report and display it in the terminal.

• The report will show you the total number of tests, failed tests, and skipped tests. It will not show you the count of passed tests.
• You can save the JUnit report output to an XML file by specifying the file name. This can be done in the configuration file as shown below.

const { defineConfig, devices } = require('@playwright/test');

module.exports = defineConfig({
  reporter: [
    ['junit', { outputFile: 'results.xml' }]
  ],
});

Now the JUnit output is stored in the specified file and this file will show the status of the test including pass like below.

Set Multiple Reports in the Configuration File

Playwright allows you to use various reporters, which means you can create different kinds of reports at the same time. For instance, you can use the list reporter to get detailed information in the terminal and an HTML or JSON reporter to see test results in a more visual or organized way. You can set this up in the playwright.config.js file.

const { defineConfig, devices } = require('@playwright/test');

module.exports = defineConfig({
  reporter: [
    ['list'],
    ['json', { outputFile: 'test-results.json' }]
  ],
});

Custom Report

• Playwright allows you to create custom reporters to suit your specific reporting needs. A custom reporter can be implemented by extending the default reporter class and overriding its methods to format the test results in the desired structure.

• Create a file with the name myReporter.js.

class MyReporter {
  onBegin(config, suite) {
    console.log('================================');
    console.log(`Starting the run with ${suite.allTests().length} tests`);
    console.log('================================');
  }

  onTestBegin(test) {
    console.log(`Starting test ${test.title}`);
  }

  onTestEnd(test, result) {
    console.log(`Finished test ${test.title}: ${result.status}`);
  }

  onEnd(result) {
    console.log('================================');
    console.log(`Finished the run: ${result.status}`);
    console.log('================================');
  }
}

export default MyReporter;

Command Line: npx playwright test –reporter=”./reporters/myReporter.js”

Playwright Config:

• Rather than specifying the file name for the reporter in the command line, it can be configured in the reporter property within the playwright.config.js file.

const { defineConfig, devices } = require('@playwright/test');

module.exports = defineConfig({
  reporter: './reporters/myReporter.js',
});

Generated Custom Report

• Running this command will generate the Custom report and display it in the output or terminal.

Integrating Third-Party Reporters with Playwrights

Playwright’s growing community brings an increasing number of third-party integration and reporting tools.

Allure Reporter

• Allure is very versatile. It supports multiple languages and produces HTML reports with a user-friendly graphical layout and comprehensive test details. Here’s how to integrate Allure reporting into your project.

• You need to install the Allure dependency in the project to generate the report. Run the below command to install the Allure dependency.

Command: npm install -D allure-playwright

• Use the command below to execute the tests and enable Allure reporting via the command line.

Command Line: npx playwright test –reporter=allure-playwright

Playwright Config:

• Instead of specifying the reporter in the command line, we can configure it under the reporter property in the playwright.config.js file.

const { defineConfig, devices } = require('@playwright/test');

module.exports = defineConfig({
  reporter: 'allure-playwright',
});

If you want to use multiple reporters, you can specify them like this:

const { defineConfig, devices } = require('@playwright/test');

module.exports = defineConfig({
  reporter: [
    ["line"],
    ["allure-playwright"]
  ],
});

After running the tests, the results will be saved in the “./allure-results” folder, which you can customize or configure with additional options.

Generate and View the Allure Report

• To generate the Allure report for the executed tests, run the command below. This will create a folder named allure-report containing the report.

Command: allure generate ./allure-results -o ./allure-report

• To view the generated report in your browser, run the command below. This will open the report in your default browser.

Command: allure open ./allure-report

• Overview of the executed tests in the Allure report.

Detailed view of the executed tests in the Allure report.

Monocart Reporter

• Monocart helps create detailed and interactive test reports that show test results like pass/fail status, duration, errors, and more in a clear format.

• To use Monocart,  install it in your project by running the command below.

Command: npm i -D monocart-reporter

• Use the command below to execute the tests and enable Monocart reporting via the command line.

Command Line: npx playwright test –reporter=monocart-reporter

Playwright Config:

• Instead of specifying the reporter in the command line, we can configure it under the reporter property in the playwright.config.js file.

const { defineConfig, devices } = require('@playwright/test');

module.exports = defineConfig({
  reporter: [
    [
      'monocart-reporter', 
      {
        name: "My Test Report",
        outputFile: './monocart-report/index.html'
      }
    ]
  ],
});

Generate Monocart Report

• The results will be saved in the specified output folder after running the tests. You can check the folder to find the generated report files, including the Monocart HTML report.

• Once the tests are completed and the report is generated, you can view the Monocart report by running the below command. This will open the Monocart report in your default browser, and display the test results in an interactive format.

Command: npx monocart show-report monocart-report/index.html

• In this command, monocart-report/index.html refers to the file path and file name of the generated Monocart report.

Conclusion

Playwright’s reporting tools give useful information about your test results, helping teams find problems early and make testing better. It has different ways to show results, like List, Line, Dot, HTML, JSON, and JUnit. You can easily integrate multiple reporters and even create custom ones to meet your project’s needs.

Using Playwright’s reporting features helps teams save time, check how tests are doing, and make choices based on the data. This improves test automation and the quality of the software.

Witness how our meticulous approach and cutting-edge solutions elevated quality and performance to new heights. Begin your journey into the world of software testing excellence. To know more refer to Tools & Technologies & QA Services.

If you would like to learn more about the awesome services we provide,  be sure to reach out.

Happy Testing 🙂