Skip to main content

End-to-end Testing

We use playwright for end-to-end testing. Tests are located in the /e2e folder and are separated by page.

Configuration​

In the playwright.config.ts file you can see the setup configuration, including the browsers to test against.

Currently we include mobile browsers in normal tests, and exclude them in accessibility tests using the @a11y tag.

Accessibility​

Playwright test cases include accessibility validation. These tests have their own tag @a11y so that they can be filtered in or out.

test('Home a11y', { tag: ['@a11y'] }, async ({ page }) => {
  // […]
});

Fixtures​

Fixtures in playwright offer wrapped functionality available on each test. These are accessible on each test individually, and allows wrapping reused logic.

Axe​

This fixture sets up the axe plugin for a11y tests with the same WCAG flags.

Page Objects Models​

In playwright, Page Object Models are a way to structure test by creating a high-level API of flows that are recurrent in your test cases. They essentially are classes with functions for repeated processes to simplify maintenance in the code.

This keeps basic flows centralized and simplify refactoring test cases. If you see an opportunity to extend current models or create new ones, do so.

Assemble Page​

This model wraps commonly used selectors and functionality available on most pages: opening the mobile menu (for navigation purposes) and locate links based off of a href, to trigger navigation from the current page to a desired one.

Player Page​

This model wraps the necessary setup to create a chromium instance with the required flags for shaka to instantiate correctly. This prevents repeating the same setup over each test.

Scripts​

pnpm e2e​

This will run all tests against the browsers in the configuration file. Running this from your local machine will output an inline and HTML report displaying any failed tests and their stacktrace.

pnpm e2e:ui​

This will run playwright’s UI mode.

pnpm e2e:codegen​

This will run playwright’s codegen command. Useful to generate test cases and generating the proper locators.

Playwright and TestRail Integration​

This document explains the integration between Playwright and TestRail, covering the existing Playwright setup, mapping test specs to TestRail tests, and how the integration works.

Playwright Setup​

Playwright is configured in the playwright.config.ts file. Key configurations include:

  • Test Directory: All tests are located in the e2e/ directory.
  • Reporters: The configuration includes a JUnit reporter that outputs results to ./test-results/junit-report.xml.

Mapping Test Specs to TestRail Tests​

To map Playwright test specs to TestRail tests, the junit reporter is used to generate a JUnit XML report. The name attribute in the test cases is used as the identifier to match TestRail test cases so in order to make the proper map, the Playwright test name should include the Testrail testcase id (ie: [C97123] Error handling. Server Error).

Steps to Map Tests:​

  1. Ensure each Playwright test case has a unique and descriptive name.
  2. The junit reporter generates a report at ./test-results/junit-report.xml.
  3. The testrail.sh script uses the --case-matcher "name" option to map test cases to TestRail.

TestRail Integration​

The integration with TestRail is handled via the testrail.sh script and the GitHub Actions workflow defined in .github/workflows/playwright.yml.

Key Components:​

  1. testrail.sh Script:

    • Creates or finds a Test Run in TestRail using the find_test_run_by_name and create_test_run functions. If a test run for the PR always exists, it will override it.
    • Uploads test results to TestRail using the trcli CLI tool.

  2. GitHub Actions Workflow:

    • The Playwright workflow runs Playwright tests and uploads results to TestRail.
    • Environment variables like TESTRAIL_PROJECTID, TESTRAIL_SUITEID, and TESTRAIL_TITLE are used to configure the TestRail project and suite.
    • The TestRail CLI upload results step installs trcli and executes the testrail.sh script.

Workflow Steps:​

  1. Run Playwright tests for accessibility and end-to-end scenarios.
  2. Generate a JUnit XML report.
  3. Execute the testrail.sh script to upload results to TestRail.

Example TestRail Configuration:​

  • TestRail Suite ID: Set via TESTRAIL_SUITEID.
  • TestRail Run base title: Set via TESTRAIL_TITLE.
  • TestRail Project ID: Set via TESTRAIL_PROJECTID.
  • TestRail Base URL: Set via TESTRAIL_URL.
  • TestRail User apiKey: Set via TESTRAIL_KEY.
  • TestRail user email: Set via TESTRAIL_USER.
  • TestRail Project Name: Set via TESTRAIL_PROJECT.
  • Test Run Name: Generated dynamically using TESTRAIL_TITLE and PR_NUMBER.

By following this setup, Playwright test results are seamlessly integrated into TestRail, providing a comprehensive view of test execution and results.

Pending Challenges​

  • Use more than one project/browser to ensure a testcase is compliant in all the possible browsers.
  • Ensure that test run check inside testrail.sh considers pagination
  • Ensure that non web testcase are not considered in the list of cases to validate against in the report.

Project Structure Overview​

This project uses Playwright for end to end testing and follows the page object modal (POM) design pattern to endure scalability, maintainability, and reusability of the test code. The root folder for the automation code is named e2e, and it contains the following subfolders:

  1. base/
    This folder serves as the foundation of the test framework. It contains:
    • ActionHelper class → Utility classes and functions shared across the project.
    • AppData class → Centralized configuration and application, specific data used in tests
    • Fixtures file → shared setup/teardown logic and initialization routines.
    • Logger class → a custom logger class for tracking test steps and debugging
  2. pages/
    This directory contains the Page Object Model representations of the application’s UI. Each file corresponds to a screen or page in the app.
    Each page class contains:
    • Locators specific to that page.
    • Actions/Functions that perform operations or validations on the page.
      Benefit:
    • This encapsulation makes the test code clean, readable, and easy to maintain.
  3. tests/
    This is the primary folder for executing tests. It contains test specifications that utilize page objects to simulate user interactions and verify application behavior.
    • Each test imports relevant page objects from the pages directory.
    • Tests are written in a clear and descriptive manner, aligned with real-world user flows.
  4. sandbox/
    This folder contains experimental or developer-specific logic that does not directly affect the main testing workflow. It is not intended for use by testers and typically includes temporary, experimental, or custom implementation code.
    Important: This folder is essential to the project and must not be removed, even if it’s not used directly in test cases.

This structured approach ensures that the test suite remains organized, efficient, and adaptable to future changes in the application.

Running tests​

There are two primary ways to execute Playwright tests in this project:

  1. From the Test File (UI-based Execution)
    Playwright provides a built-in UI to run tests directly from your test spec files: - Simply click the green “play” icon next to any test or describe block in your test file. - This method is ideal for quick test runs and debugging during development.
    Note: This feature is available when using compatible editors like Visual Studio Code with the Playwright extension installed.
  2. From the Terminal (Script-based Execution)
    Tests can also be executed via terminal using NPM scripts defined in package.json.
    • Headless Mode (default) 
      pnpm run e2e
      Runs tests without opening a browser window – useful for CI/CD pipelines or faster local execution.
    • UI mode (headed) 
      pnpm run e2e:ui
      Opens Playwright test runner in interactive mode - allows you to select, filter and run tests visually

Writing new test test case and adding new page object​

  1. Go to [PAGES] folder

    • create the page e.g [AboutPage]
    • identify the locators on the page you will need to interact with
    • create function you need to implement certain task.
      kindly follow the structure 
        async verifyProfileVisibility() {
         await Logger.check(
      'verify profile page visibility',
         async () => await ActionHelper.verifyElementIsDisplayed(this.profilePageTile)); }

explanation​

  • Define your function using the async keyword to support asynchronous operations.
  • Choose a clear and descriptive name that reflects the purpose fo the function
  • Wrap your logic inside a Logger function, including:
    • A descriptive title that summarizes what the logic does.
    • The actual logic itself for tracking and debugging purposes