docs

End-to-End Testing

Run the Playwright suite against your own shop to check that customers can still browse, sign in, fill a basket and place an order in a real browser. The suite places real orders and changes customer data, so run it against a test or staging shop, see Constraints and limits. To begin, follow Set up the suite.

The suite ships with the Storefront Application V3 in testing/playwright/ and uses Playwright, an open-source browser automation framework. It drives a storefront you already run. You provide the URL of that storefront, the credentials for your shop's Storefront API and a set of test customers. The suite looks up its test data in the Storefront API instead of hardcoding product and category IDs.

Constraints and limits

Permanent

  • The suite does not start your application. Start it before the run, or point BASE_URL at a deployed environment.
  • The suite does not create customers or catalog data. You prepare them in your shop. It does place orders: the order success test buys a product with each browser's test customer, so run it against a test or staging shop.
  • The suite lives inside your application. It imports files from your application's config/ and src/client/ folders, so keep it at testing/playwright/.

Known limitations

  • If rate limiting is active for your shop, a sign-in can answer with 429. See Sign-in rate limits for the limit, how the setup reacts and how to lower the risk.
  • The tests find elements by data-testid values and by the markup of the Storefront Application V3. If you change a component or remove a test ID in your storefront, update the matching page object (the class that owns the locators of one page, see How a run works) in page-objects/, or the tests that use it fail.
  • Playwright writes an error-context.md file into test-results/ for every failed test, and the file can hold a password. See the warning under Run the suite in a pipeline.

How a run works

Playwright starts one browser project for each browser profile. Each project emulates a device:

ProjectDevice profile
chromiumDesktop Chrome
firefoxDesktop Firefox
webkitDesktop Safari
mobile-chromeGalaxy S9+
mobile-safariiPhone 15

A project loads pages from the storefront at BASE_URL and drives them the way a customer does: it clicks, fills forms and asserts on what the page shows. Every browser project starts as a user who is not signed in. A test that needs a signed-in customer either restores a saved session or signs in through the UI. A setup project, which Playwright runs before the tests of a browser project, signs in the customers and saves their sessions. See Sessions and sign-in.

Tests keep most locators in page objects. A locator is how a test finds an element on the page. Each page or reusable component has a page object, a class that owns the locators and actions of that page or component, so a test reads as a sequence of steps such as signinPage.fillLoginData(...). Page objects extend a shared Base class that provides navigate(), a waitForDeferredReloads() helper and an isMobileViewport flag.

A fixture is something Playwright builds for a test and hands in as a parameter when the test lists it by name. Fixtures hand each test the page objects it asks for, a storefrontAPIClient for catalog lookups and an rpc helper for basket and wishlist changes.

The rpc helper sends basket and wishlist requests to the RPC endpoint of your storefront at /api, the same one your client code calls.

Test code and fixtures drive a browser project that loads pages from your Storefront Application V3. The tests also send basket and wishlist requests to the application and look up products, categories and promotions in the Storefront API.

Every test that uses the browser also runs with two kinds of third-party requests blocked. The first is the Usercentrics consent loader, so the tests do not load your consent banner. The second is every request to a URL under https://www.googletagmanager.com/gtag/, the path that serves the Google tag script of Google Tag Manager. The suite blocks nothing else. The Google Tag Manager container script still loads, and requests to any other address, such as your own tracking endpoint, still go out.

Sessions and sign-in

Signing in through the UI in every test would be slow, and it can hit a sign-in rate limit when your shop has one active. The suite signs in once for each browser project instead and saves the sessions.

Each browser project depends on its own setup project, named <browser>-setup. The setup project signs in one customer for each role in AUTH_ROLES. A role names a customer whose session the suite saves:

  • default is the customer of the browser project.
  • hydration is TEST_CUSTOMER_EMAIL6. The hydration checks use its session. They load each page type and fail when the browser console reports a hydration problem. Hydration is the step where the browser attaches Vue to the HTML that the server rendered, and Vue logs a problem when the two differ.

The setup opens the customer's orders page to confirm the session, then saves the browser state to .auth/<browser>-<role>.json, which Git ignores. A run limited to one browser (--project=webkit) signs in only that browser's customers.

If your shop rate limits sign-ins, the setup waits and signs in again, see Sign-in rate limits.

A test opts into a saved session with authState(role) from support/auth.ts:

Only the tests that call authState start signed in. When a browser project runs without its setup project, such as the happy path with --no-deps, its tests run as a user who is not signed in.

The setup project signs in through the browser and the Authentication API, which may return 429 when rate limited. It opens the orders page and saves the session to .auth. Browser tests restore it with authState and start signed in.

The setup deletes the old session file of a role before it signs in. authState rejects a saved file older than 24 hours with its own message, so a leftover file never passes as a live session.

Do not use authState in a test that signs out. Signing out ends the session on the server for every later test that restores it.

Sign-in rate limits

If rate limiting is active for your shop, the Authentication API answers a sign-in with 429 when one customer signs in too often. How many sign-ins a customer may do per minute depends on your shop. The setup then waits 61 seconds, one rate limit window, and signs in once more. Only the setup waits. A test that signs in through the UI does not wait, and in CI a failed test is retried at once, without the one-minute wait.

Every sign-in attempt counts against the limit of its customer, and the setup signs in again on every retry. Locally, a lower --workers value or one --project at a time reduces the risk. In a pipeline workers is already 1, so the risk comes from several browser jobs that run at the same time against the same customers.

What the setup does when a sign-in fails

CaseResult
A TEST_CUSTOMER_* variable is missingThe setup skips that role at once.
The sign-in is still rate limited after the one-minute waitThe setup skips that role at once.
The password is wrongThe setup skips the role after the last attempt.
A saved session lacks the access token of the Checkout web componentThe setup skips the role after the last attempt.
Any other failure that is not a rate limitThe setup skips the role after the last attempt.

The other tests of the browser project keep running as users who are not signed in. The tests that restore the missing session fail with No saved session.

Shop data

A test looks up its products and categories in the Storefront API instead of hardcoding IDs, then drives the UI with what it finds. What happens when the shop does not match depends on the test.

  • The data is missing and the test checks for it. A catalog gap is data that your shop lacks and a test needs. The test skips with a reason that names the data to add. Some of these skips also carry a catalog-gap annotation and log [<id>] <reason>, and the skip-rate reporter lists those separately, see Read the results. Other skips can name missing data too, so read their reasons.
  • The shop structure or a customer's state differs from what the test assumes. The test fails. This covers the shop structure the tests rely on, such as the filters of the default Storefront Application V3, and customer state, such as orders on a customer that should have none.
  • The test skips on purpose. A test marked desktop-only or mobile-only skips in the other viewport. The express checkout tests skip unless E2E_EXPRESS_CHECKOUT_ENABLED is true or 1. Tests that need a second country skip on a shop that has only one.

The lookups cover a bounded part of your catalog. The storefrontAPIClient pages through a limited number of products to find a match, it looks for most categories among your root categories and their direct children, and it reads only the first page of promotions and campaigns. Data outside that window counts as missing, so a product far down a large catalog or a deeply nested category does not satisfy most tests. The limits are constants in the storefrontAPIClient source under testing/playwright/. The SETUP.md file in testing/playwright/ of your application lists the data each test looks for and repeats the setup steps of this page. It ships with the suite version you run, so when it differs from this page, follow SETUP.md.

Set up the suite

You end with a passing happy path test against your shop.

Before you start, you need:

  • The Node.js and pnpm versions listed under Prerequisites in the README at your application root.
  • A running storefront that the machine reaches, either a deployed environment or a local pnpm dev server.
  • Your SCAYLE account and your Storefront API key, as the values of STOREFRONT_TENANT_SPACE and STOREFRONT_API_KEY in your application's .env file. The suite has no default for either, so set both even when your application runs without them.
  • A test or staging shop. The tests add products to baskets, change customer data and place orders.

Prepare the workspace

Do this block only when your pnpm workspace does not cover the suite yet, which is usual for a project outside the SCAYLE monorepo. Skip any step your project already meets. When testing/playwright is already listed in your workspace and workspace:* is gone from its package.json, go on with Install the suite and run the happy path.

  1. Create pnpm-workspace.yaml at your application root when it does not exist.
  2. Add testing/playwright under packages in that file when it is not listed. A file that lists only . needs the extra line:
  3. In testing/playwright/package.json, replace workspace:* for @scayle/storefront-api with a published version when it still shows workspace:*. Take the version from your application's package.json, from pnpm-lock.yaml, or from node_modules/@scayle/storefront-api/package.json in an installed project.

Install the suite and run the happy path

  1. Install the dependencies from the application root:

    If pnpm install stops with a missing catalog entry, your pnpm-workspace.yaml does not define it. Define the missing entries in that file: put entries that a package.json marks catalog: under catalog:, and entries marked catalog:app under catalogs: and app:, each with a version. Take the versions from the catalogs: section of pnpm-lock.yaml or from the installed packages.
  2. Change to the suite's folder:
  3. Install the browsers:

    On a bare Linux machine or runner, use pnpm exec playwright install --with-deps. It installs the system libraries the browsers need and requires root rights.
  4. Copy testing/playwright/.env.example to testing/playwright/.env.
  5. In that file, or in the .env file at your application root, set BASE_URL to the homepage of your shop including the country prefix.
  6. In the same place, add STOREFRONT_TENANT_SPACE and STOREFRONT_API_KEY. .env.example lists neither.
  7. From testing/playwright/, run the happy path:

The happy path opens the homepage, goes to a category through the main navigation, adds the first product to the wishlist and the basket, and checks that checkout asks a user to sign in. A passing run lists one passed test and ends with the skip-rate line. It shows that your storefront, BASE_URL and Storefront API credentials work.

See Environment variables for every variable.

The configuration prints a warning for each .env file it does not find. The warning alone is not a failure. If neither file nor your shell holds the variables, the run then fails with STOREFRONT_API_KEY is not set. See Troubleshooting for the other errors.

The happy path needs no test customers. Register the test customers and set the TEST_CUSTOMER_* variables before you run the other specs.

Test customers

You register these customers once in your shop through its own sign-in page, and the suite never creates them. All customers share one password, set in TEST_CUSTOMER_PASSWORD. Register TEST_CUSTOMER_EMAIL1 with a mailbox you control, because the password reset test sends a real reset email to it. The order success test places an order with each browser's customer, so TEST_CUSTOMER_EMAIL1 to 5 need a saved delivery address and an adult birth date, so the checkout offers invoice. The order success test selects invoice itself and fails with a clear message when the checkout disables it. The account flyout test asserts the first name Aqa in the greeting of the desktop customers, so register TEST_CUSTOMER_EMAIL1 to 3 with that first name. Across all browser projects, the sign-in setup signs in TEST_CUSTOMER_EMAIL1 to 6. Each browser project has its own customer for the basket and account tests, so parallel runs do not clash on basket state.

VariableUsed byState it needs
TEST_CUSTOMER_EMAIL1chromium, and the sign-in, orders and wishlist tests in every browser projectMore than 8 orders, so the order list has a second page. Empty wishlist and basket. First name Aqa, a saved delivery address, an adult birth date.
TEST_CUSTOMER_EMAIL2firefoxEmpty basket. First name Aqa, a saved delivery address, an adult birth date.
TEST_CUSTOMER_EMAIL3webkitEmpty basket. First name Aqa, a saved delivery address, an adult birth date.
TEST_CUSTOMER_EMAIL4mobile-chromeEmpty basket. A saved delivery address, an adult birth date.
TEST_CUSTOMER_EMAIL5mobile-safariEmpty basket. A saved delivery address, an adult birth date.
TEST_CUSTOMER_EMAIL6The hydration role, signed in by every browser projectItems in the wishlist and the basket, so the hydration checks cover non-empty pages.
TEST_CUSTOMER_NO_ORDERSThe Orders page empty state testNo orders.

A run limited to one browser project needs that browser's customer, TEST_CUSTOMER_EMAIL1, TEST_CUSTOMER_EMAIL6, TEST_CUSTOMER_NO_ORDERS, and the password variables.

Run the tests

Run these commands from testing/playwright/.

The happy path has its own script, pnpm test:e2e:happy-path, which pnpm test:e2e does not run. The script passes --no-deps, so it skips the sign-in setup and runs as a user who is not signed in. Without --project, it runs in every browser project.

UI mode and the VS Code extension do not run setup projects unless you select them. Select <browser>-setup before you run a test that restores a signed-in session.

The suite can also scan your pages for accessibility problems with axe. These checks are off by default and only report. See Accessibility Testing.

A run with the CI environment variable set uses a different profile than a local run, see Run the suite in a pipeline.

Configuration

The suite is configured in testing/playwright/playwright.config.ts. These are the settings you are most likely to look up. Local values apply when CI is not set. Any non-empty CI value counts as set, including false, and a set CI also stops the suite from reading .env files.

SettingLocalCI (CI set)
testDir./tests./tests
fullyParalleltruetrue
workersPlaywright's automatic count1
retries02
forbidOnlyfalsetrue
baseURLBASE_URL, default https://localhost:3000/de/Same
traceon-first-retryon-first-retry
ignoreHTTPSErrorstruetrue
projectsThe browser projects, each with a <browser>-setup project, and the a11y-crawl projects with A11Y=1Same
reporterjunit, list with steps, and skip-rate, plus playwright-testrail-reporter when TESTRAIL_RUN_ID is setSame

The table shows the values in the configuration file. With retries at 0, a local run records no trace unless you pass --retries=1 or --trace on. The setup projects turn traces off, so a retried sign-in does not record the typed password. A pipeline that passes --reporter on the command line replaces the reporters of this file, and the skip-rate reporter then runs only if the pipeline lists it.

Read the results

A run with the reporters of the configuration file prints the list reporter output with steps, writes a JUnit report to test-results/results.xml, and prints a skip-rate line at the end. The rate is the skipped tests divided by the skipped and executed tests, for example:

After that line, the skip-rate reporter lists each skip that carries a catalog-gap annotation with its project and reason, then the other skips grouped by reason with a count. The reporter never fails a run and applies no threshold. If you want a gate, check this line or results.xml in your own pipeline step.

A high skip rate usually means your catalog lacks data the tests look for. Skips listed as a catalog gap name data to add. Other skips can name missing data too, so read their reasons, see Shop data. Even a healthy first run on a shop with partial data has some skips.

When TESTRAIL_RUN_ID is set, the configuration also adds the playwright-testrail-reporter package, which sends the results to TestRail. Without the variable, that reporter is off. The titles of the shipped tests start with a TestRail case ID such as C2139186:, so keep it when you edit a test, or use your own ID scheme consistently.

Run the suite in a pipeline

You end with a pipeline job that runs the suite, reports its results and keeps the files you need to debug a failure.

test-results/ can hold passwords. Playwright writes an error-context.md file into it for every failed test. The file holds a snapshot of the page, which can include a password typed into a form. The .auth/ folder holds the tokens of live sessions. Restrict who can download the artifacts of the job, and never upload .auth/.

  1. Install the browsers in the job with pnpm exec playwright install --with-deps, because a runner is usually a bare Linux machine. If your pipeline uses a Playwright Docker image instead, keep its version the same as the version of @playwright/test.
  2. Pass every variable through the environment of the job: BASE_URL, STOREFRONT_TENANT_SPACE, STOREFRONT_API_KEY and the TEST_CUSTOMER_* variables, see Environment variables. CI is set in a pipeline, so the suite reads no .env file. Store the keys and passwords as masked variables of your pipeline.
  3. From testing/playwright/, run pnpm test:e2e. The CI profile runs one worker and retries a failed test twice, and it fails the run on a leftover test.only, see Configuration.
  4. Keep test-results/results.xml and the skip-rate line of the log. A --reporter option on the command line replaces the reporters of the configuration file, and the skip-rate reporter then runs only if you list it. The run never fails on the skip rate, see Read the results.
  5. Keep test-results/ as an artifact of the job, with restricted access, see the warning above.

Browser jobs that run at the same time share the sign-in limits, see Sign-in rate limits. The accessibility checks have their own pipeline steps, see Run the checks in a pipeline.

Write a test

Start from a suite that passes the happy path, see Set up the suite. You end with a spec that uses the shipped fixtures and page objects.

  1. Pick the spec file for the feature. Specs live in testing/playwright/tests/e2e/ and follow the {feature}.e2e.spec.ts naming, for example plp.e2e.spec.ts. Add the test to the existing file for that feature, or create a new file for a new feature.
  2. Import test from fixtures/fixtures, not from @playwright/test. Import expect from @playwright/test.
  3. Ask for the page objects and fixtures the test needs in the destructured parameters.
  4. Look up the data the test needs through storefrontAPIClient:
  5. Group the actions in test.step() calls. Each step shows as its own entry in the report.
  6. If the test needs a signed-in customer, restore a session with test.use(authState('default')) in the describe block, as shown in Sessions and sign-in.
  7. Before you commit, run pnpm lint and pnpm typecheck from testing/playwright/.

Add a page object

A page object extends Base and declares its locators as readonly properties in the constructor. It finds elements with getByTestId(), which matches the data-testid attributes of your markup.

To make a page object that takes only page available as a fixture, import it and add it to pageObjectMap in fixtures/pageObjects.ts:

The map generates one fixture for each entry and derives its type, so tests can then ask for giftCardBanner by name. A page object with other dependencies, such as basketPage, which also needs rpc, is declared as an explicit fixture in fixtures/fixtures.ts, and its name and type go into the Fixtures interface there.

Environment variables

The suite reads variables from the .env file in testing/playwright/, then from the .env file in your application's root directory. A value exported in your shell wins over both files. With CI set, it reads neither file.

VariableRequiredWhat it does
BASE_URLNoThe homepage of the shop under test: the scheme, the origin and the country prefix, for example https://localhost:3000/de/. Falls back to https://localhost:3000/de/.
STOREFRONT_TENANT_SPACEFor the happy path and catalog lookupsYour SCAYLE account. The suite builds the Storefront API host from it and has no default. Tests that use storefrontAPIClient, the happy path included, fail without it.
STOREFRONT_API_KEYFor the happy path and catalog lookupsYour Storefront API key. The suite uses it to look up products, categories, promotions and campaigns. Tests that use storefrontAPIClient, the happy path included, fail without it.
TEST_CUSTOMER_EMAIL1 to 6For signed-in testsThe test customers. See Test customers.
TEST_CUSTOMER_NO_ORDERSFor the Orders page testA test customer with no orders.
TEST_CUSTOMER_PASSWORDFor signed-in testsThe password shared by all test customers.
TEST_CUSTOMER_WRONG_PASSWORDFor the sign-in testAny value that is not a real password. Only the negative sign-in test uses it.
E2E_SHOP_IDNoThe shop country ID, as a number. By default the suite reads it from the page data of the rendered storefront. Set it only when your storefront does not expose that data.
E2E_EXPRESS_CHECKOUT_ENABLEDNoSet to true or 1 to run the express checkout tests. They skip otherwise.
E2E_RPC_TIMEOUT_MSNoTimeout in milliseconds for one RPC request the tests send, such as adding a product to the basket. Defaults to 5000. A value that is not a number above 0 is ignored with a warning.
TESTRAIL_RUN_IDNoThe ID of a TestRail run. When set, the suite adds playwright-testrail-reporter and sends the results to that run.
A11YNoSet to 1 to turn on the accessibility checks. See Accessibility Testing.

BASE_URL takes no query string or hash, because the suite resolves relative paths against BASE_URL, which drops a query string or hash. It needs a scheme such as https://. The suite adds a missing trailing slash, which keeps the country prefix when specs resolve paths against it.

Best practices

Let a test fetch its own data

Use storefrontAPIClient to find something in the live catalog that satisfies the test, and skip with a reason when nothing matches. A hardcoded ID fails the day that product leaves the catalog.

Locate elements by data-testid

A CSS class or a visible label changes for styling or copy reasons that have nothing to do with the behavior a test checks. A data-testid is more stable, because it changes mainly when the element it marks is gone or restructured. Prefer it, and use a role or text locator only where no test ID exists.

Wait for state, not for time

In tests/, page-objects/ and support/, the lint configuration fails every page.waitForTimeout() call, and it warns on waitForLoadState('networkidle') and the force option. Wait for an element or a response, and call waitForDeferredReloads() before an interaction that a deferred prop could reset when its data arrives late, such as picking a size on a product page.

Use paths relative to BASE_URL

In the same folders, the lint configuration applies the recommended Playwright rules. It also rejects a path written as a string or template literal that starts with / in goto, waitForURL, toHaveURL and the second argument of navigate(), because such a path skips the country prefix of BASE_URL. Use a path relative to it, such as ROUTES.basket. The check only sees paths written directly in the call, not a path held in a variable.

Keep tests independent

Locally, tests run in parallel. A test sets up its own state instead of relying on a previous test, or an ordering dependency shows up as flakiness.

Troubleshooting

Tests fail with "No saved session"

Issue: A test fails with No saved session for role "<role>" in project "<project>", or with a message that the saved session is older than 24 hours.

Possible causes: The <browser>-setup project could not sign in the customer, so it skipped, see Sessions and sign-in for the cases. The setup project did not run, which happens in UI mode and in the VS Code extension unless you select it. The session file is older than 24 hours.

Solution:

  1. Read the skip message of the setup project and the console output. They name the cause, with the password masked.
  2. Fix the variable or the password, or wait a minute if the sign-in was rate limited.
  3. Run the browser project again. Each setup run deletes the old session file first.

Prevention: Check the customers with a manual sign-in before the first run.

Tests that look up catalog data fail

Issue: Tests fail with STOREFRONT_API_KEY is not set or STOREFRONT_TENANT_SPACE is not set.

Possible causes: The variable is missing from testing/playwright/.env, from your application's .env and from your shell. With CI set, the suite reads no .env file.

Solution: Set both variables where the suite can read them. In a pipeline, pass them as environment variables.

Prevention: Run the happy path first. It fails fast on a missing key.

The run stops with "BASE_URL must be the homepage of the shop under test"

Issue: Playwright exits when it loads the configuration, with this message or with Invalid URL.

Possible causes: BASE_URL has a query string or a hash, or it has no scheme such as https://.

Solution: Set BASE_URL to the homepage of the shop, such as https://localhost:3000/de/.

Prevention: Use only the scheme, the origin and the country prefix in BASE_URL.

A test times out waiting for an element

Issue: A test fails with a timeout on a locator, although the element renders when you open the page by hand.

Possible causes: The test opened a different URL than you expected, for example because of a stale BASE_URL or a redirect. Or the element does not render under the test's conditions, such as another country or a missing precondition.

Solution: Check page.url() right before the failing assertion. Run the test with pnpm test:e2e:ui to watch it and see the real page state at the point of failure. In UI mode, select the <browser>-setup project first when the test restores a saved session.

Prevention: Assert on the URL after each navigation in a multi-step journey, so a wrong page fails at the step that caused it.

A failure shows a 429 or a timeout earlier in the log

Issue: A basket test or the shop data check fails, and the log shows a 429 or a timeout before the failure.

Possible causes: Your shop rate limited the requests of the suite, see Sign-in rate limits. The failing assertion is then a consequence of the limit, not a real difference in your data.

Solution: Run the failing test again before you look for a data problem.

Prevention: See Sign-in rate limits for how to lower the risk.

Tests pass locally and fail in CI

Issue: A suite that is green on your machine produces failures in the pipeline.

Possible causes: The CI profile differs from the local one, see Configuration. A variable that only lives in a .env file is missing, a leftover test.only fails the run, and the shop may answer more slowly.

Solution: Reproduce the CI profile locally with CI=true pnpm test:e2e, and export every variable in your shell first.

Prevention: Do not assert exact counts from a live catalog. Assert on presence and shape, and use storefrontAPIClient for the data.