Load Testing

Overview

As technical complexity rises, so does the potential for performance bottlenecks and system failures under load. It is therefore crucial to validate and test critical performance characteristics under realistic traffic conditions. The SCAYLE Storefront Application comes with a comprehensive load testing suite using Artillery to support this.

The load testing coverage of the Storefront Application is continuously extended with every release. The current focus lies on performance validation for key user journeys and system capacity testing. Load testing ensures that your application can handle expected traffic loads while maintaining performance, stability, and user experience.

Getting Started

By leveraging the load testing framework Artillery, we can simulate realistic user behavior and traffic patterns to validate system performance under various load conditions. This approach allows us to identify performance bottlenecks, validate system capacity, and ensures that your application can handle traffic spikes during peak shopping periods. Ultimately, a robust load testing suite contributes significantly to the overall performance, reliability, and scalability of our storefront applications.

Prerequisites

We recommend getting familiar with Artillery's capabilities and load testing concepts before proceeding with this guide:

• Artillery Documentation
• Using Artillery with Playwright
• Load Testing Best Practices

Testing Environment Setup

To extend testing capabilities, it also utilizes @playwright/test and the integrated Playwright E2E test suite.

Artillery

Artillery is a modern load testing tool that provides utilities to:

• Simulate User Behavior: Create realistic user journeys and traffic patterns.
• Multi-phase Testing: Configure complex load patterns with warm-up, ramp-up, and peak phases.
• Performance Metrics: Collect and analyze response times, throughput, and error rates.
• Browser Automation: Use Playwright integration for realistic browser-based testing.
• HTTP Testing: Direct API endpoint testing for backend performance validation.
• Scalable Execution: Run tests across multiple workers and environments.
• Real-time Monitoring: Monitor test execution and system performance in real-time.
• Flexible Configuration: Environment-driven configuration for different testing scenarios.

Playwright Integration

The Artillery integration utilizes the integrated Playwright E2E test suite (located at testing/playwright/) for browser automation, providing:

• Real User Simulation: Browser-based testing that simulates actual user interactions.
• Complete User Journeys: Test full user flows, including JavaScript execution and DOM interactions.
• Cross-browser Testing: Validate performance across different browser engines.
• Page Object Model: Leverage existing page objects for consistent test implementation.
• Performance Validation: Measure frontend performance metrics under load conditions.

Setup & Configuration

The Artillery integration is handled as a dedicated sub-package within the Storefront Application. It is located in the testing/artillery/ directory, alongside the integrated Playwright E2E test suite at testing/playwright/. To ensure it functions properly, you must manually install its dependencies, as this isn't done automatically with the main application install. The following command should be executed within the testing/ directory as it's a dedicated PNPM workspace:

bash

Copy

More informations can be found in the Artillery documentation.

The load-test.yml and stress-test.yml files are the central places to configure how your load tests are run. You can check and modify the following features:

Test Phases: This defines how the load (number of virtual users) is injected over time with configurable arrival rates and durations.

Base URL: The base URL for your application should be defined via an environment variable. A fallback is provided, which defaults to localhost if no environment variable is set:

yaml

Copy

Scenario Distribution: By default, all test scenarios are executed with configurable weights to simulate realistic user behavior patterns.

Browser Configuration: Playwright browser settings are optimized for load testing scenarios with memory and performance considerations.

More information on how to configure the project to use different load patterns or to execute the tests on different environments can be found in the Artillery documentation.

Environment Variables

As a prerequisite for running load tests that require specific configuration (e.g., for test duration, arrival rates, or scenario weights), you need to set up the appropriate environment variables.

The list of required environment variables is available in the .env.example file within the root project directory of the Storefront Application. They can be set in the working .env file, as well as in the respective CI tool, e.g. as environment variables in GitLab or Github.

Base Configuration

Load Test Configuration

Load tests simulate realistic user behavior with gradually increasing traffic patterns.

Scenario Weights (Load Test)

Stress Test Configuration

Stress tests identify performance bottlenecks and system breaking points under high load.

Scenario Weights (Stress Test)

Running Tests

Tests can be executed locally from the Artillery directory. By default, tests are stored within the testing/artillery/ directory and executed when the command is triggered.

By default local tests will be run in standard mode. In this mode, detailed logging is not enabled, but you can view the test execution progress and results in the terminal.

Load Tests

Load tests simulate realistic user behavior with gradually increasing traffic patterns to validate system performance under normal expected loads.

bash

Copy

Stress Tests

Stress tests identify performance bottlenecks and system breaking points under high load conditions.

bash

Copy

Debugging and Logging

To aid debugging, you can run tests with verbose logging. This provides detailed information about test execution, allowing you to monitor performance metrics and identify issues. After execution, the output provides comprehensive performance statistics and error analysis.

Verbose mode can be executed using the command:

bash

Copy

Test Scenarios

The Artillery suite includes comprehensive test scenarios covering key user journeys:

Homepage Visits

• Landing page performance and navigation
• Country detection modal handling
• Basic page load validation

Product Browsing

• Category and Product Listing Page interactions
• Product card interactions and navigation
• Filter and sorting functionality

Product Details

• Individual Product Page performance
• Product variant selection
• Product image loading and display

Shopping Cart Operations

• Add to cart functionality from Product Pages
• Basket management and updates
• Cart persistence and validation

Wishlist Operations

• Add/remove items from wishlist
• Wishlist persistence and management
• Cross-page wishlist functionality

Search Functionality

• Product search and filtering performance
• Search suggestions and autocomplete
• Search Result Page navigation

Navigation Flows

• Cross-page navigation patterns
• Breadcrumb and category navigation
• Mobile and desktop navigation

Examples

Basic Load Test Configuration

yaml

Copy

Test Function Implementation

typescript

Copy

Environment Variable Usage

yaml

Copy

Extending Existing Tests

Adding New Test Scenarios

1. Define the scenario in YAML:

yaml

Copy

The think option in the above example is used by Artillery to pause a virtual user for a specified number of seconds. This is useful for simulating more realistic user behavior. For example, think: 5 would pause the virtual user for 5 seconds. Given a value like [1, 4], the a virtual users would randomly wait between 1 and 4 seconds.

You can find more details about the think option in the official Artillery documentation.

2. Implement the test function:

typescript

Copy

3. Add environment variable:

bash

Copy

Modifying Existing Scenarios

Update the scenario configuration and enhance the test function:

yaml

Copy

typescript

Copy

Adding New Test Phases

yaml

Copy

Creating Custom Testing Scenarios

Custom Load Test Configuration

yaml

Copy

Custom Test Functions

typescript

Copy

Custom Environment Variables

bash

Copy

Running Custom Tests

bash

Copy

Reporting and Analysis

After running a load test with Artillery, the next crucial step is to analyze the results to understand your application's performance. Artillery provides detailed metrics and reports to help you identify bottlenecks and make informed decisions.

Reported Metrics

Artillery reports a variety of metrics in real-time to the console, providing a summary at the end of the test run. Key metrics include requests per second (RPS), request latency percentiles (p95, p99), and HTTP response codes. These are essential for understanding your application's throughput and performance under load.

For a comprehensive list and detailed explanation of all available metrics, please refer to the official Artillery documentation.

Generating an HTML Report

While the console output provides a good summary, Artillery can generate a more detailed and user-friendly HTML report.

To generate an HTML report, you first need to run your test with the --output flag to save the results in a JSON file.

bash

Copy

Once the test is complete, you can use the report command to convert the JSON file into an HTML report.

bash

Copy

This will create a self-contained HTML file (test-report.html) in the specified directory.

The HTML report provides interactive charts and detailed breakdowns of the metrics, making it easier to visualize your application's performance over the duration of the test. You can analyze trends in response times, requests per second, and error rates to pinpoint performance issues.

Best Practices for Load Testing

Test Planning and Design

1. Begin with low arrival rates and gradually increase to identify breaking points.
2. Design test scenarios based on actual user behavior analytics.
3. Use dedicated test environments to avoid impacting production systems.
4. Monitor system resources during test execution.

Configuration Management

1. Use environment variables for all configurable parameters.
2. Ensure scenario weights sum to 100% for proper distribution.
3. Balance test duration with meaningful data collection.
4. Use realistic arrival rates based on expected traffic patterns.

Test Execution

1. Use multi-phase tests with a gradual ramp-up to avoid overwhelming the system.
2. Monitor application and infrastructure metrics during tests.
3. Implement proper error handling and retry mechanisms.
4. Verify test data is valid and accessible.

Performance Analysis

1. Monitor response times across different scenarios.
2. Track requests per second and concurrent users.
3. Monitor error rates and failure patterns.
4. Track CPU, memory, and network utilization.

CI/CD Integration

1. Integrate load tests into CI/CD pipelines.
2. Establish performance baselines for regression testing.
3. Set up alerts for performance degradation.
4. Generate comprehensive performance reports.

Browser Configuration

The Artillery Playwright integration includes optimized browser settings for load testing:

yaml

Copy

These settings optimize browser performance for load testing scenarios while maintaining realistic user behavior simulation.

Important Notes

Always ensure you have:

• Proper environment configuration
• Appropriate test environment setup
• Monitoring in place for system resources
• Clear understanding of test impact on your infrastructure

Best Practices:

• Start with low arrival rates for initial testing
• Monitor system resources during test execution
• Use appropriate test environments (not production)
• Review test results and adjust configuration as needed
• Implement proper error handling and retry mechanisms
• Use realistic test data and scenarios
• Monitor both application and infrastructure metrics