Overview

Military B2C retail store with around 10k items on site. Mainly the store uses 2 types of products (configurable and simple), two types of users (guest ad registered), and two types of checkout (In-store pick up and Shipping).

Testing type:
Smoke, load
Duration:
70 hours
Team:
1 load QA engineer
Platforms and tools
:
JMeter, JMeter AWS Servers, Postman, Kibana, New Relic, Jira, Slack

Challenge

Clients expect huge sales during Black Friday. The ultimate goal is to focus on mitigating performance risks, measurement of the system’s performance under load, and gathering necessary metrics for making sure that the system meets performance requirements. Eventually, replicate the expected production volumes and provide reporting based on performance criteria.

Solution

The initial step was clarifying from the client to gather and provide metrics from the Google Analytics to see the real traffic on the website site during a normal day (like count of placed Orders per day, visiting, popular categories etc.) At the same time performing smoke tests of the website for understanding its specifics, and how it actually works in general.

The next step was the development of a dynamic load scrip. Under the “dynamic” script, we try to avoid static requests and try to simulate real users behaviour. One positive case in details described below:

E.g. Open Homepage -> Go to login page -> Select random test user from the file and log in -> make an API call to the rest to get all categories and sub-categories -> Select random URL -> Check condition if the current category has products -> Randomly select a filter from all available -> grab all sorting and apply random -> Confirm that result page has product -> Select random PDP from the gird -> Check if product “Out of Stock” -> Defined the product type -> Defined type of product shipping -> Add to Cart -> Cart -> and finally place Order. 

For maximum coverage of all cases we created the next three groups and calculated the next users counts:

 1. Thread Group: Authorised Checkout (Home -> Login -> CLP -> CLP (filterable and sortable) -> PDP -> Add to Cart -> Authorised Checkout  – 16 users

 2. Thread Group: Guest Checkout (Home  -> CLP -> CLP (filterable and sortable) -> PDP -> Add to Cart -> Guest Checkout  – 16 users 

 3. Thread Group: Browsing the site (Home  -> CLP -> CLP (filterable and sortable) -> PDP – 164

 4. Thread Group: Searching (Home -> Search -> Filtering, Sorting) – 8

 5. Thread Group: My Account (Login -> Go through all my account tabs -> Logout) – 4

Duration: 90 min

Ramp Up Period: Load testing was initiated from a minimal number of users (60 users) and gradually increased to support more and more users. The peak was reached about 60 min after the start and then kept at the same level (208 users)

Results

When the script was launched we created a detailed report with all run statistics like (total request count, server hits, errors and timely responses for all requests. We detected vulnerable areas of the site and provide the client with solutions for improving current areas.