Planning A Black Friday Load Test

We plan and execute load testing to help our clients prepare for Black Friday and other peak trading periods. A common mistake we have observed is where the incorrect test volumes are used, thus reducing the value of the performance testing carried out.

There are 2 steps to getting this right:

1. Business-driven demand modelling
2. Demand forecast validation

Business-driven Demand Modelling

The first step is to quantify the demand your web site is likely to experience over its peak trading period.

The flow chart below outlines our process for doing this, using the example of an online retailer:

 In an ideal world, the business would have accurate forecasts of demand for the forthcoming peak period. However, in our experience this is often not the case. Often, demand forecasts are aspirational or poorly defined.

We help our clients forecast demand using a number of techniques, including:

• Time series decomposition
• Regression analysis
• Arrival rate distribution analysis
• Bursty demand analysis

We’ve observed substantial growth in mobile and mobile application demand in recent years. The forecast should be broken down by channel, because each could generate different technical demand:

Once the forecast business demand has been established, this must be translated into a technical demand forecast. Our methodology for delivering this includes the following techniques:

• Systems analysis
• User behaviour modelling

If you are forecasting for an e-commerce site, it is important to consider demand from ‘scrapers’. Many online stores have their inventory and prices ‘scraped’ by automated robots operated by rival companies or resellers. In our experience this demand can be substantial.

It's worth noting that these forecasts can be reused for capacity planning purposes.

Demand Forecast Validation

Remember any forecast will be wrong – it’s just a question of how wrong! Once the peak period has passed, calculate your forecast error.

Here is an example:

If you are within 10%, pat yourselves on the back. If you are not, as in the example above, then it's an opportunity to learn why and update your forecasting method for greater accuracy in the future. 

Summary

Demand planning is fundamental to delivering accurate load tests for Black Friday. Unless you can predict demand with a reasonable level of accuracy then how can you deliver reliable performance testing for peak?