The Pros & Cons of Performance Testing with Simulated vs Real Traffic

Sales seasons, promotions, product launches, and perhaps Black Friday should be the most rewarding time for companies, having invested limited resources. Often, the end result includes long page load times, dissatisfied consumers, and ultimately site outages. The bottleneck here being the IT systems that seamlessly seem to support usual demand on a normal day. 

Performance testing is known to have saved many industries this heartache. What is it then? It simply measures how well and how quickly an application performs a specific function (for example loading a page displaying products). In technical terms, it means applying traffic to an infrastructure (test or production environment) by replicating real life users in such a way that potential performance risks are exposed or mitigated against.

How does one then go about it? Testing is usually done against an infrastructure that is specially for test and/or one that is live. There are advantages and disadvantages of testing against either, although testing against production (live) ensures the end-to-end testing of the entire technology stack. Capacity and performance non-functional requirements must be defined, as this determines what type or combination of testing is conducted. A test scenario is created, run, analysed and results interpreted and reported against previously defined NFRs.

Download our guide to discover the benefits of performance testing and the secrets of its delivery

Deciding on Real Traffic vs Simulated Traffic

There are two ways to test in production: either real life traffic generated from logs is replayed, or traffic is simulated. The question still begs, which is better: "Real traffic or simulated traffic”? To sufficiently answer this question, the pros and cons of each must be considered.

Whilst simulation testing strives for perfection in creating realistic end-to-end scenarios, production replay is more practical for this approach. However, there is no focus on feature testing for real playback. Through simulation, one can modify the user journey to target certain pages more or exclusively. Simply replaying real traffic would not help a company plan for growth or to scale properly.

How to select the appropriate test?

In summary, replaying production workload removes the benefits of having a controlled workload and scenario. This is however possible with simulated traffic, though there is no exact representation of user behaviour. Testing decisions though can be made based on a good model of impact and risks. The choice of which manner to test in production therefore depends on each’s pros and cons. These are further factors and considerations that need to be made.

Discover more about improving performance during trading peaks, - download the ebook here.