Why Testing at the End Doesn't Work

Testing at the end is a common approach

Most companies follow the approach of performance testing at the end based on the assumption that nothing can be tested before the code is completely written. It is assumed to be a cost effective solution, as the test team only gets involved at the end of the cycle.

Testing at the end leads to poor quality and high cost 

We recently encountered a client who was replacing their online shop with a new solution. They performance tested at the end, and found a number of performance defects. It was very difficult to identify the root cause of the defects and due to the pressures to deliver the project on time, the client went live on the new solution, despite a number of defects.

A few weeks later, during  the client's annual peak in demand, the web site crashed. This was followed by another outage when subsequent performance fixes were deployed to live. These fixes had not been thoroughly tested, due to the need to deploy quickly. The outages had a catastrophic impact on the company’s reputation and potentially resulted in a loss of customers to competitors.

A significant cost was then incurred as the client tried to identify and fix the performance defect and stabilise the service.

How do we deliver quality software whilst controlling project cost?

In our example, the test team had to reverse engineer to understand the cause of the crash. The root cause was the  cumulative effect of four different code changes:

We know that fixing performance defects in live is 100-times more costly than fixing during the design phase:

In our example, a huge amount of time and cost was spent by the development, test, regression and release management teams to fix the issues. Even if we test at the end and find defects, it is 15-times more costly than fixing in the design phase. So why not shift left and fix the issues earlier, rather than at the end?

Our Software Performance Engineering service, along with Risk Modelling, kicks in right at the requirements and design phase of the project, identifying and eliminating risk as soon as possible. This means less time is needed in early testing, which in turn reduces time spent regression testing before release. Which finally reduces the number of critical bugs leaking in production.  And because things aren’t broken – everyone can move faster. New functionality is added on time and to budget.

References

Integrating Software Assurance into the Software Development Life Cycle (SDLC); Maurice Dawson, Darrell N Burrell, Emad Rahim, Stephen Brewster, Oklahoma State University - Main Campus

If you would like to learn more about our Modelling and Performance testing solutions, please click below, to see our latest webinar.