Automating Performance Test Analysis to Speed Up Software Delivery

Benefits of Automated Test Analysis

Lots of test cycles require lots of analysis!

Performance testing will happen to some extent during the development of most software solutions. It might occur as a step within an instance of our continuous integration (CI) pipeline before we green light our release candidate, or maybe even as one of the latter stages of a project using a waterfall methodology. The key thing is that once performance testing happens, we need some form of analysis to determine whether or not we’ve met our non-functional requirements (NFRs) and if we have any underlying risks that need to be resolved before we can release our software. One of our clients follows an Agile software development methodology and in a 10-week cycle we run around 50 performance tests for them: 

Once we have run a performance test we should (provided appropriate monitoring is in place) end up with a big pile of raw performance data from different components of the system under test.

In most cases, the next step is to hand all this data to performance analysts and get them to wade through it manually and quickly make sense of it all. They then determine whether or not we have any performance related issues. This, as you will soon find out, has drawbacks!

Manual test analysis takes too long and is error prone!

Simply put, the issue with this approach is people, and when I say people I mean our performance analysts.  That’s not me taking a shot at their analytical abilities or whether or not they’re nice people either, it’s just that humans are predictably slow and error prone!

Once we combine these slow error prone analysts with the heavily compressed timescales applied to our performance testing, you know we’re asking for trouble. They just won’t be able to keep up with the swathes of data being thrown at them to perform the depth of analysis required, which is when preventable defects start to slip through the cracks and we find ourselves in a viscous cycle of move fast and break things, which in the long run can slow us down and cost more money!

Copyright zelger.org

How can we become more agile?

So the questions we now have are: how can we move fast while at the same time building performance into our software/application? How can we scale up how many iterations of performance testing we can do? How can we improve the depth to which we perform our analysis? How can we provide all this within the time and budget we have left to perform it!?

The Answer - automate test analysis!

The answer, or at least the part of it we’re interested in right now, is to automate! That’s not to say let’s automate bits here and there, no we should aim to remove all human interaction from our analysis.

Here at Capacitas that’s exactly what we’ve done, we’ve automated all human interaction in the crucial stage of analysis immediately after the conclusion of a performance test. Years of experience in the field of performance engineering coupled with our in house development expertise have allowed us to craft a comprehensive set of algorithms, which when plugged into our test analysis tool quickly identifies pathologies in data that we know indicate performance issues.

Benefits of automation

The benefits of this are huge :

• We have reduced the time it takes to analyse test results by 58% and we think we can go further!
• We have increased the software/hardware components we perform routine analysis on by 740%
• By codifying the analysis methodology, we can vastly improve accuracy and consistency
• We can quickly expand and improve the methods of analysis we apply to our data
• We can train our junior staff quickly

If you would like to learn more about our Modelling and Performance testing solutions, please click below, to see our latest webinar.