Performance testing requires careful planning and preparation in order to ensure smooth running, and minimise delay.
Some factors are beyond the ability of the performance tester to influence. Examples of these include: software delivery, environment build and deployment, and software stability, test harness and stubs, etc. If the risk of these factors is high then planning contingency is necessary.
However, there are a number of decisions and performance preparation tasks that need to be resolved well in advance of the test execution phase. Some of these tasks can be complex and can involve external dependencies, and require budget provision and perhaps procurement. It is not uncommon for these tasks to be overlooked or minimised by project managers.
The strategy provides a framework for conducting performance testing. The strategy should be defined early in the development lifecycle once the requirements and architectural outline of the project are known. The following elements are highlighted,
The performance test strategy is a high level document and does not replace the more detailed performance test plan. It is worth considering if the performance test strategy should be combined with the overall test strategy owning to the synergies with operation, tooling, and other aspects (data, build availability, environments, etc.)
Performance risks assessment is conducted to establish the dominant performance risks of the system being changed. Risk is defined at various levels including: Infrastructure, system, service, component, function. Risk is rated of in terms of the likelihood of occurrence, and the potential impact of the change. Relevant mitigation activities are determined.
By exploiting this process it is possible to hone the performance testing scope to a manageable level, ensuring that testing costs are managed, and performance strategy is targeted in the right way, at the correct time. As importantly, the assessment provides a basis for gaining a consensus on scope.
Performance risks assessment provides input into the strategy and plan.
Performance objectives and goals provide realistic business and technical targets for gauging the outcome of performance test cases (i.e. pass/fail criteria). Performance objectives and goals should be linked back to the requirements included in non-functional requirements and Service Level Agreements (SLAs).
Performance objectives should be set early on in the project lifecycle prior to, and ideally as input into, the design stage. They are typically expressed in terms of:
Performance testing requires that a realistic business workload is applied to the System Under Test (SUT). If the test workload has no basis in reality then test results will be meaningless. Workload characterisation defines the load and session concurrency profiles of the new production system. The output from the process is constituted of the following:
A workload model is built by leveraging the following information:
In conducting workload characterisation it is useful to consider how existing and legacy systems and services are used. Measurement and analysis of legacy systems can provide invaluable data for building or validating the workload model. If the business model of usage is changing this data is less helpful. In this situation, assumptions regarding usage may have to be made and tested at a later point.
Workload characterisation may take several days, sometimes weeks, and should be planned at an early point.
Test scenarios are designed and linked with the workload model. The relevant performance objectives and goals are stated for each test scenario. This ensures that pass/fail criteria are understood. Scenarios are classified as follows:
Performance testing requires a load generation capability in order to emulate a large workload. Whilst it is possible to orchestrate a large group of users to do the same thing, this approach has severe limitations. Ordinarily a load generation tool is used to capture and replay load. It is possible that your company has already made an investment in tooling. If not then other options will need to be considered. Decisions will need to be taken about your requirements. For example,
It is vital to examine the requirements at an early point as evaluation & procurement processes, if required, can be time-consuming.
Free load generation tools should not be rule out. For example, Apache JMeter is a free tool with a credible record. Capacitas have successfully deployed JMeter, and post regular blogs that describe how to get best value from JMeter testing.
Only once the application has been scripted can judgements be made on the level of complexity and the amount of additional script engineering time required. Each application has unique interactions.
For some projects it is prudent to plan a proof-of-concept at an early point once early code deployments are available. The objective is not to create the final test scripts and test load but rather to validate any planning assumptions already made and take action if further activities are required.
The amount of scripting work is usually contingent on the number of scripts, and the scripting complexity (journeys, protocols). It is also possible the external functions may need to be coded where the native scripting language does not provide equivalent functions. Timely preparation is therefore important.
It may be possible to start scripting work at an early point if the application is functionally stable, the user interface is unlikely to change significantly, and an environment is available. Functional environments can be used if the scripts are likely to port successfully. A judgement call is required on the timing. The scripting proof-of-concept may give some guidance.
It is probable that additional facilities will be needed to support performance testing. Some examples are listed below,
Requirements for utilities and facilities should be developed and progress early in the project lifecycle preferably when the application code is being built.
Performance testing requires that the manifestation of performance on the SUT is quantitatively described in order to understand the relationship between the load drivers (E.g. user sessions, transactions, batch tasks) and the symptoms or performance. This is performance measurement.
Load testing tools are typically able to quantify and store test results that characterise the load drivers (Transactions, Users) in terms of their interaction with the SUT. Typically response times and transactions and throughput [volume per unit time] measures are available and a variety of statistics are presented to describe these interactions. Other useful measures of throughputs are available including [HTTP] hits, concurrent processes, Errors, Failures etc.
System, application, and database performance may be measured using a variety of tools. There is an arsenal of potential commercial and free options available to achieve this and selection can be a time-consuming process often taking several weeks. Some tools are quite demanding to install and an allowance may be needed for this.
Bespoke application instrumentation can be built into customised application code to measure the performance of visible and internal application components. Commercial tools also provide this capability but are expensive. If the bespoke route is preferred then the development requirements need to be included in the project scope, and then designed, built and verified.
Performance testing is highly sensitive to the quantity, quality, and consistency of test data. Application data will be needed in sufficient volume, and with the correct attributes (seeding, indexing, coverage, etc.). If this data profile is invalid the pattern of data access will may be incorrect thus giving meaningless, often over-optimistic results
The process of generating test data may be time-consuming, requiring strong technical input on aspects of design, build, and testing. If projects are able to provide representative populations of data then this option is the logical choice. Sometimes data generation tools may be used.
This activity requires serious consideration and planning as a design process where data has to be created from scratch.
The best advice suggests that performance test environments should be as “Like-Live” as possible, cost permitting. Where compromises are required on specifications or loading factors then these may necessitate modelling or extrapolation to ensure that a meaningful performance result is obtained. The process of modelling is not straightforward and needs to be highly abstracted to achieve the best outcome. If modelling is to be pursued then model development and validation time should be included in the project plan.
Time to build the environment will be contingent on supplier’s schedules, where the environment does not pre-exist, and internal resources to build and configure the environment to meet the design. Once the environment is handed over to the performance testing team it is likely there will be residual configuration issues which one must expect. This is a significant challenge in most performance testing projects. The best advice is to pursue this activity from an early point with vigour in order to avoid delays.
To learn more about performance testing, such as its benefits and the secrets of delivery - download it here.