Key Performance Metrics to Watch in Load Tests

Imagine that your team is working on an e-commerce web application with high functional test coverage. You've even measured response times under load to ensure that it could handle the anticipated peak load. Everything checks out ahead of a key holiday shopping day, but as traffic spiked, shoppers began complaining about errors.

It turns out that you were watching the wrong performance metrics. Response times were low, but error rates were relatively high under pressure. Expensive queries were causing databases to become unavailable, and customers were unable to make purchases. The entire scenario could have been avoided by watching the right performance metrics.

Let's take a look at how to avoid scenarios like these by watching the right metrics and using the right tools.

Are you looking at the right performance metrics and using the right tools in your load tests?

Why Should You Load Test?

Developers and test engineers use many different types of tests. Unit tests ensure that a piece of code returns the right result, integration tests ensure that pieces of code work together and user interface tests ensure that the visual elements work across different devices and platforms.

Load tests, also known as performance tests, are often the most neglected type of test. If functional tests pass, it's easy to assume that the application is ready to launch. The problem is that, while some pieces of code may function, they may not scale well to hundreds or thousands of users.

For example, suppose that a developer writes a piece of code that contains an N+1 query that puts unnecessary load on a database. The query will return the correct result and may pass functional tests, but when the query is executed thousands of times, it could overload the database and crash the application.

The worst part is that these issues may not be noticeable until it's too late. For example, an e-commerce website could work just fine until Black Friday or a startup's web application might work perfectly until it gets picked up on Product Hunt. These are the wrong time for something to go wrong!

LoadNinja cuts down on load testing time by enabling test engineers to record and instantly reply load tests without tedious correlation or coding. Since we use real browsers, you also don't have to worry about adjusting scripts with dynamic elements like session IDs and cookies.

Performance Metrics to Watch

There are many different ways to measure the performance of an application. While response times are the most obvious metric, they don't tell the whole story when it comes to performance. For example, average response times may be low, but there could be a significant number of outliers with unacceptably high response times.

Before looking at load testing metrics, it's important to consider how to conduct load tests. Most load testing tools use protocol-based approaches that mimic requests and responses. It's important to test with real browsers in order to gain the most accurate insights — especially with front-end heavy apps.

LoadNinja generates loads using tens of thousands of real browser to provide the most accurate performance data. For example, a React application with a huge front-end will include both the time to download the JavaScript and the time to compile and use it on the front-end.

Let's take a look at six of the most important metrics to watch and the value that they provide.

Response Metrics
  • Average response time measures the average amount of time that passes between a client's initial request and the last byte of a server's response, including the delivery of HTML, images, CSS, JavaScript, and any other resources. It's the most accurate standard measurement of the actual user experience.
     
  • Peak response time measures the roundtrip of a request/response cycle, but focuses on the longest cycle rather than taking an average. High peak response times help identify problematic anomalies.
     
  • Error rates measure the percentage of problematic requests compared to total requests. It's not uncommon to have some errors with a high load, but obviously, error rates should be minimized to optimize the user experience.
Volume Metrics
  • Concurrent users measure how many virtual users are active at a given point in time. While similar to requests per second (see below), the difference is that each concurrent user can generate a high number of requests.
     
  • Requests per second measure the raw number of requests that are being sent to the server each second, including requests for HTML pages, CSS stylesheets, XML documents, JavaScript files, images, and other resources.
     
  • Throughput measures the amount of bandwidth, in kilobytes per second, consumed during the test. Low throughput could suggest the need to compress resources.
How to Analyze the Data

Most load testing tools provide a visualization that breaks down response times, error rates, and other key performance indicators. In addition, they may provide statistical data to support the charts that provide insights into averages, outliers, and other helpful metrics.

LoadNinja-Analytics.png

Example of LoadNinja Analytics

LoadNinja goes a step further by helping test engineers diagnose the actual bottlenecks. By breaking down performance by step and action, you don't have to dig deep to find the culprit for performance issues. Our real-time VU Inspector and VU Debugger make it very clear where the issues lie.

In addition to analyzing the data, it's important to ensure that it's relevant and up-to-date. Load testing should be integrated into continuous integration cycles to quickly identify issues before they reach production. For example, many development teams include it as part of a nightly build.

The Bottom Line

Load tests are just as important as functional tests in the Agile development process. By incorporating load tests into CI processes, you can avoid costly downtime during peak usage and ensure the best user experience at all times. LoadNinja makes the process much easier from start to finish.

Sign up for a free trial of LoadNinja today and see how easy it is to get started!