What Is Load Testing?

There’s no doubt that performance bottlenecks on websites can be extremely costly. According to Gartner, the average cost of downtime is $5,600 per minute across all organizations, but they can be even higher for mission-critical services. A five-hour outage at Delta Air Lines in August 2016 cost the company an estimated $150 million!

Load tests identify performance bottlenecks in an application before they reach production users by simulating a load. For example, a load test may identify a N+1 SQL query that wasn't caught by standard functional tests — and there would have been no way to identify the bottleneck without simulating a load.

There are two types of load tests:

• Protocol-based load tests make requests through a protocol, such as the HTTP or FTP. After making a series of requests, the load tests measure the response times, error rates, and other metrics that relate to the user experience.
• Browser-based load tests spin up real browser instances to measure the full user experience. The resulting response times, error rates, and other metrics include the impact of JavaScript, CSS, images, and other assets.

Most load tests are run ad hoc by outside performance engineers, but using modern tools, it's easy to incorporate them into existing continuous integration (CI) workflows. By using load tests and shifting left, organizations can identify problems earlier on and shorten development cycles while minimizing the odds of defects reaching production.

JMeter & On-Premise Solutions

On-premise load testing is common among organizations with sensitive data (e.g. healthcare) or those that run their application behind a firewall. The most popular on-premise load testing framework used by these organizations is JMeter, an open-source tool developed by the Apache Software Foundation, which offers tremendous flexibility.

JMeter’s GUI for Developing Load Test Plans – Source: Wikipedia

After installing the application and building a load test plan using a built-in GUI, test engineers run the scripts via a command-line interface and injectors on any computer and operating system with Java. A CSV (or XML) file is generated during the test with the results, along with an HTML report that provides some of the high level details.

JMeter also supports distributed load testing for organizations that require more firepower than a single computer. Under its recommended approach, a single master controller initiates the test across multiple slave systems, which then generates a desired load, and totals the results for analysis in a single location — translating to great scalability.

With on-premise solutions, there's a lot more flexibility in terms of how and when the tests are run. On the other hand, there's a high initial cost to acquire the necessary hardware and set up the environment, plus ongoing maintenance costs to keep both the hardware and the software components up-to-date.

LoadNinja & Cloud Solutions

Cloud-based load tests are common among Agile organizations that want to get up-and-running quickly at a minimal cost. LoadNinja is one of the most popular cloud-based load testing providers that leverages tens of thousands of cloud-based browsers to run realistic browser-based tests in the cloud.

LoadNinja's Test Script Recorder with Databank – Source: LoadNinja

After recording a test using the platform’s web-based interface, test engineers can easily incorporate the script into their existing CI processes to execute with each merge or deploy. Then developers can diagnose any issues by digging into virtual-user browser sessions and accessing in-depth analytics.

LoadNinja can instantly scale from one to tens of thousands of real browsers with a few clicks without the need for any on-site hardware. In practice, it’s nearly impossible to accomplish the same thing with on-premise hardware. In addition, private endpoints make it possible to run these load tests against applications that sit behind a firewall on local machines.

The benefit of cloud-based solutions is that there are minimal start-up costs and it's highly scalable. On the other hand, the cost per run (not including startup costs) may be higher than an on-premise solution. These costs must be weighed against the benefits of running a load test, but they’re important to consider for smaller firms.

There’s Always the Hybrid Approach

Many organizations could benefit from a hybrid approach between on-premise protocol-based load tests, such as JMeter, and cloud and browser-based tests, such as LoadNinja.

Developers running JMeter on their development machines may be able to generate a quick load that's sufficient to identify immediate bottlenecks before committing any code. That way, there's less risk of a bottleneck down the road and any concerns are fixed before going further.

On the other hand, performance engineers can use LoadNinja for a more comprehensive analysis using cloud-based browsers. These tests may make sense to run before each merge to production rather than with each commit. They can even be configured to run automatically with Jenkins or other CIs to automatically flag any issues.

The hybrid approach to load testing enables developers to catch any problems faster without running ultra-frequent cloud-based load tests at a huge scale. At the same time, the cloud-based tests can be run after important merges to identify any issues before they reach production users to ultimately ensure the highest levels of quality.

So What's the Answer?

The best answer just may be a combination of the two. As we’ve discussed, load testing is a critical part of the modern Agile development process. With both on-premise and cloud-based solutions available, businesses must carefully decide what's right for them in both the short- and long-term.

To find out what suits you best, you should at least try the cloud-based approach.