What is Test Coverage? 

Test coverage refers to how well the number of tests executed cover the functionality of an application. In other words… How many bases, in terms of application functionality, are covered with your testing processes? Evaluating test coverage helps you guauge what you're testing and how much you're testing. 

Test Coverage vs. Code Coverage

Both measurements can be useful in assessing the quality of your application, but it's useful to define the difference between the two. 

Test Coverage

Test Coverage stems from system requirements and is considered a QA team activity. Test coverage refers to how well the number of tests executed cover the functionality of an application. This can be thought of more of a black-box test, since it is purely focused on testing functional or non-functional elements of the application without having 'insider' knowledge of the internal application design or structure. 

Code Coverage

Code Coverage involves testing specific lines of code through unit testing practices and typically is done by developers. Code coverage refers to which application code is exercised when the application is tested. Code coverage can be referred to as more of a white-box testing activity because it is usually necessary for the tester to have a keen understanding of the applications internal code and structure. 

Different Types of Test Coverage

Product Coverage

What are the functional areas of the product you’ve tested? Evaluating product test coverage helps understand from a functionality and features perspective, which components are we exercising and confident that we understand the level of performance we can expect in production. 

For example, let's say you own an eCommerce platform. In an eCommerce application you might want to ensure you can test the Account Creation, Login, Search, Add to Cart, and Check out functions, as they're core to the purpose of your application and end-user experience. Product Coverage should always be taken in consideration with end-user experience. 

Risk Coverage

What are the risks associated with not testing certain components/features/scenarios? Risk Coverage helps illuminate some of the key areas from a business perspective. This involves evaluating the user-journeys or product functionalities that have the most associated risk if something were to go south in production. A good way to evaluate risk is by invoking the following formula:

Risk = Probability of Occurrence x Loss Due to Occurrence

The goal here is to estimate what the value of each test that you could run would be and to prioritize based on business needs. Oftentimes, it seems dauting for testers to come up with a good estimate for the value of a loss - one important thing to note is that this information is typically available or tracked by other departments. For example, if you manage a web property or eCommerce platform, your organization's demand generation team within the Marketing department has key KPIs and metrics that they track that can provide a concrete foundation for your risk analysis. When you're trying to estimate risk, try to avoid reinventing the wheel - be collaborative and ask other departments (Marketing teams and Operations teams are a great place to start) for key user journeys and the cost of downtime to use in your estimates. 

Challenges With Performance Test Coverage

On the surface, performance testing can seem like a daunting task. There's a level of hesitation, intimidation, and fear that can really hinder a teams ability to execute properly. Part of this is the desire to test everything, which is not always a reasonable expectation. Plus in a world where everything moves quickly and a high standard of performance is expected, it can be difficult to continue to efficiently make progress. The top challenges teams face include:

• Determining Coverage
• Executing on Coverage

Determining Coverage

Determining Coverage can be really challenging. Teams usually are torn between:

1. Doing too much
2. Doing too little
3. Focusing on testing paths that are not relevant to the expected use cases

A lack of priority and direction can hinder the efficiency and effectiveness of performance teams. To combat this, teams can invoke the Pareto principle to help evaluate and prioritize what tests should be run. The Pareto Principle, also referred to as the 80:20 Rule, states that 80% of results in a system come from 20% of the causes. What this means is that essentially, only a few main variables affect the outcomes, and most other factors will return little or no impact. 

The Pareto principle helps improve the overall software process throughout the Software Development Life Cycle (SDLC) and has heavily influenced the Agile management model, where 80% of the team’s efforts are focused on what is essential at any stage of the product’s development.

The 80/20 rule in Agile is a flexible solution that helps make the development process more efficient and reliable. With this in mind, it’s easier to prioritize & identify the scenarios that yield highest value for end users.

Statistics show that users never use 45% of an app’s features, 19% are in rare use, 16% are used occasionally, while only 20% are used frequently or always. Based on this statistic, we can sum up that if we focus on 20 percent of the core functionality of the app, we can get real benefits and guarantee its further development. The remaining 80% of features will serve as additional bonuses for more sophisticated users.

Applying The Pareto Principle

1. Step 1: Map the typical set of expected user transactions
2. Step 2: Assess the % of usage among active users
   1. For existing features: See traffic and usage
   2. For new features: Guesstimate
3. Step 3 (optional): Assess risk when users have bad experience in transactions
   1. Risk = Probability of Occurrence x Loss Due to Occurrence​

Executing on Coverage

Executing on Coverage can be difficult for a number of reasons. What we've seen with our customers and partners is that executing on coverage becomes challenging for the following reasons: 

1. Keeping up with an agile team
   1. Traditionally, creating performance test scripts take lot longer than functional scripts
2. Making performance results readable and consumable by dev team
   1. Translation time during handoff from performance tester to developer
3.  Corelating performance results to end user experience 
   1. Translating request-response time in results to experience end users face

Overcoming Execution Challenges

Challenge: Script creation takes anywhere between several hours to few days just for transactions

Solution: Create scripts from the client side that requires –

• No dynamic correlation or programming for playback
• No scripting to capture client side AJAX

Challenge: Translation time is long during handoff from performance tester to developer

Solution: Use tools that provide developer-friendly metrics that is understandable out of the box

Challenge: Corelating test results with end user experience

Solution: Use real browsers at scale for load tests to get a holistic view of how end users experience your application under stress

How Can LoadNinja Help?

1. Make test creation more efficient with InstaPlay
2. Data-drive different scenarios
3. Get browser-based results LoadNinja’s TrueLoad technology

LoadNinja is the only performance testing platform on the cloud to load test web applications with real browsers at scale. It has helped teams both large and small become more efficient,while increasing accuracy and quality in every single one of their builds. 

Check out the full webinar (including a video of how to do this in LoadNinja) here.