What is continuous performance testing

In the world of e-commerce, performance issues can lead to substantial revenue loss, presenting a crucial challenge for enterprises. This scenario is particularly distressing for engineering teams, as nothing upsets them more than the frantic troubleshooting and patching required in the middle of the night. And for business owners witnessing customers migrate to competitors during the year’s busiest sales periods simply because their website or mobile application performs faster is profoundly disheartening.

Here are a few examples of how software performance issues can affect busy e-commerce platforms:

• 2012: Amazon estimated that a one-second slowdown in page loading would cost it $1.6 billion in sales every year.
• 2013: 39% of e-retailers claimed they lost money due to performance or stability problems.
• 2014: Catchpoint Systems conducted a comprehensive evaluation of the aggregate web app performance on Black Friday and found out that compared to the identical timeframe in 2013 desktop web pages were 19.85% slower, while with mobile web pages the situation was even worse – 57.21% surge in loading time.
• 2015: Some major e-commerce sites buckled under the heavy traffic loads during Cyber Monday peak retail traffic. 

With stakes this high, it is not surprising that companies are willing to make substantial investments in solutions that can quickly identify and address performance bottlenecks. The central question is: “How to do it efficiently, systematically and dependably?”

Continuous performance testing can help prevent these problems by tackling them during the development process instead of after an application is in production. By establishing a comprehensive test environment, businesses can reduce the need for last-minute troubleshooting, enhance their application’s speed and responsiveness, and provide a seamless experience that meets all functional requirements.

The idea behind continuous performance testing is quite simple: as software engineers routinely and continuously integrate new code, develop additional features, and make changes to an application, wouldn’t it be great to know what kind of impact each alteration has on the app’s performance? To do that, the new application build has to be tested for performance under stress – after the build process is complete, but before the build is released to production. 

Continuous performance testing is gaining increased attention and support from the developer and vendor communities. This highlights the growing recognition of the crucial role that sustained performance optimization plays in delivering exceptional user experiences. Since most development teams already have a functioning Continuous Integration (CI) process, performance testing simply becomes a step in the CI pipeline. It is typically carried out after regression testing but before production deployment. This strategic placement allows for the systematic identification and resolution of performance bottlenecks in a controlled testing environment, well before the application enters production.

The goals of performance tests can be broadly classified into four distinct categories, each addressing crucial aspects of software optimization and end-user satisfaction:

Performance testing objectives must correlate to operational needs and represent real-world business scenarios concerning actual customers. Start with input from the business side: what transaction volume and response latency is required to support the immediate business? What scenarios are most critical: checking out many small orders simultaneously or processing a relatively modest number of “big baskets” containing dozens of items? Each company has different performance bottlenecks. These KPIs must be met by every production release, becoming the top priority in all test cases throughout the maintenance cycles.

Even if your current applications exhibit excellent performance, this doesn’t guarantee that you won’t experience any crashes in case of a sudden surge in the number of customers, requests, or data sets. Preparing for growth presents a significant challenge. Running regular performance tests helps build a capacity planning model and provides an opportunity to automatically scale your production environment before a surge in demand triggers any serious issues.

Business users typically receive either “green” reports that tell that performance is good or panic reports about catastrophic failures. There is nothing in between. Yet, a well-rounded continuous testing strategy should include a bunch of key metrics, like response time, maximum throughput, resource utilization, etc. These metrics allow users to see the performance dynamics and prepare for potential changes in advance. 

A comprehensive understanding of your weak points and the load factors that can potentially disrupt your digital services is crucial for formulating both upgrade plans and backup strategies. While conducting a system-wide capacity upgrade well in advance of demand isn’t always feasible or the sole solution for managing performance risks, understanding your vulnerabilities and potential areas of exposure remains a crucial aspect of sound risk management. Performance stress testing serves as the means through which you uncover these weak points, supplying you with all the essential information ahead of time to facilitate informed decisions.

The challenge with continuous performance testing lies in the practical application of this approach within dynamic development environments, ensuring that it seamlessly integrates into existing workflows without causing disruption. Making a fully automated performance regression and stress testing system is a time-consuming and complex task, which for many development teams still falls into the category of “rocket science”. We will look at the complexities in detail in the next blog post.

Grid Dynamics has always been an avid supporter, advocate and pioneer of continuous performance testing. Performance tests are an integral part of how we design, certify and deliver all mission-critical digital platforms for our clients. 

This practice fully matches our standards, especially when it comes to cloud enablement and continuous delivery. To solidify our experience in this area in 2010 we released Jagger, an open-source framework for continuous automated performance testing. It was created to support large-scale digital platform development for our clients.