Challenges of continuous performance testing
The main reason website and application performance testing is not already continuous in many companies is clear: it’s hard to implement. Why? Let’s look at a few CPT implementation issues:
Cost: Continuous Performance Testing takes money. Production-like infrastructure and an engineering team of specialists to create, automate, and maintain test cases are two major expense categories. These costs all come out of a project’s budget. Organizations that don’t allow for these expenses before they start implementing CPT, or don’t know how to estimate them, can easily find themselves unable to complete their CPT projects.
Lack of skills: Your current development team may be full of good application and test engineers who are nevertheless not skilled at performance testing. Performance testing is still a rare area of specialization that requires knowledge of multiple disciplines, including performance requirements and analysis; application architectures and design of all elements in the stack; performance profiling and tooling; test data management; design of stress test scenarios; and test automation execution — a skill mix that’s hard to find in the job market.
Availability of adequate performance testing environments: Teams find that it’s hard to allocate, configure, and maintain dedicated environments for performance testing, which must be similar to your production environments right down to configurations and data —and it’s hard to keep them updated, too. The effort involved scares many organizations away from going down this path.
Immature continuous integration and continuous delivery practices: Continuous performance testing is an advanced discipline related to automated software testing, integration and delivery. If the organization doesn’t yet have continuous CICD processes for development in place, it’s almost impossible to run continuous performance testing.
No focus on performance tests, because other tests are “more important”: As the old saying goes, “the squeaky wheel gets the grease.” When an application experiences frequent quality problems, fixing functionality issues is a top priority. Investments in unit tests, basic CI infrastructure, regression testing, and integration testing are typically prioritized over performance testing. It doesn’t help the case for CPT that it is traditionally performed, almost as an afterthought, at the end of the development cycle. This is not an easy mindset to change.
All these challenges can be overcome. The good news is that Continuous Performance Testing is becoming increasingly more accessible and affordable, so more companies are putting it into practice. In subsequent blog posts, we will discuss practical approaches to overcoming the common challenges described above, and present specific recommendations on how to successfully implement CPT.
