Continuous Integration with Performance Certification and Results

By Murshid Mustafa – Development Manager – ADP Cobalt Performance Team


Often performance engineering teams face challenges fitting into an agile practice or methodology. As finding, debugging and fixing performance issues can take considerably longer time compared to functional problems. So it’s critical to expose the issues early in the development cycle. Early failure is cheap and easy to fix.

Typical hurts of a conventional model

  • Found performance issue(s) a week before the release date
  • Can not release business committed new feature
  • Changes to the product delivery date
  • Uncertainty in the release creates a culture of stress
  • Unhappy customers


In a typical software development cycle, performance certification fits in at the end. It gets delayed till the product is fully baked with unit and functional testing. This leaves all the performance bugs rolled up over time creating a snowball effect.

As we understand performance problem identification and resolution can take considerably longer time compared to a functional issue resolution. This practice in many cases causes late detection and resolution of the performance issues. Causing high risk to the releases.

Screen Shot 2014-02-05 at 11.54.23 PM

Why continuous performance integration and results?

  • Early detection of the performance problems
  • Decreasing the cycle time by resolving performance issues faster
  • Better release planning with peace of mind
  • Save time and resources
  • Increase business revenue targets
  • Happy customers


One of the biggest challenges in bringing performance certification early in the development cycle is product maturity or functional stability. So its true that we can’t fully evaluate peak product performance early while the product is under development. But we can break the entire performance suite into small pieces and design the tests to touch the functionality that is ready for certification. Stability of the system on stress and system reliability can be measured post-functional freeze.

Screen Shot 2014-02-05 at 11.53.38 PM

The approach can be different if we are developing a brand new product compare to releasing new functionality of an already marketed product. We at ADP/Cobalt are seeing the benefits of it almost every day.

Screen Shot 2014-02-05 at 11.55.18 PM

The biggest value that continuous integration brings is the availability of performance results right away. Having build-to-build comparison makes the life easy detecting issues in between the builds. The graphs above shows how easy it gets to detect the issues. This saves critical time debugging, identifying the root cause of the issue. It’s huge considering how difficult and time consuming it can get debugging a performance problem.

Be aware of the challenges

Achieving continuous integration with performance presents several challenges.

  • “System configuration and code dependencies are not in sync” This can be a hit to the productivity of the CI system. Always be on top of the code changes and new features making sure the system is configured properly to delivering expected output.
  • “Debugging or resolving issues in the code or in the system may require working with other teams e.g. development, IT, etc.” The resolution of these issues should be very timely. Make sure the stakeholders understand the needs and time constraints in order to have this system up and running at all times.
  • “System is not scalable” It’s possible over time with added features and functionality the system may deteriorate its scalability. Plan regular system maintenance and perform capacity planning studies. Make proper adjustments to the hardware and configuration to be able to run the system with proper performance and scalability.
  • “The results are varying run-to-run” Continuous integration without the accuracy of the results is meaningless. We have to run these tests in a controlled environment and inputs/outputs of all systems (configurations, network, load) should be understood correctly.
  • “Execution time of the tests is very long” Often performance tests suites takes long time to run; in order to generate results that gives proper average to compare. Also tests that gage performance, reliability and stability over time can be significantly longer. Choose your tests wisely, not every test has to be part of the CI. In order to have effective continuous integration, keep a good balance of tests and the number of build getting the daily coverage.


Key learnings

  • Continuous integration with performance certification has it’s cost
  • Partner with development and provide a self service model to get maximum benefits
  • Always maintain a controlled, clean room performance environment
  • Fixing performance bugs can be expensive on the clock so better start early
  • It requires a change in culture, make sure all organizations Dev, QA, RM, IT are enrolled with this philosophy
  • Yes it can be done

About collectivegenius
Everyone has a voice and great ideas come from anyone. At Cobalt, we call it the collective genius. When technical depth and passion meets market opportunity, the collective genius is bringing it’s best to the table and our customers win.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: