Continuous Integration with Performance Certification and Results
February 6, 2014 Leave a comment
By Murshid Mustafa – Development Manager – ADP Cobalt Performance Team
Often performance engineering teams face challenges fitting into an agile practice or methodology. As finding, debugging and fixing performance issues can take considerably longer time compared to functional problems. So it’s critical to expose the issues early in the development cycle. Early failure is cheap and easy to fix.
Typical hurts of a conventional model
- Found performance issue(s) a week before the release date
- Can not release business committed new feature
- Changes to the product delivery date
- Uncertainty in the release creates a culture of stress
- Unhappy customers
In a typical software development cycle, performance certification fits in at the end. It gets delayed till the product is fully baked with unit and functional testing. This leaves all the performance bugs rolled up over time creating a snowball effect.
As we understand performance problem identification and resolution can take considerably longer time compared to a functional issue resolution. This practice in many cases causes late detection and resolution of the performance issues. Causing high risk to the releases.
Why continuous performance integration and results?
- Early detection of the performance problems
- Decreasing the cycle time by resolving performance issues faster
- Better release planning with peace of mind
- Save time and resources
- Increase business revenue targets
- Happy customers
One of the biggest challenges in bringing performance certification early in the development cycle is product maturity or functional stability. So its true that we can’t fully evaluate peak product performance early while the product is under development. But we can break the entire performance suite into small pieces and design the tests to touch the functionality that is ready for certification. Stability of the system on stress and system reliability can be measured post-functional freeze.
The approach can be different if we are developing a brand new product compare to releasing new functionality of an already marketed product. We at ADP/Cobalt are seeing the benefits of it almost every day.
The biggest value that continuous integration brings is the availability of performance results right away. Having build-to-build comparison makes the life easy detecting issues in between the builds. The graphs above shows how easy it gets to detect the issues. This saves critical time debugging, identifying the root cause of the issue. It’s huge considering how difficult and time consuming it can get debugging a performance problem.
Be aware of the challenges
Achieving continuous integration with performance presents several challenges.
- “System configuration and code dependencies are not in sync” This can be a hit to the productivity of the CI system. Always be on top of the code changes and new features making sure the system is configured properly to delivering expected output.
- “Debugging or resolving issues in the code or in the system may require working with other teams e.g. development, IT, etc.” The resolution of these issues should be very timely. Make sure the stakeholders understand the needs and time constraints in order to have this system up and running at all times.
- “System is not scalable” It’s possible over time with added features and functionality the system may deteriorate its scalability. Plan regular system maintenance and perform capacity planning studies. Make proper adjustments to the hardware and configuration to be able to run the system with proper performance and scalability.
- “The results are varying run-to-run” Continuous integration without the accuracy of the results is meaningless. We have to run these tests in a controlled environment and inputs/outputs of all systems (configurations, network, load) should be understood correctly.
- “Execution time of the tests is very long” Often performance tests suites takes long time to run; in order to generate results that gives proper average to compare. Also tests that gage performance, reliability and stability over time can be significantly longer. Choose your tests wisely, not every test has to be part of the CI. In order to have effective continuous integration, keep a good balance of tests and the number of build getting the daily coverage.
- Continuous integration with performance certification has it’s cost
- Partner with development and provide a self service model to get maximum benefits
- Always maintain a controlled, clean room performance environment
- Fixing performance bugs can be expensive on the clock so better start early
- It requires a change in culture, make sure all organizations Dev, QA, RM, IT are enrolled with this philosophy
- Yes it can be done