Any engineering team can deploy new software frequently (i.e. multiple times per day or week). What’s really hard is deploying new software frequently without introducing performance or stability regressions. Regardless of the deployment strategies used by your organization, it’s absolutely critical that you have the proper visibility into performance and stability across the entire software development and operational lifecycle. At Instana, our customers have shown us how monitoring and analysis needs to change as organizations mature their build and delivery processes.
Organizations that use the waterfall methodology and deploy new software infrequently (monthly or less) have a significant need for production root cause analysis. This is due to the tendency for these organizations to have application architectures that are either monolithic, SOA, or a combination of both. SOA and monolithic applications are known to have large amounts of code that perform many various functions, are difficult to debug, and have too many functional scenarios to test properly in pre-production. As a result of all these factors, production root cause analysis is critical since the majority of problems will show up after deployment to production.
Organizations that have adopted CI/CD and/or frequently deploy new software (daily or weekly) have less need for production root cause analysis and much greater need for immediate release quality analysis. This is a direct result of what is shown in following charts:
It may seem counter-intuitive at first, but deploying multiple times per day can lead to significantly fewer defects in production. The charts above contain real data from an Instana customer. When you look at the charts you can see that when deploying once or less per day, this customer had hundreds of alerts per day. When they implemented an approach of utilizing microservices, deploying multiple times per day, and gaining immediate feedback with Instana, the resulting alert rate dropped to less than 10 per day. If you look closely at the chart you can see they went through some learning pains but quickly figured out how to get it right.
The Key to Deploying Better Software Faster
There’s no single, silver bullet to successfully deploying multiple times per day and reducing alerts at the same time BUT you absolutely cannot get there without continuous, immediate performance and quality feedback at every stage of your build and deploy pipeline. The image below depicts the various stages of the software development lifecycle and indicates the multiple opportunities to make informed decisions to either rollback or promote forward.
If you’re to the point where you can deploy multiple times per day, it means that this entire process takes a few hours or less from the time code is committed through production deployment. A few hours is not a lot of time to implement monitoring and determine if your build is acceptable as it progresses through integration testing, functional testing, load testing, canary testing, etc. If you have to manually configure monitoring, forget about it. If you have to manually dig through log files, forget about it. If you have to manually correlate response times, with container metrics, with host metrics, etc … forget about it.
How to Achieve Continuous, Immediate Feedback
I won’t sugar coat it. Achieving continuous, immediate quality and performance feedback is a serious technical challenge. Here are the major problems to solve:
- Discover and monitor the full technology stack without human intervention
- Create observability within every service for every request type without manual coding
- Trace every request without sampling
- Stream all collected traces and data to backend in real-time
- Correlate all data from every layer of every component in real-time
Addressing each of those technical challenges requires solving multiple other technical challenges. It’s a rabbit hole you probably don’t have time or desire to dive into. If you can actually build all of these capabilities on your own, you might as well create your own company and start selling your monitoring product. Of course, Instana has already solved all of those problems and more.
Instana discovers, monitors, traces, and analyzes your software deployments immediately. It provides the ultimate level of visibility and understand when you need to fix issues in production. You can try Instana in your own environment with our free trial today.