Why I Joined Instana
Every once in a while, life presents an opportunity that you just can’t ignore. 5 years ago I got a phone call from a friend at a company I knew very little about. After I explored the opportunity and tested the software I was pretty sure I was making a good move when I agreed to start working for AppDynamics. That turned out to be a great decision both financially and professionally and I am forever grateful for my time there.
Some people say that helping to build a successful company like AppDynamics is a once in a lifetime event. For many of us that’s probably true, but in my case life has presented another opportunity with massive potential.
What is Instana?
In case you never heard of Instana here’s a quick description. Instana is an AI powered Application Performance Management (APM) solution designed for modern application architectures and technologies. AI is definitely a buzzword these days but there are a few companies actually doing it and Instana is one of them.
I’ve worked as an Aerospace Engineer, Systems Administrator, Monitoring Architect, Tech Evangelist, and Competitive Intelligence Manager.
Some of you reading this post will undoubtedly know me and be familiar with my background. Others will have no idea who I am or why you should pay attention to this article. So, for those in the latter group, here’s my background.
My first career was as an Aerospace Engineer working for Lockheed Martin. That was a cool job and while I still love aircraft I felt the need to get into the IT world and leave engineering behind.
Next, I spent a number of years as a Systems Administrator where I was constantly tinkering with kernel scheduler algorithms and trying to tune all of the subsystems for optimal performance. This really got me interested in application performance across the entire stack.
Building upon this, my next role was as a monitoring architect for Wachovia’s Investment Bank and Wells Fargo’s Wholesale organizations. This is where I really learned the most about business applications, their lifecycle, and the myriad of problems that IT practitioners face on a regular basis.
Capping it all off I transitioned from the practitioner side to the vendor side where I was a Technology Evangelist and then ran the CompIntel program at AppDynamics.
I feel fortunate to have had a front row seat at the digital transformation show over the past 20 years. Witnessing the focus shift from hardware to software; from monolithic to distributed; from 3-tier to SOA; from systems of record to systems of engagement; from client server to rich internet application to mobile first; from mainframe to… wait the mainframe’s still not dead. Seriously, will those things ever go away???
And the languages, oh the myriad of languages… C/C++, Java, .NET, PHP, Ruby, Python, Node.js, Go… and whatever awesome language is next.
The transition from physical servers to virtual servers was like a massive tidal wave hitting the IT world. Nearly everyone adopted server virtualization at the same time. It wasn’t perfect for all use cases but boy did it provide some serious flexibility, convenience and speed in the IT provisioning world. (What, I can have a physical server in 1 month or I can have the virtual server in 1 hour? I’ll take the virtual one please.) Sometimes I wonder if anyone has conducted a study to figure out just how much extra disk, memory and CPU we burned up by loading all of those extra operating systems.
Hey, look over there. Here comes the next tidal wave… containers and microservices. This is a shock to absolutely no-one. Getting rid of all those additional OS’s makes a ton of sense as does breaking down larger services into smaller, more focused microservices. But with this shift in technologies and architecture comes a new set of problems that need to be addressed in order to successfully manage such dynamic and complex environments.
The Problems Facing the Container and Microservices Crowd
When everything is working as expected and users are not complaining, life is good. As soon as you run into your first major customer impacting incident the typical response is to gather up just about anyone you can via Slack, HipChat, conference call, etc. and ask, “What do we have for monitoring?” This scene plays itself out across the entire globe daily so don’t feel too bad if you resemble the description.
Monitoring these modern environments is challenging and here are 5 major reasons why:
- Rate of Deployment – One of the great benefits of modern architectures is that they enable an incredible rate of new deployments. Many organizations are deploying new code into production multiple times per day or week at this point. Traditional APM tools require some form of manual configuration for effective monitoring and alerting. Modern APM MUST not require manual configuration of monitoring or alerting.
- Rate of Change – Containers enable portability, right sizing, and dynamic scaling like no other technology I’ve seen before. In fact, I was on a call with a Gartner analyst recently and he stated that over 50% of containers run for less than 1 minute. Traditional APM is built around rigid data models that do not handle change well. They typically only report the data they collect once per minute and have low granularity (1 minute) metrics. Modern APM MUST automatically adapt to dynamic environments and must report data within a few seconds in high granularity (1 second).
- Number of Technologies – Technology is advancing at an incredible pace and many development teams have been given the freedom to choose the best technology for the job instead of being forced to suffer through using the “corporate standard”. Traditional APM uses manually configured plugins or extensions to support new technologies, often just a collection of low granularity KPI’s. Modern APM must be capable of quickly supporting, and then automatically monitoring new technologies shortly after release and treating those technologies as first-class entities.
- Size of Environment – The ability to densely pack individual hosts with containers creates a significant issue for almost every monitoring solution. Environments with 10’s or 100’s of thousands of components create a flood of data. Traditional APM solutions were not architected for this massive amount of data and are forced to reduce the amount of data they collect in order to prevent complete tooling failure. Modern APM MUST be able to ingest and process the data from 10’s and 100’s of thousands of components in real-time.
- Complexity of Orchestration – In order to effectively manage resources in a containerized microservices environment you really need the assistance of an orchestration tool. Kubernetes seems to be the orchestration tool of choice for the time being and although it’s incredibly helpful it is very complex and has a steep learning curve. To make matters worse, if your orchestration tools break (they are just another application after all) then your once dynamic environment becomes completely static until you can repair the orchestration tool. Traditional APM tools either don’t monitor the orchestration layer at all or just provide high level KPI’s. Modern APM MUST understand the orchestration tool and let you know exactly what is malfunctioning at any given time. It must also accurately depict the end result of the orchestration engine deploying containers into the runtime. This is critical for ensuring that orchestration is properly configured and not running away with out of control container growth (a very expensive byproduct of improper configuration).
Wrapping it All Up
So why did I join Instana? There are many reasons but here are the major ones:
- Instana has solved all of the problems outlined above and more. This was no easy task and it required the creation of brand new technologies that are the intellectual property of Instana. Nobody will solve all of these challenges just by cobbling together a bunch of open source big data tools. This gives Instana a major architectural competitive advantage.
- Artificial Intelligence is vital to understanding the performance and quality of a massively complex microservices environment. Humans can’t keep up but machines don’t intrinsically possess the human knowledge to truly understand the root cause of problems. Instana embeds expert knowledge into our AI engine just like an IT veteran would impart knowledge on an inexperienced co-worker.
- Successful companies are built around amazing people. Instana is loaded with exceptionally talented individuals, many of whom I had already worked with in the past. Great teams will accomplish great things together.
Ultimately success requires alignment of product with its intended market. You have to have the right product at the right time solving the right problems. It’s abundantly clear to me that we’ve entered the container age and Instana is in a great position to help companies through the associated management headaches. I’m thrilled to be part of this team and over the next few months I’ll share much more detail on the underlying technology and the value it brings to our customers.