The Democratization of APM

APM Predictions for 2018

As 2017 comes to an end and the new year begins, there are more and more predictions about trends and topics in 2018. APMdigest printed an interesting 4-part compilation about 2018 Application Performance Management Predictions which includes my prediction of Democratization of APM. I wanted to describe the concept of Democratization of APM in a bit more detail.

In the APMdigest article, I stated that the democratization of APM will bridge the divide between Ops, Dev and SRE teams and improve communication for better quality of service. This concept has significant benefits to the organization and users as you will see.

For an Application Performance Management (APM) vendor it is important to watch trends and technologies as an APM tool has to adapt to these trends. Sometimes these are minor changes, like supporting a new framework and every 5-10 years a major change disrupts the whole industry. The rise of microservices, containers and serverless has been such an earthquake that is shaking up the traditional APM vendors.

Looking back on 2017, this trend has materialized in all kinds of businesses from small startups to big enterprises. Every business is adopting containers at a fast pace, using new platforms like Kubernetes and cloud based platforms like Google Container Engine or AWS ECS/Fargate.

The fast adoption of these trends, which lead to a lot of complexity and challenges in managing the new applications, is just a reaction to a far bigger trend which is part of whole industries changing in a more digitized world and adopting software as their business DNA.

Democratization of Technology

The need for faster release cycles to adapt to these business challenges has led to a strong adoption of DevOps.

DevOps (a clipped compound of "development" and "operations") is a software engineering culture and practice that aims at unifying software development (Dev) and software operation (Ops). The main characteristic of the DevOps movement is to strongly advocate automation and monitoring at all steps of software construction, from integration, testing, releasing to deployment and infrastructure management. DevOps aims at shorter development cycles, increased deployment frequency, more dependable releases, in close alignment with business objectives. [Wikipedia]

The transformation to DevOps is a big cultural change which affects people, processes and technology. One of the fundamental and often forgotten paradigms that enables DevOps is Democratization of IT.

Democratization of Technology refers to the process by which access to technology rapidly continues to become more accessible to more people. New technologies and improved user experiences have empowered those outside of the technical industry to access and use technological products and services. [Wikipedia]

The cloud is one of the biggest game changers of our industry by democratizing all layers of IT infrastructure. With EC2 and S3, AWS made servers and storage available to developers just by pressing a button or calling an API. And cloud providers added a lot of services: databases, load balancers, container platforms and serverless services, where developers do not have to bother about infrastructure at all and can just execute their functions in the cloud.

This also leads to a change in profiles when transforming to DevOps and Cloud Native architectures. As things are available out-of-the-box or can be deployed by the developer as part of the application with containers being the only contract they have, less experts are available and needed.

As an example we could look at databases. A few years ago, choosing a database was a strategic corporate level decision. If a company had chosen Oracle, they had separate DBA teams that operate, monitor and optimize the instances and clusters. They were a gatekeeper for every deployment. When a developer needed a new database they would have to show a design, discuss it with the DBAs and wait until an instance was delivered. I personally had projects where dynamic SQL was forbidden by the DBAs for security and performance reasons.

Today a developer can spin up an AWS RDS instance in seconds and scale it easily or can deploy a MongoDB instance as a container with the microservice application he builds. This is democratization at its best - there is no centralized decision making anymore and no limitations.

The same is true for Application Performance Management. APM used to be an expert field. I have been in the APM space for almost 20 years and have worked with hundreds of customers. With almost every implementation I was part of in the past, there were only a few people (1-2 I would say on average) that were able to really use the APM tool they had chosen. APM tools were complex to configure and it was an expert task to identify problems by manually correlating tons of data.

Coming back to DevOps which strongly advocates automation and monitoring all steps of software construction, this also means that every member of a DevOps team requires access to APM to be able to operate their service and drive optimizations in terms of performance, stability, user experience, and business value.

There are three drivers for Democratization of APM in my point of view:

  1. Automated Visibility
  2. Artificial Intelligence
  3. User Experience

Automated Visibility and APM

One of the elementary values of APM is giving the user visibility into their applications so that the mysterious black box turns into an understood white box.

With the complexity and dynamism of modern cloud native applications and the demand of every APM user getting visibility quickly without having the experts knowledge of doing complex instrumentation, correlation and configuring tons of plugins, democratization requires an APM tool to work out-of-the-box, providing visibility without configuration:

  • Attaching to application services and automatically instrumenting critical code, exceptions, logs and distributed calls to other services
  • Attaching to processes and services and automatically instrumenting critical metrics and telemetry data, like garbage collection usage, thread states or cache sizes
  • Providing automated infrastructure visibility and metadata - especially from cloud environments like AWS but also from container infrastructure such as Kubernetes or DC/OS
  • Providing automated visibility into the End-User-Experience by automatically instrumenting client code in Browsers and Mobile Applications
  • Providing automated visibility into applications, services and APIs by understanding boundaries, frameworks and deployment units to automatically identify the logical units under monitoring

Artificial Intelligence and APM

We see AI as a support system for the user which could ultimately lead to autonomous operation in the future. Today our AI supports the user to make APM easier to use:

  • Providing a Full Stack Application model that keeps track of all the dependencies of services and infrastructure components
  • Providing automated and dynamic thresholds for metrics that keep track of the health of single metrics and components
  • Providing Key Performance Indicators for each service and understanding a normal behaviour of these KPIs to understand the quality of the service. Automate the process of alerting by understanding anomalies and outliers
  • Predict problems by analyzing trends and patterns on the underlying event streams and monitoring KPIs
  • Automate root cause analysis by automatically correlating data based on the Full Stack Application Model and the intelligence of predicting and understanding problems

User Experience and APM

The user experience (UX) of APM will change dramatically as democratization gets more important. Not only will every developer need access to important monitoring data, but also new user groups like product managers, site reliability engineers (SRE) and managers will need the information to optimize the business faster and with greater frequency.

This means that APM tools will have to support different users and use cases out-of-the-box.

Experts will want to “slice and dice” data in realtime to support their tasks of optimizing infrastructure and applications and will need to combine this with cost relevant data and business metrics to understand the value and price of each change.

On the other hand, non expert users will want to have easy access to their information and APM tools will have to present data in a new compatible way - supported by AI to help finding the needle in the haystack and allowing to see optimizations and opportunities quickly.

Democratization of APM and Instana

At Instana, 2018 will be the year of Democratization of APM – we are planning to release our UX 2.0 concept early this year which will change the way APM is used and will allow more users groups to access the highly valuable data the APM provides.

We are also optimizing our Automation and AI to go beyond the advanced capabilities we already have today to provide intelligence for operating modern applications on a new level.

Our team at Instana is really looking forward to 2018, and I am proud to work with such a talented group, including our great customers, to bring APM to every user that needs it.