How to Supercharge Automatic Tracing with Kubernetes Primitives

November 16, 2020


Kubernetes has taken the world of containers by storm. More and more companies, small and large, are deploying their containers in Kubernetes clusters they run, or on automated, batteries-included clusters run by Cloud Providers.

There is so much power built into the Kubernetes API! What if I were to tell you that a Kubernetes primitive, called the MutatingAdmissionWebhook Admission Controller, can be used to supercharge observability automation?

Let’s dive in and look at just how these primitives can be used to make monitoring and tracing even better.

Kubernetes Admission Controllers for Extensibility

Kubernetes offers a rich API, with a lot of primitives to work with. One of these primitives is Admission Controllers. According to the Kubernetes documentation:

“An admission controller is a piece of code that intercepts requests to the Kubernetes API server prior to persistence of the object, but after the request is authenticated and authorized.”

Among the many admission controllers built into Kubernetes, two are specifically designed for extending Kubernetes with additional behavior: ValidatingAdmissionWebhooks and MutatingAdmissionWebhooks.

  • ValidatingAdmissionWebhooks allow you to specify an endpoint, provided possibly by a deployment running on the same K8S cluster or some external API, that gets to veto the creation of a particular resource, like a Deployment or a Pod. Validation webhooks are used in many Kubernetes-centric security and compliance use-cases. A paramount example is the Open Policy Agent (OPA) Gatekeeper, which enforces policies executed by the Open Policy Agent.
  • MutatingAdmissionWebhooks also allow you to specify an endpoint, provided possibly by a deployment running on the same K8S cluster or some external API. Unlike the ValidatingAdmissionWebhooks, however, the focus is to modify the resources to be created. Common use-cases include automatically adding certificates to services and other resources needing them (see, for example, kube-lego, kube-cert-manager, and the emerging Cert-Manager)[1], or deploying side-cars in every pod as part of a service mesh like Istio.

In the remainder of this blog, you’ll learn how Instana is using a MutatingAdmissionWebhook to push the envelope of automating observability.

An epiphany strikes

One of the foundational tenets of Instana is that observability should be entirely automated and commoditized. Simply put, you should be able to simply deploy an agent and automatically begin tracing and monitoring everything. Achieving this level of simplicity is actually pretty hard, and we constantly keep working at it. Case in point, and unlike most runtimes we support, the tracing of .NET Core and Node.js support is not yet entirely
automated. Both are very simple to deploy: you add a library to your application and an environment variable or two, and you are good to go. But it is not entirely seamless.

One day, it all came suddenly together: tracing for Node.js and .NET Core would just work if we could make sure that all pods in a Kubernetes cluster fulfilled the following two conditions:

  1. have the right Instana tracer packages available somewhere in the pod and…
  2. have the right environment variables set in every container.

Both of the conditions above can be accomplished via a MutatingAdmissionWebhook: it is far from easy to implement, but it would provide an amazing user experience. So I went ahead and prototyped it.

The making of a Mutating Webhook

Let’s break down how the Instana AutoTrace Webhook accomplishes the seamless automation of tracing Node.js and .NET Core across an entire Kubernetes cluster:

  • Make available the right Instana tracer packages in every pod: Containers in a Kubernetes pod can mount volumes. And there is one special type of volume called emptyDir, which is an ephemeral volume (its content disappears when the pod dies). With a MutatingAdmissionWebhook, it is rather simple to ensure that each container in every pod mounts a pod-specific emptyDir volume. But how do we bring the Instana tracers into the emptyDir? Well, with an Init Container, obviously! Which is why there is now the instana/instrumentation image, which we update with pre-compiled tracers for Node.js and .NET Core.
  • Set the right environment variables set in every container: This one seems easy, because you can just edit the PodSpec to add environment variables… until you realize that you actually need different values for some of the environment variables that activate tracing for .NET Core depending on the flavour of the Standard C Library
    that is used by your container’s user-land, musl libc for Alpine-based containers and the GNU C Library (glibc) for virtually everything other base image. I will not go into the details of how we set the right values to inject, because a bit of mystery keeps it interesting, but believe me, it is geeky aplenty!

Getting this far required solving a few more tricky things along the way, and the various Instana teams involved rose enthusiastically to the challenge. Little riles up Instana engineers more than “hey, let’s automate that”.

And that’s it: we have now an MutatingAdmissionWebhook that you deploy with one simple Helm command, and will take care of ensuring that each and every container running Node.js and .NET Core created thereafter is seamlessly, automatically traced by Instana.

What’s next for the Instana AutoTrace Webhook

The Instana AutoTrace Webhook is straightforward to install and easy to operate. When we release a new version of the instrumentation image, you upgrade the Helm chart (which is hosted in our new, spiffy Helm repository at and are up to speed with the latest and greatest Node.js and .NET Core tracing.

So, what are we going to do next to make it even better? Well, we want to integrate the Instana AutoTrace Webhook into the Instana Agent Operator, so that even the upgrade is automatically taken care of for you in the purest Instana Way (and this is but one of the things we have in store for the Instana Agent Operator for 2021; stay tuned because it is going to be a wild ride).

Also we are working on automating tracing for further popular technologies on Kubernetes that are not yet entirely automated by Instana. Because, let’s be frank here: you have way better things to do than setting up observability tools; for example, you could use those observability super-powers to make your applications as awesome as humanly possible!


With the addition of .NET Core and Node.js, Instana now has fully automated tracing with AutoTrace for an industry leading 9 languages. When combined with the power of Unbounded Analytics, which gives you unfettered access to every single trace and call and extremely powerful analysis tools, Instana ensures you get very quickly to the underlying cause of issues in even the most sprawling microservice systems. To see the power of Instana AutoTrace and Unbounded Analytics, sign up for a free trial of Instana today.

  1. As a personal note, I find it a source of never-ending sadness and head-shaking that in 2020, one of the hardest things to do on Kubernetes is still wrangling with certificates. I would love nothing better to see the functionalities of Cert Manager available out of the box on all Kubernetes offerings that do not have them already. And I was very positively impressed by OpenShift’s certificate automation.

Play with Instana’s APM Observability Sandbox

Announcement, Developer, Product, Thought Leadership
At Instana, we recently improved the installation process for our self-hosted customers. Instana’s self-hosted platform now utilizes a fully Docker based installation process. In a previous blog post, Lessons Learned From Dockerizing...
An ever-increasing number of System architectures and deployment strategies depend on Kubernetes-based environments. Kubernetes (also known as k8s) is an orchestration platform and abstract layer for containerized applications and services. As such,...
Announcement, Developer, Product, Thought Leadership
To be successful in Observability, you must have the ability to observe the behavior of a system and derive its health state from it. But deriving the health state of any given...

Start your FREE TRIAL today!

As the leading provider of Automatic Application Performance Monitoring (APM) solutions for microservices, Instana has developed the automatic monitoring and AI-based analysis DevOps needs to manage the performance of modern applications. Instana is the only APM solution that automatically discovers, maps and visualizes microservice applications without continuous additional engineering. Customers using Instana achieve operational excellence and deliver better software faster. Visit to learn more.