In my last blog post, I talked about the history and rapid adoption of the OpenTelemetry project. Today we discuss the progress made in terms of end-to-end compatibility between distributed tracing with OpenTelemetry and Instana AutoTrace, and how we are embracing OpenTelemetry as a first-class citizen in Instana.
Despite the foundation provided by W3C Trace Context, end-to-end interoperability is actually much harder than it sounds.
Normally, vendors have to integrate code upstream, inside the OpenTelemetry SDKs you deploy in your code and infrastructure, in order to receive data in a format they can process. This creates all types of headaches because you may need to rebuild and redeploy your applications to add the missing libraries. This task might not be a “biggie” for a PoC, but it becomes daunting at scale when you need to wrangle tens of teams and hundreds of code bases.
At Instana, we really do not want to give you libraries to integrate in your code. One of the fundamental product philosophies of Instana is: “Deploy the Instana agent; it does the work for you.” And we wanted to achieve the same refreshing simplicity with OpenTelemetry, which is why we came up with a unique, super cool way of getting the full value of both Instana and OpenTelemetry without changing any code!
In Instana you will see traces associated with the infrastructure powering those applications, which is one of the killer features of AutoTrace and Unbounded Analytics.
Instana’s support for OpenTelemetry is predicated on:
- Native support within the Instana agent for OpenTelemetry’s otlp protocol that OpenTelemetry uses to transmit data
- Seamless integration with OpenTelemetry applications in Instana’s Dynamic Graph
The support for otlp is straightforward: you point the OpenTelemetry SDK, typically by means of one environment variable, to your local Instana agent, and let the telemetry be delivered to your Instana backend. Today we support collecting traces; metrics support will come in the next few releases.
End-to-end distributed tracing with OpenTelemetry and Instana’s AutoTrace™
Instana AutoTrace and OpenTelemetry support out-of-the-box W3C Trace Context, which is the foundation for end-to-end traces mixing spans (the building blocks of traces in both AutoTrace and OpenTelemetry) generated by the two different tracing systems into one trace.
However, despite the foundation provided by W3C Trace Context, end-to-end interoperability is actually much harder than it sounds. There are a lot of corner cases to deal with in terms of trace gaps, e.g., if one or more applications is not reporting tracing data. For example, we can know in most cases whether we have a gap, but we do not know how large it is (one span or more?) or if the data will come in late(r). Additionally, we had some assumptions to clear from our instrumentation and trace data processing about how trace identifiers are created.
Integrating OpenTelemetry in Instana’s Dynamic Graph
Instana supports hundreds of technologies, and the rich telemetry it collects is correlated by means of semantic relationships that describe how the different entities in your infrastructure — like hosts, zones, containers, namespaces, clusters and all the rest — fit with one another. All this information is available at your fingertips, for example, via the Context Guide. The data powering the Dynamic Graph is collected by specific sensors in the Instana agent, and we meld this data with the data sent to Instana by OpenTelemetry in a way that is unique and novel. When receiving otlp data, the Instana agent will find out which process on the same host is sending the data, and send to the Instana backend data about the process, the container it runs in, etc. In this way, in Instana you will see traces associated with the infrastructure powering those applications, which is one of the killer features of AutoTrace and Unbounded Analytics.
Next steps
There are a few areas in which we definitely plan to further invest with regards to OpenTelemetry:
- Trace Semantic conventions: Similar to OpenTracing and Instana’s own AutoTrace tagging best practices, OpenTelemetry issues guidance on how to tag particular pieces of metadata on spans. We intend to create mapping inside Instana from those semantic conventions to AutoTrace’s. When compliant with the semantic conventions, your OpenTelemetry tracing data will look their best in Instana!
- Resources: Resources are a concept in OpenTelemetry akin to Instana’s own entities. You can specify key-value pairs describing different aspects of the application emitting telemetry, from host and process data to cloud provider attributes. There is a certain amount of overlap between the data exposed by OpenTelemetry over resources and what Instana’s Dynamic Graph entails, and we are interested in exploring how the two concepts can work and help one another.
- Metrics: OpenTelemetry not only provides APIs and implementations to collect trace data, but also metrics (and logs, see next point). Metrics is getting stable enough to start being adopted. While ecosystems like the Java Virtual Machines have widely adopted libraries to express metrics (micrometer.io and Dropwizard above all others, both supported natively by Instana), we think there is significant potential for adoption in other runtimes. Just as we support pushing OpenTelemetry traces from your app to the Instana agent, we intend to do the same with OpenTelemetry metrics, and offer you the same seamless, delightful experience that you already experience with Instana’s micrometer.io and Dropwizard integrations across all runtimes.
- Logs: The same thing we said about metrics may be applicable to logs, although the Logs part of OpenTelemetry is really far, far away from being ready for adoption. Also, while many runtimes do not have strong contenders in terms of which API to use for logs, every single app out there already has logging enabled with existing libraries. It remains to be seen how much traction the logging portion of OpenTelemetry can gain.
Conclusions
It has never been easier to realize the full potential of your investments in OSS. With Instana’s seamless support for OpenTelemetry, users now have the same level of visibility and correlation that they get with Insana’s AutoTrace technology. If you’re an Instana customer, you can see the power of Instana’s OpenTelemetry support, now in Tech Preview, simply by asking your Customer Success Manager for access. If you don’t already have an Instana instance, you can try Instana for free with a no strings attached free trial today. OpenTelemetry support will be coming to the trial and every Instana instance in the near future.
