OTEL + Real-Time Observability: The New OTEL paradigm

March 29, 2023

Developer, Engineering, Product, Thought Leadership

Prometheus, Jaeger, Zipkin, and OTLP are some of the most popular Open Telemetry (OTEL) platforms in use. Instana has integrated those OTEL technologies (full integrations list) into the Instana Real-Time Observability fully and without limitations. In fact, Instana has been supporting OTEL for over 5 years, which was described in a 2018 blog post: https://www.instana.com/blog/integrating-jaeger-and-zipkin-tracers-into-instana/.

What that means is you can have the vendor agnostic OTEL telemetry you want combined with the significant value-added services of Instana’s Real-Time Observability platform. The diagram below illustrates the integration and shows that the value-added path where OTEL telemetry can either be fed into the OTEL Collector or into the Instana Agent, just as the Instana native sensor telemetry.

The second diagram below illustrates the native Instana telemetry path, from which you can see that the similarities in the implementation. The significant value added is the Instana Real-Time Stream Processing technology, which enables the industry-leading high precision performance information delivered by Instana.

Telemetry and traces are the primary measurements gathered from applications and infrastructure and are the inputs to Observability platforms to show the health of applications. Instana gathers both open telemetry metrics and traces as well as its own telemetry and traces and integrates them with Instana’s intrinsic AI-driven Smart Alerts, real-time Dynamic Graph, analytics, and other capabilities.

What is OTEL + Real-Time Observability?

The additional value-added feature Instana provides is the integration of OTEL telemetry together with the Instana agent to provide high speed telemetry ingestion and high-capacity storage. The OTEL and Real Time Observability diagram shows how that information flow occurs, from gathering the telemetry from the applications and services to feeding it into either the Instana Agent or the OTEL Collector and then feeding it into the Stream Processing flow.

The Stream Processing technology processes and stores the telemetry, traces, and context for your applications and services. It’s there that AI-driven Smart Alerts, real-time Dynamic Graph, analytics, and other capabilities are added.

Feeding the OTEL telemetry into the Instana Agent is like adding a turbo- or supercharger to your vehicle. It makes the visibility of issues and incidents occur faster, so that you may initiate automated or remedial actions faster.

The Open Telemetry Collector which is a vendor-agnostic implementation of how to receive, process and export telemetry data, but it’s also often the limiting factor in how rapidly the telemetry can be collected and provides no accommodation or method to store the telemetry for later use.

The OTEL Collector leaves it to the user to implement a OTEL telemetry storage which frequently reduces the amount of metrics and traces that can be stored. There are also no widely accepted OTEL analytics platforms, which leaves a huge gap for using the OTEL information for application issue triage.

How this looks when used for an application is shown in the OTel-Shop screenshots above and below, which are from the Instana OTEL-Shop dashboards. They show the stack correlation and Upstream/Downstream services correlation dropdowns within the dependency map. These provide immediate visibility and navigation capability through the services that comprise a transaction to pinpoint application issues.

What OTEL + Real Time Observability Provides

The ultimate goal of your Observability platform, whether it uses vendor sensors or OTEL sensors is to provide you with the information you need to rapidly identify, prevent, or remediate issues and incidents. To do that requires advanced contextual correlation for finding anomalies. Correlating that kind of information is not vendor-specific, but the quality of the correlated information depends on the granularity and quality of the data of the platform implementation. That’s where Instana provides value to OTEL.

This integration gives you the best of both worlds – extensible, “vendor-agnostic” Open Telemetry and end-to-end transaction traces combined with Instana’s very advanced alerting, analytics, and remediation capabilities.

Instana provides the most precise and comprehensive telemetry and trace analytics without comprise. That’s because it stores one second metrics for 24 hours and collects full end to end traces without sampling. This holds true whether the information is gathered from Instana or OTEL sensors.

Why is that important? Because problem discovery time matters, especially for highly distributed cloud-native microservices applications. If you use an OTEL platform that samples information, it will leave visibility gaps within your application.

Most Observability platforms are using AI to compensate for those gaps, but even advanced AI can’t completely compensate for missing information and generally has to wait for additional trace information to authoritatively provide accurate analytics. With Instana, precise and complete information IN = precise and accurate analytics OUT.

OTEL integration with Instana’s powerful Dependency map, Upstream/Downstream service correlation, and Stack visibility provide the most comprehensive set of OTEL value-added capabilities available. It does so without comprising OTEL vendor-agnostic freedom and provides you with the complete Observability platform you need to effectively use OTEL with your applications.

Sign up for a 14 day Free Trial here