Tutorial: Instrumenting a Custom Java HTTP Framework with the Instana Tracing SDK

The Instana agent supports many Java HTTP frameworks, and in most cases tracing will be supported out-of-the-box as soon as you run the Java agent.

However, in case you have a custom in-house HTTP server that is not supported, you can use the Java Trace SDK to add tracing for your framework.

This tutorial shows how to do this.

Example Code

The example code for the custom-http-sample tutorial can be found on github.com. It implements the following scenario:

custom http server tutorial

If you run the example and open http://localhost:8080, you will get the response Hello, Stan!. However, if you analyze the calls in Instana, you will see that the trace is broken.

Trace 1: tracing screenshot 01

Trace 2: tracing screenshot 02

The outgoing requests are implemented using the Apache HTTP client, which is supported by Instana's automatic tracing. The missing link is the entry span (incoming call) GET /greeting at the custom HTTP server.

Enable Tracing for the Custom HTTP Server

For supported frameworks, the Instana agent instruments Java applications automatically. For the custom HTTP server, we need to add some annotations to tell the Instana agent which method calls should start a new span.

First, add mvn dependency

<dependency>
  <groupId>com.instana</groupId>
  <artifactId>instana-java-sdk</artifactId>
  <version>1.2.0</version>
</dependency>

Second, add a @Span annotation to the method handling the HTTP request in the custom HTTP server.

@Span(type = Span.Type.ENTRY, value = "my-custom-http-server")
public String apply(Map<String, String> headers) {
  // ...
}

This annotation tells the Java agent to generate an ENTRY span (aka. server span) whenever this method is called.

Third, add the following to the Instana agent's configuration.yaml file to tell it that Java classes in package com.instana.sample.* should be scanned for the @Span annotation:

com.instana.plugin.javatrace:
  instrumentation:
    sdk:
      packages:
        - 'com.instana.sample'

Now, the custom server gets instrumented, and you see that the outgoing call to /name is triggered from the incoming call to /greeting.

tracing screenshot 03

However, the trace is still broken, because the trace ID is not passed on between the incoming call and the outgoing call.

Handling Incoming Tracing Headers

Instana uses the following HTTP headers to pass tracing information from one service to the next:

  • X-INSTANA-T: Trace ID
  • X-INSTANA-S: Parent span ID
  • X-INSTANA-L: Level 0 means the trace should be suppressed, which might be useful for health checks

In order to correlate the trace context in the customer HTTP server, you need to add the following method to GreetingHandler:

private void correlateTracing(Map<String, String> request) {
  String level = request.remove(SpanSupport.LEVEL);
  String traceId = request.remove(SpanSupport.TRACE_ID);
  String parentSpanId = request.remove(SpanSupport.SPAN_ID);

  if (SpanSupport.SUPPRESS.equals(level)) {
    SpanSupport.suppressNext();
  } else if (traceId != null && parentSpanId != null) {
    SpanSupport.inheritNext(traceId, parentSpanId);
  }
}

Note that we remove the headers after evaluating them, leaving only the original headers as they would be sent without Instana tracing.

However, you cannot just call correlateTracing() in apply(), because the headers need to be evaluated before the method with the @Span annotation is called. As a workaround, you could rename the apply() method to doApply() and add a step as follows:

@Override
public String apply(Map<String, String> headers) {
  correlateTracing(headers);
  return doApply(headers);
}


@Span(type = Span.Type.ENTRY, value = "my-custom-http-server")
private String doApply(Map<String, String> headers) {
  // ...
}

When you now run curl http://localhost:8080, you will see the complete Trace in Instana

tracing screenshot 04

Adding Tags

So far, our my-custom-http-server span is just a generic ENTRY span without any additional annotations. If you look at the details, they look quite empty.

span without annotations

You can call the SDK's SpanSupport.annotate() method within an active @Span context to add annotations to the current span. Instana supports part of OpenTracing's semantic conventions as described in the Java Trace SDK documentation. In order to add HTTP annotations, add the following lines at the top of the doApply() method:

@Span(type = Span.Type.ENTRY, value = SPAN_NAME)
private String doApply(Map<String, String> headers) {
  SpanSupport.annotate(Span.Type.ENTRY, SPAN_NAME,"tags.http.url", "http://localhost:8080/greeting");
  SpanSupport.annotate(Span.Type.ENTRY, SPAN_NAME,"tags.http.method", "GET");
  SpanSupport.annotate(Span.Type.ENTRY, SPAN_NAME,"tags.http.status_code", "200");
  // ...
}

Now the details of the ENTRY span for the custom HTTP server show the specified HTTP metadata.

span with annotations

Note that the corresponding EXIT span from Jetty to the custom HTTP server had HTTP metadata already, because the client uses the Apache HTTP client which is supported by Instana's auto tracing.

Marking a Span as Erroneous

At some point, you might want to mark a call as erroneous. This can easily be done by adding the error tag, as specified in OpenTracing's semantic conventions.

SpanSupport.annotate(Span.Type.ENTRY, SPAN_NAME,"tags.error", "true");

Adding this take, makes your call be visualized as erroneous in Instana.

erroneous span

Summary and Outlook

This tutorial showed the most important API needed to instrument a custom HTTP framework with Instana's distributed tracing.

For a full reference on our SDK, please view the SDK's JavaDoc. For example, the @SpanParam and @SpanReturn annotations make it really easy to capture parameters or return values as additional information far spans, which might be more convenient than explicitly calling SpanSupport.annotate() in some scenarios.