Tutorial: Instrumenting a Custom Scheduler with the Instana Tracing SDK

Schedulers are tools to run jobs periodically. The most popular Java scheduler is quartz, which is supported out-of-the-box by the Instana.

However, in case you have a custom in-house scheduler that is not supported, you can use the Java Trace SDK to add tracing for your scheduler.

This tutorial shows how to do this.

Example Code

The example code for the custom-scheduler-example tutorial can be found on github.com. It implements a simple scheduler, that calls a Hello World service on http://localhost:8080/ every 4 seconds and prints the response to the console.

Source Not Monitored By Instana

If you run the example, you will see that the HTTP calls are coming from an unmonitored source.

unmonitored source

Why is the source not monitored?

The example code uses the Apache HTTP Client for the outgoing call. The Apache HTTP Client is supported by the Instana agent. So why do we not see the source of the request in Instana?

The reason is that Instana instruments outgoing calls only if they happen in an existing trace context. In the example, the outgoing call is triggered by the custom scheduler which is not instrumented. Therefore, the call is not instrumented because no existing trace context is found.

Enabling Tracing for the Custom Scheduler

In order to enable tracing for the custom scheduler, you first need to add the Java Trace SDK as a dependency. For Maven, it looks like this:

<dependency>
  <groupId>com.instana</groupId>
  <artifactId>instana-java-sdk</artifactId>
  <version>1.2.0</version>
</dependency>

In order to create a new trace context whenever the HttpClientJob is called, add a @Span annotation to the job's run() method:

@Span(type = Span.Type.ENTRY, value = "my-custom-scheduler")
public void run() {
  // ...
}

This annotation tells the Java agent to generate an ENTRY span whenever the run() method is called.

Third, add the following to the Instana agents configuration.yaml file to tell it that Java classes in package com.instana.sample.* should be scanned for the @Span annotation:

com.instana.plugin.javatrace:
  instrumentation:
    sdk:
      packages:
        - 'com.instana.sample'

Now, each call to HelloWorldJob.run() creates a trace context, and the outgoing HTTP request is performed within this trace context. You see this in Instana as follows:

complete trace

The source of the HTTP GET request is now traced, including all metadata gathered from the automatic instrumentation of the Apache HTTP Client:

outgoing call

The scheduler job itself is visualized as an empty call with an unmonitored source:

outgoing call

Adding Tags

As shown above, the scheduler job is visualized as an empty call without any annotations. You can call the SDK's SpanSupport.annotate() method to add custom metadata. Just call SpanSupport.annotate() at the beginning of the HttpClientJob.run() method:

Override
@Span(type = Span.Type.ENTRY, value = SPAN_NAME)
public void run() {
  SpanSupport.annotate(Span.Type.ENTRY, SPAN_NAME, "tags.batch.job", "hello world job");
  // ...

In setting tags.batch.job for the ENTRY span of the job, as specified in OpenTracing's semantic conventions, Instana now treats the span as if it were of type BATCH and knows that there is no missing source.

span with annotation

You can also search for the custom tag in the Unbounded Analytics view:

search for custom tag

Marking a Span as Erroneous

At some point, you might want to mark a batch job as erroneous. This can easily be done by adding the error tag, as specified in OpenTracing's semantic conventions.

SpanSupport.annotate(Span.Type.ENTRY, SPAN_NAME, "tags.error", "true");

Adding this makes your call be visualized as erroneous in Instana.

erroneous span

Summary and Outlook

In this tutorial, we created a new trace context for our custom scheduler, thus enabling the batch job to be recognized as a source for outgoing HTTP calls in Instana.

For a full reference on our SDK, please view the SDK's JavaDoc. For example, the @SpanParam and @SpanReturn annotations make it really easy to capture parameters or return values as additional information far spans, which might be more convenient than explicitly calling SpanSupport.annotate() in some scenarios.