Achieving Observability in Go

October 1, 2020

Post

Go is a statically typed, natively compiled and garbage-collected programming language. Originally designed by Google, Go is now developed as an open source project.

Given the natively compiled nature of Go, adding observability at runtime is not the preferred method. I’m not saying that it couldn’t be achieved, but as developers who develop native executables, we are super careful of other people fiddling around in our binaries at runtime.

Therefore, at Instana, we decided against instrumenting Go applications at runtime, opting for the least invasive way of providing “observability wrappers.”

Why Use Wrappers for Go Instrumentation?

There are three general options when adding Observability to Go(Lang):

  • Open Source Libraries
  • Wrappering Code
  • On-the-fly run-time instrumentation

Adding observability to your service can be achieved by using libraries such as OpenTracing or Zipkin. While it may sound easy, by the time you factor in a bit of time to add a trace here, a bit of time there and a “bit” more time every time your code changes, it quickly becomes anything but. In reality, manually instrumenting applications can increase your overall application costs by as much as 40% (sometimes even more). And that’s without even taking into account keeping in mind how you visualize and display traces and what metadata you need to collect where.

Manually adding tracing, metrics and health monitoring is tedious, time consuming, and exponentially more complicated and expensive with the amount of teams and services required to keep aligned to a standard. Even when putting a lot of time into it, if a problem arises, you cannot really be sure that one specific element or outgoing or incoming call was captured. Resulting in endless toil and false negatives. It is assembly-level observability, and distracts from delivering business value.

Wrappers bring in separation of concerns. While the code or library does what it is designed to do, the wrapper adds additional concerns such as logging, or in the case of observability, tracing, or metrics points.

In addition to that, wrappers can bring in new or additional features later, by just updating the observability library, without any code changes at all.

Minimally invasive, minimal code change, that’s the formula.

What to Wrap?

When implementing wrappers for existing libraries in the ecosystem, there is always the question of priority and order.

At Instana, we have an opinionated way of looking at the problem, and to come up with the order of implementations. Obviously, customer demand plays into the decision, but is not the top priority.

Connectivity: When we investigate distributed traces, the highest importance is end-to-end visibility with tracing throughout the whole system. Breaking traces because the tracing information is not forwarded is a deal breaker. Therefore, our #1 priority is always connectivity between services. Making sure the trace and span information are forwarded through all channels of communication, the so-called “trace continuity”. End users must have coverage of the most adopted HTTP standard libraries, GRPC, Kafka and so on.

Userbase: With an ecosystem heavily relying on open source libraries, the land of Go tends to have several different libraries doing the same thing. That being said, the community usually converges on using just one or two of them. To provide benefits to a large user base, Instana opts to support the most used libraries first. Libraries with smaller adoption may still be supported, if the need arises in our end users.

Persistence:Serving most requests, in the end, end up at one point or another interacting with some kind of database or other. So, providing insight into databases is extremely important to get the end-to-end visibility we’re all looking for. Complexity of database wrappers heavily depends on the necessary drivers though, and prioritizing their support relies again on the number of user requests.

Usability is the Key

While implementing a good number of observability wrappers, not only in Go but also in other programming languages, a few best practices emerged.

Before we get started with the best practices, we should define what good means in terms of wrappers:

Stability: As a user there is nothing more frustrating than updating the source code after a library update (especially when you update to get support for a new feature). Therefore, finding the right spot to wrap functionality is the key. Look for the position which is expected to stay stable for the longest time. Like the net/http interfaces in the stdlib. We chose to implement our custom Roundtripper which is designed to intercept http calls both ways (request and response).

Minimal changes: The next important element is requiring the least amount of code changes, if change is inevitable. As said above, people hate to change code after a library update, even if the change would provide new cool functionality. If there is no way around it though, keep the impact minimal. In the best case, it’s just changing the import path and providing an object which looks like the original one. If interfaces are being used, provide an alternative implementation.

Once per code base: If possible, and feasible, offer a solution which must be “activated” only once per code base. I want to stress the “feasible” angle, since in Go it is often not expected to change the standard behavior. Like the standard lib HTTP library. There is the http.DefaultClient, but do you really want to exchange that without the user noticing?

Reuse functionality behind the scenes: Whenever you can, reuse native concepts and APIs of the wrapped framework and the runtime. An excellent example here is the transport of the trace context across an application. To keep track of which trace is being recorded, information like the trace ID or span ID has to be forwarded into the wrapper call. For drop-in replacement and stability reasons, it is always recommended to reuse functionality already provided by the wrapper framework and the runtime. In Go it is common to pass around context objects which are not only used to implement operation timeouts or cancelation, but also hold context-specific attributes. If a method has a context.Context parameter, it’s a good idea to use it for propagating the trace context, since both are usually tied to the same scope.

Good, Better, Awesome Wrappers

Looking at wrapping functionality, there are multiple different implementation options. Sometimes we are forced to use a specific option, since it is the only possible one, but if given the chance to choose, another lesson learned is the order of which implementation to use, from worst to best.

Wrapper Functions: Wrapper functions should be the last resort, as it often requires rather invasive changes that are best avoided. Given the common best practice in Go to favor structs over interfaces, it can be a necessary evil:

func wrappedFunction(ctx context.Context, val string) error {
     return originalFunction(ctx, val)
}

Make use of type inference: When you have the need to build wrapper functions, favor wrapper functions (factory functions) over objects that resemble the original one. If an object is necessary though (like in a database connection pool) make sure the wrapper object looks exactly like the real one. That enables the user to exchange the import path, use the factory function, and move on. Minimal code change. Stability may be impacted though, if the library adds new fields or methods to the object at a later point in time.

type ResourceHandler struct {}
func (ResourceHandler) Handle() {
     // ...
}
type APIClient struct {}
func (o APIClient) ResourceHandler() ResourceHandler {
     // ...
}
type WrappedResourceHandler struct {
     ResourceHandler
}
func (h WrappedResourceHandler) Handle() {
     // extend the wrapped method, e.g. log the call
     log.Println("Handle() has been called!")
     // pass the call to the wrapped method
     h.ResourceHandler.Handle()
}
type APIClientWrapper struct {
     APIClient // proxy calls to the wrapped type by default
}
func (c APIClientWrapper) ResourceHandler() WrappedResourceHandler {
     // wrap and return the handler
     return WrappedResourceHandler{ResourceHandler: c.APIClient.Handler()}
}
// to use the wrapped ResourceHandler we need to replace
c := APIClient{}
// with the following
c := APIClientWrapper{APIClient: APIClient{}}
// and this code will still work, but now calling the WrappedResourceHandler.Handle()
c.ResourceHandler().Handle()

Implement interfaces: As mentioned before, if the library provides one or more interfaces, start adding the functionality using additional interface implementations. Many, if not most, production grade libraries use semantic versioning, therefore the interface should stay stable. Adding methods to the interface, however, may still break the user’s code base.

type Pool interface {
     Acquire() interface{}
     Release(element interface{})
}
type Wrapper struct {
     p Pool
}
func (w Wrapper) Acquire() interface{} {
     // Returning an element from the pool
     return w.p.acquire()
}
func (w Wrapper) Release(element interface{}) {
     // Returning the element to the pool
     w.p.release(element)
}
// Original use of the pool
o := pool.NewPool(...)
// With wrapper
o := Wrapper{pool.NewPool(...)}

Hooks / Tracing points: If a library provides hooks, callbacks or tracing points, use them. It’s always best to hand the responsibility of calling a logger / tracer at the correct point in time to the actual implementer of the library, in the assumption, they know best. Those “plugin interfaces” tend to be more stable then the normal “Public API” and are independent of potential changes in the library. For example, the Go standard library provides the net/http/httptrace.WithClientTrace() method that injects a set of hooks to be called while executing a request. In this case the only thing you need to do is to provide implementation which is as simple as:

func GetConn(hostPort string) {
     // collect connection parameters
}

As a request to all library developers, please add hooks for logging and traceability to your libraries.

Pitfalls and Don’ts

After all the lessons learned so far, there are three more things we’d like to mention.

No new dependencies: Make sure your library is self-contained, and avoids bringing in new, unwanted dependencies. If a service doesn’t use Kafka, why would a library that provides observability bring a dependency on it? Unfortunately, it seems to be a common pattern when looking into dependencies of libraries. If you implement a wrapper for a Kafka client, make this wrapper a separate module. Same goes for specific database drivers, or any other 3rd party dependency that is not strictly necessary for your task.

Go versions: Having written about semantic versioning before, Go also provides an upgrade guarantee for all versions of 1.x. Still, from time to time there may be changes on a level which is not covered by that guarantee, such as adding new types and methods to the standard library. With observability libraries you must sometimes dig deep, up to a level where changes can happen beyond the scope of major releases. Thankfully, Go also provides
Build Constraints. Use them and make sure you make changes to (specific) versions of Go.

Library versions: Handling different Go versions is “easy”. Unfortunately, we don’t have the same functionality for most libraries. If the implementation diverges too much, make sure to split your wrapper for that library. Working too hard to make every version work with just one package of your wrapper often leads to unmaintainable code.

Conclusion

Go is an opinionated language, so the Go way to a solution is sometimes slightly different to other languages.

After multiple iterations of the Go wrapper library, we learned a lot on how to provide high-quality, easy-to-use instrumentation for Go. Always keep the code as intact as possible. If there are interfaces or hooks, use them. Only fall back to the less convenient wrappers types, such as wrapper functions or wrapper objects, if there is no way around it.

Keep in mind, the less code changes the user needs to do to their code, the better the experience. It is also much easier to add additional elements and features later on.

Play with Instana’s APM Observability Sandbox

Announcement, Product
What are Service Level Objectives (SLO) SLOs are important pieces that are used to define Service Level Agreements (SLAs). As described by Wikipedia, “SLOs are specific measurable characteristics of the SLA such...
|
Featured
When it comes right down to it, there are many different types of Docker monitoring but most people don’t realize how many layers of monitoring are required to understand the performance and...
|
Featured
IBM MQ (previously Websphere MQ and MQ Series) is an enterprise-oriented messaging middleware. According to the IBM website, “It works with a broad range of computing platforms, applications, web services and communications...
|

Start your FREE TRIAL today!

As the leading provider of Automatic Application Performance Monitoring (APM) solutions for microservices, Instana has developed the automatic monitoring and AI-based analysis DevOps needs to manage the performance of modern applications. Instana is the only APM solution that automatically discovers, maps and visualizes microservice applications without continuous additional engineering. Customers using Instana achieve operational excellence and deliver better software faster. Visit https://www.instana.com to learn more.