Part 1: What is a Kubernetes Operator?
We recently published the first alpha version of our Instana Agent Operator, built with Quarkus and the upcoming Quarkus Kubernetes Client Extension. In this Blog series, we’d like to share our experience and show how these technologies can be used to implement a Kubernetes operator in Java.
The series consists of three parts:
- Part 1: What is a Kubernetes Operator gives a general overview of what an operator is.
- Part 2: Getting Started with the Quarkus Kubernetes Client Extension shows how to set up the a Quarkus application that interacts with the Kubernetes API server.
- Part 3: Writing a Kubernetes Operator in Java shows how to implement typical operator functionality, like watching a custom resource.
This is part 1, the introduction to Kubernetes operators.
What is a Kubernetes Controller?
Before going into operators, we should have a look at Kubernetes controllers. Controllers are one of the basic building blocks of Kubernetes. There are few dozen built-in controllers, all of which run on the Kubernetes master in a Pod named
kube-controller-manager in the
One of these built-in controllers is the replica set controller. Its job is to take care of replica sets. The following is a typical example of a replica set resource:
apiVersion: apps/v1 kind: ReplicaSet metadata: name: kuard spec: replicas: 3 selector: matchLabels: name: kuard template: metadata: labels: name: kuard spec: containers: - name: kuard image: gcr.io/kuar-demo/kuard-amd64:1
If this looks unfamiliar, it’s because you usually do not create replica sets explicitly. You create a
Deployment, and the built-in deployment controller creates a
ReplicaSet for you. Deployments do more than just creating replica sets. For example, deployments implement rolling updates. Replica sets have a single purpose: They specify the number of
replicas, i.e. the desired number of Pods running.
The replica set controller makes sure that the number of Pods matches the number of
replicas defined in the replica set. If a
Pod is deleted, the replica set controller starts a new one. If the replica set is modified, the replica set controller adds or deletes Pods to get back to the desired number of replicas.
What is an Operator?
The replica set controller example above illustrates the most important features implemented by controllers:
- Watching the primary resource, which is the
ReplicaSetin this example.
- Watching the secondary resource, which is the
Podin this example.
- Interacting with the API server: Whenever a
ReplicaSetis created or updated, or whenever a
Podis created or deleted, the replica set controller talks to the API server and creates or deletes Pods until the desired state is restored.
These concepts are implemented in a generic, extensible way in Kubernetes. You can create your own custom resources, and write your own custom controllers watching these resources and interacting with the API server.
An operator is a custom controller with a focus on operating a specific application.
A good example of such an application is a distributed key/value store. Kubernetes provides deployments (or replica sets) to scale up the number of Pods. However, for a particular key/value store additional steps might be necessary when Pods are added, like re-balancing the data. In order to implement this, you might create your own custom resource (similar to
ReplicaSet, but with application-specific configuration) and you may implement your own controller (like the replica set controller, but taking care of re-balancing when Pods are added or deleted). This custom controller would be the operator for the key/value store.
Development Tools for Creating Operators
From a high level perspective, operators are applications talking to the Kubernetes API server via HTTPS. They can be written in any programming language, and the can be deployed inside the cluster or outside the cluster. However, it’s most common and most convenient to bundle them into a Docker image and run them inside the cluster.
As operators implement functionality similar to built-in controllers, it is a natural choice to re-use the same libraries that Kubernetes uses internally. This is the origin of the CoreOS operator SDK. It provides tools to generate the necessary boilerplate code, and uses Kubernetes internal libraries as dependencies. As Kubernetes is written in Go, operators are written in Go as well.
For Java developers, Quarkus and the upcoming Kubernetes client extension provides a promising alternative to the Go ecosystem. Java applications built with Quarkus can be compiled into native executables similar to Go binaries. This allows for small Docker images with fast startup times. Moreover, the upcoming Kubernetes client extension implements much of the functionality needed to watch resources and to interact with the API server.
The next Blog post in this series will show how to set up a Quarkus project with the Kubernetes client extension. Part 3 will show how to implement typical controller functionality, like watching for updates of custom resources.
As a side note, I would like to introduce owner references and garbage collection. In many cases, Kubernetes’ built-in garbage collection is sufficient to clean up resources created by operators, so operators don’t need to implement explicit clean-up.
In Kubernetes, every resource can be annotated with an owner reference. For example, when the replica set controller creates a
Pod, it sets the
ReplicaSet as the owner reference for that
ownerReferences: - apiVersion: apps/v1 blockOwnerDeletion: true controller: true kind: ReplicaSet name: kuard uid: aa8ecdff-9989-11e9-9cc6-0242806076bd
When the replica set is deleted, the controller does not need to do anything. Kubernetes will implicitly delete all resources having the replica set as the owner reference, thus terminating the Pods associated with that replica set.
Operators should adapt this mechanism and use the custom resource as owner reference for all generated resources. That way, whenever the user deletes a custom resource, Kubernetes will take care to remove all associated resources. This works even if the operator is no longer running.
This blog post introduced operators. An operator is a custom controller with a focus on operating a specific application.