Self-hosted on Kubernetes

The Instana self-hosted installation on K8s consists of a database backend, kubectl plugin, a K8s operator, and number of components and services, each one running as an individual container running in the Kubernetes cluster after deployment.

Supported Orchestrating Systems

We support the following K8s distributions to run the Instana backend platform on:

  • K8s 1.16+
  • Rancher 2.x

Overview Architecture

Running Instana on your own K8s cluster requires installing/updating/scaling more than 30 components and five different databases. This would go way beyond what a regular customer would be able to maintain without excessive training in both Instana operations and K8s. We therefore opted for the implementation of a so called operator. Operators are a standard pattern in K8s to allow the deployment/update/scaling/maintenance of complex software. The following paragraphs will illustrate how our operator is structured and what elements will be installed into K8s cluster.

The Instana Operator

The actual operator is a simple k8s deployment consisting of one container. After installation it will ensure the installation of the required CRDs and will start tracking changes to these. You won't interact with the operator directly, for that we have built a kubectl-plugin. The following explanation will describe internals you should never have to touch directly but will see when accessing the K8s cluster.

Operator

The operator is provided as a Docker image:

containers.instana.io/instana/release/selfhosted/operator:

This image and all its versions are availabale from our container registry containers.instana.io. The actual implementation follows the operator pattern and is based on the operator-sdk. After installation into the cluster it will take care of all changes to the aforementioned CRDs and create/update/delete cores and units. The operator is resilient against interruptions/restarts by relying on constant evaluation of the cluster state and persistent FSMs to manage long running processes.

CRDs

Custom Resource Definitions are extensions of the Kubernetes API. They allow to add new resources and to let these be managed by an operator. In our case we created two CRDs for the different aspects of Instana in K8s.

cores.instana.io

A core represents all components shared by an Instana installation. Each core has a set of associated databases which will be used by the core itself and all tenants with their respective tenant units created as members of the core.

The operator supports multiple cores in the same K8s cluster to allow more finegrained separation of resources.

units.instana.io

Units represent individual data pools in Instana. A unit could represent a department (sre/dev/qa/...), an area (development/staging/production/...), or any other logical grouping required by the customer. Data from one unit is not visible by any other unit.

Above the TUs is the tenant, allowing further grouping. The tenant only appears as a logical construct and allows to define certain common properties for all its TUs (e.g. authentication, RBAC, ...).

The operator supports the creation of an arbitrary number of TUs across an arbitrary number of namespaces for an arbitrary number of cores.

Namespaces

In the following paragraphs you will find a recommended layout of namespaces for running Instana on Kubernetes. The following paragraphs will be based on this image. Namespace Layout

Operator Namespace

The operator should get its own namespace where it will create/delete various configmaps during its lifetime. Theses configmaps represent the persistent state of state machines used to interact with Instana installations. The operator doesn't expose anything outside its namespace. All interactions happen indirectly via creating/updating/deleting the unit/core-CRs.

Core Namespace

A core namespace contains all the shared components.

The most important ones being:

  • acceptor: The acceptor is the main entry point for the Instana agent and receives raw TCP traffic
  • eum-acceptor: The End-User–Monitoring-acceptor receives HTTP traffic coming from the EUM-scripts injected into your webapps
  • serverless-acceptor: The serverless-acceptor receives HTTP traffic containing metrics/traces from your serverless applications
  • butler: This is the Instana IdP, handling all things security/athentication/authorization-related. It exposes the SignIn pages via HTTP

Unit Namespace

A unit namespace can contain several or a single tenant unit. Tenant units can be placed in arbitrary namespaces but each namespaces needs to have a dedicated ingress component which will be installed by the operator.

Multi-tenancy

Each Instana installation supports multiple tenants with each tenant consisting of multiple tenant units (TU). Let's best look at an example: Let's assume for a moment there are two departments, ecommerce and intranet. ecommerce has 3 environments, dev, preprod, prod. intranet has two environments, dev and prod. In this case you would create a tenant ecommerce with the TUs dev/preprod/prod and intranet with the TUs dev/prod. The units are separated and will receive data only for agents associated with them. Using TUs it is also possible to have distinct userbases so only devs will be able to access the dev tenant units. An important side note: The tenant is a concept that only exists in our data model, components are associated with either a unit or a core (see next chapters).