Prometheus

Introduction
Configuration
- Static configuration
- Kubernetes configuration
Metrics collection
Alerting
- SNMP Notifier

Introduction

Collecting metrics data with Prometheus is becoming more popular. With Instana, it is easy to capture Prometheus metrics and correlate them using our extensive knowledge graph. A typical example is custom business metrics.

The Instana Prometheus sensor doesn't require a Prometheus server. The sensor captures metrics directly from the endpoints that are exposed by the monitored systems. For each Instana agent, specify which Prometheus endpoints you want to poll and which metrics should be collected from them using regular expressions.

Configuration

Static configuration

For the Prometheus sensor to be able to collect data, metric sources must be defined in the agent configuration file <agent_install_dir>/etc/instana/configuration.yaml. The sensor picks up any type of metric. With a regular expression, you can define which metrics should be captured for a specific metric endpoint. Prometheus sensor has basic authentication support, and it can be defined on the global level or per endpoint.

# Static Prometheus Configuration
com.instana.plugin.prometheus:
  username: '' # username used for all endpoints
  password: '' # password used for all endpoints
  customMetricSources:
  - url: 'http://localhost:8080/metrics' # endpoint with exposed metrics
    metricNameIncludeRegex: '^sample_app_request'
  - url: 'http://localhost:9100/prometheus' # endpoint with exposed metrics
    metricNameIncludeRegex: '^sample_app_request'
    username: '' # username used for specific endpoint
    password: '' # password used for specific endpoint

Regex in the example above configures a sensor to gather all metrics starting with sample_app_request e.g. sample_app_request_duration. In case the regex is not defined, the sensor collects all exposed metrics until the defined limit (600 metrics per metric type).

Kubernetes configuration

The Prometheus sensor detects if Kubernetes is being used and automatically gathers all available pods on the node the sensor is running on. The URL field in the configuration is still required, but instead of specifying the whole URL, the path part is sufficient, e.g. /metrics.

# Kubernetes Discovery Prometheus Configuration
com.instana.plugin.prometheus:
  customMetricSources:
  - url: '/metrics'
    metricNameIncludeRegex: '^sample_app_request' 
  - url: '/prometheus-metrics'
    metricNameIncludeRegex: '^sample_app_request'

Metrics collection

Prometheus metrics appear as "Prometheus Apps" associated with the host from where they are collected. You can query Prometheus custom metrics using the "Dynamic Focus", "Event & Alerts", and the "Grafana Plugin" with entity.type:prometheus.

The sensor collects all core metric types, up to 600 metrics per type:

Counters
Gauges
Histograms
Summaries
Untyped

Alerting

Instana provides a Prometheus Alertmanager Webhook alert channel to send HTTP POST requests with the payload format of the Prometheus Alertmanager Webhook described at Prometheus Alertmanager Webhook Receiver configuration.

To configure a Prometheus Alertmanager Webhook alert channel, go to "Settings → Team Settings → Events & Alerts → Alert Channels → Add Alert Channel" and click on "Prometheus Alertmanager Webhook".

Instana sends alerts through this alert channel as HTTP POST requests to the configured Prometheus Alertmanager Webhook Receiver, for example Alert Snitch or SNMP Notifier.

SNMP Notifier

The SNMP Notifier project relays Prometheus alerts as SNMP traps to any configured SNMP receiver.

To ensure proper interaction with Instana alerts, we provide a Docker image for SNMP Notifier with a custom alert mapping in our private Docker registry, see Run with Docker.

Alert Channel Configuration

Prometheus Alertmanager Webhook Receiver URL: http://{SNMP-Notifier-Host}:9464/alerts

SNMP Notifier Configuration

To configure the SNMP Notifier service, please specify the following as container environment variables:

SNMP_VERSION = SNMP version. "V2c" and "V3" are currently supported. (Default: "V2c")
SNMP_DESTINATION = SNMP trap server destination. (Default: "127.0.0.1:162")
SNMP_RETRIES = SNMP number of retries. (Default: "1")
SNMP_TRAP_DEFAULT_OID = Trap OID to send if none is found in the alert labels. (Default: "1.3.6.1.4.1.98789.0.1")
SNMP_COMMUNITY = SNMP community (V2c only). (Default: "public")
SNMP_AUTHENTICATION_ENABLED = Enable SNMP authentication (V3 only). (Default: "false")
SNMP_AUTHENTICATION_PROTOCOL = Protocol for password encryption (V3 only). MD5 and SHA are currently supported. (Default: MD5)
SNMP_AUTHENTICATION_USERNAME = SNMP authentication username (V3 only).
SNMP_AUTHENTICATION_PASSWORD = SNMP authentication password (V3 only).
SNMP_PRIVATE_ENABLED = Enable SNMP encryption (V3 only). (Default: "false")
SNMP_PRIVATE_PROTOCOL = Protocol for SNMP data transmission (V3 only). DES and AES are currently supported (Default: "DES").
SNMP_PRIVATE_PASSWORD =  SNMP private password (V3 only).
SNMP_SECURITY_ENGINE_ID = SNMP security engine ID (V3 only).
SNMP_CONTEXT_ENGINE_ID = SNMP context engine ID (V3 only).
SNMP_CONTEXT_NAME = SNMP context name (V3 only).

Run with Docker

sudo docker login https://containers.instana.io -u="_" -p="${AGENT_KEY}"

Start Docker container

sudo docker run -p 9464:9464/tcp --env SNMP_DESTINATION=127.0.0.1:162 --env GROUP_ID=1001 --env USER_ID=1002 containers.instana.io/instana/release/product/snmpnotifier:2.164.224-0

Run in Kubernetes

Create secret for instana docker registry

kubectl create secret docker-registry instana \
 --docker-server=https://containers.instana.io \
 --docker-username=_ \
 --docker-password=${AGENT_KEY}

Deployment and Service Descriptor

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: instana-snmp-notifier-deployment
  labels:
    instana.io/component: instana-snmp-notifier
spec:
  replicas: 1
  selector:
    matchLabels:
      app: instana-snmp-notifier
  template:
    metadata:
      labels:
        app: instana-snmp-notifier
    spec:
      containers:
        - env:
            - name: GROUP_ID
              value: '1001'
            - name: USER_ID
              value: '1002'
            - name: SNMP_DESTINATION
              value: 127.0.0.1:162
          image: containers.instana.io/instana/release/product/snmpnotifier:2.164.224-0
          name: instana-snmp-notifier
          ports:
            - containerPort: 9464
              name: web-endpoint
              protocol: TCP
          resources:
            limits:
              cpu: 500m
              memory: 512Mi
            requests:
              cpu: 500m
              memory: 256Mi
      imagePullSecrets:
        - name: instana
---
apiVersion: v1
kind: Service
metadata:
  name: instana-snmp-notifier-service
spec:
  type: ClusterIP
  ports:
    - name: web-endpoint
      port: 9464
      targetPort: 9464
      protocol: TCP
  selector:
    app: instana-snmp-notifier