Prometheus

Introduction
Configuration
- Static configuration
- Kubernetes configuration
Metrics collection
Alerting
- SNMP Trap Notifier

Introduction

Collecting metrics data with Prometheus is becoming more popular. With Instana, it's easy to capture Prometheus metrics and correlate them using our extensive knowledge graph. A typical example is custom business metrics.

The Instana Prometheus sensor doesn't require a Prometheus server. The sensor captures metrics directly from the endpoints that are exposed by the monitored systems. For each Instana agent, specify which Prometheus endpoints you want to poll and which metrics should be collected from them using regular expressions.

Configuration

Static configuration

For the Prometheus sensor to be able to collect data, metric sources must be defined in the agent configuration file <agent_install_dir>/etc/instana/configuration.yaml. The sensor picks up any type of metric. With a regular expression, you can define which metrics should be captured for a specific metric endpoint.

# Static Prometheus Configuration
com.instana.plugin.prometheus:
  customMetricSources:
  - url: 'http://localhost:8080/metrics' # endpoint with exposed metrics
    metricNameIncludeRegex: '^sample_app_request'
  - url: 'http://localhost:9100/prometheus' # endpoint with exposed metrics
    metricNameIncludeRegex: '^sample_app_request'

Regex in the example above configures a sensor to gather all metrics starting with sample_app_request e.g. sample_app_request_duration. In case the regex is not defined, the sensor collects all exposed metrics until the defined limit (700 metrics per metric type).

Kubernetes configuration

The Prometheus sensor detects if Kubernetes is being used and automatically gathers all the available pods on the node the sensor is running on. The URL field in the configuration is still required, but instead of specifying the whole URL, the path part is sufficient, e.g. /metrics.

# Kubernetes Discovery Prometheus Configuration
com.instana.plugin.prometheus:
  customMetricSources:
  - url: '/metrics'
    metricNameIncludeRegex: '^sample_app_request' 
  - url: '/prometheus-metrics'
    metricNameIncludeRegex: '^sample_app_request'

Metrics collection

Prometheus metrics appear as "Prometheus Apps" associated with the host from where they are collected. You can query Prometheus custom metrics using the "Dynamic Focus", "Event & Alerts", and the "Grafana Plugin" with entity.type:prometheus.

The sensor collects all core metric types, up to 600 metrics per type:

Counters
Gauges
Histograms
Summaries

Alerting

To configure, head over to "Settings → Team Settings → Events & Alerts → Alert Channels → Add Alert Channel".

Prometheus WebHook URL format

The Prometheus WebHooks are send as an HTTP PUT to the configured Prometheus Alertmanager Webhook Receiver.

SNMP Trap Notifier

The SNMP notifier sends SNMP traps to any given SNMP receiver. To ensure proper interaction with instana alerts, we provide a SNMP notifier with a custom alert mapping as docker image in our private docker registry.

Alert Channel Configuration

Webhook Url: http://{SMPT-Notifier-Host}:9464/alerts

Prometheus Alertmanager Configuration

To configure the Notifier Service, please specify the following as container environment variables (see below).

  SNMP_VERSION = SNMP version. V2c and V3 are currently supported (Default: V2c)
  SNMP_DESTINATION = SNMP trap server destination. (Default: 127.0.0.1:162)
  SNMP_RETRIES = SNMP number of retries (Default: 1)
  SNMP_TRAP_DEFAULT_OID = Trap OID to send if none is found in the alert labels (Defualt: "1.3.6.1.4.1.98789.0.1")
  SNMP_COMMUNITY = SNMP community (V2c only). (Default: "public")
  SNMP_AUTHENTICATION_ENABLED = Enable SNMP authentication (V3 only).
  SNMP_AUTHENTICATION_PROTOCOL = Protocol for password encryption (V3 only). MD5 and SHA are currently supported. (Default: MD5)
  SNMP_AUTHENTICATION_USERNAME = SNMP authentication username (V3 only). Passing secrets to the command line is not recommanded, consider using the (Default: USERNAME)SNMP_NOTIFIER_AUTH_PASSWORD = SNMP authentication password (V3 only). (Default: PASSWORD)
  SNMP_PRIVATE_ENABLED = Enable SNMP encryption (V3 only).
  SNMP_PRIVATE_PROTOCOL = Protocol for SNMP data transmission (V3 only). DES and AES are currently supported (Default: DES).
  SNMP_PRIVATE_PASSWORD =  SNMP private password (V3 only). (Default: SECRET)
  SNMP_SECURITY_ENGINE_ID = SNMP security engine ID (V3 only)
  SNMP_CONTEXT_ENGINE_ID = SNMP context engine ID (V3 only)
  SNMP_CONTEXT_NAME = SNMP context name (V3 only)

Run as Docker

docker login https://containers.instana.io -u="_" -p="${AGENT_KEY}"

Start docker container

 sudo docker run -p 9464:9464/tcp --env SNMP_DESTINATION=127.0.0.1:162 --env GROUP_ID=1001 --env USER_ID=1002 containers.instana.io/instana/release/product/snmpnotifier:2.164.224-0

Run in Kubernetes

Create secret for instana docker registry

kubectl create secret docker-registry instana \
 --docker-server=https://containers.instana.io \
 --docker-username=_ \
 --docker-password=${AGENT_KEY}

Deployment and Service Descriptor

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: instana-snmp-notifier-deployment
  labels:
    instana.io/component: instana-snmp-notifier
spec:
  replicas: 1
  selector:
    matchLabels:
      app: instana-snmp-notifier
  template:
    metadata:
      labels:
        app: instana-snmp-notifier
    spec:
      containers:
        - env:
            - name: GROUP_ID
              value: '1001'
            - name: USER_ID
              value: '1002'
            - name: SNMP_DESTINATION
              value: 127.0.0.1:162
          image: containers.instana.io/instana/release/product/snmpnotifier:2.164.224-0
          name: instana-snmp-notifier
          ports:
            - containerPort: 9464
              name: web-endpoint
              protocol: TCP
          resources:
            limits:
              cpu: 500m
              memory: 512Mi
            requests:
              cpu: 500m
              memory: 256Mi
      imagePullSecrets:
        - name: instana
---
apiVersion: v1
kind: Service
metadata:
  name: instana-snmp-notifier-service
spec:
  type: ClusterIP
  ports:
    - name: web-endpoint
      port: 9464
      targetPort: 9464
      protocol: TCP
  selector:
    app: instana-snmp-notifier