Prometheus

Introduction
Configuration
- Scraping
  - Kubernetes configuration
  - Static configuration
- Remote Write
  - Infrastructure correlation
  - TLS Encryption for Remote Write
Metrics collection
Alerting
- SNMP Notifier
Troubleshooting
- Remote Write High Metric Delay

Introduction

Collecting metrics data with Prometheus is becoming more popular. With Instana, it is easy to capture Prometheus metrics and correlate them using our extensive knowledge graph. A typical example is custom business metrics.

The Instana Prometheus sensor doesn't require a Prometheus server. The sensor captures metrics directly from the endpoints that are exposed by the monitored systems. For each Instana agent, specify which Prometheus endpoints you want to poll and which metrics should be collected from them using regular expressions.

Note that if your metrics are exposed by a Java library (like Micrometer, or the Prometheus Java client library, or jmx_exporter), there is no need to expose them via HTTP because our Java plugin can collect metrics from these libraries directly:

Micrometer shows how to collect Micrometer metrics.
Prometheus-java-client shows how to collect metrics from the Prometheus Java client library or from Micrometer's Prometheus Meter Registry.
Jmx shows how to collect custom metrics from JMX beans.

Configuration

Scraping

Kubernetes configuration

The metric sources need to be defined in the agent configuration file <agent_install_dir>/etc/instana/configuration.yaml as a list of endpoints. The agent automatically recognizes when it is running in a Kubernetes environment, and gathers IPs and container ports from running pods. For Kubernetes environments, the url field should not contain a host and port information, only the metrics endpoint URL for example /prometheus/endpoint/metrics.

With a regular expression, you can define which metrics should be captured for a specific metric endpoint in the metricNameIncludeRegex. Prometheus sensor has basic authentication support that can be defined on the global level (all endpoints) or per an endpoint.

com.instana.plugin.prometheus:
  # Global (all) endpoints username/password configuration (optional)
  username: ''
  password: ''
  customMetricSources:
  - url: '/prometheus/endpoint/1'            # metrics endpoint, the IP and port are auto-discovered
    metricNameIncludeRegex: '.*'             # regular expression to filter metrics
    username: ''                             # endpoint specific username/password configuration
    password: ''
  - url: '/prometheus/endpoint/2'
    metricNameIncludeRegex: '.*'

Note: the sensor is not going to monitor endpoints specified as full URLs, eg. https://golang-myapp.server.com/prometheus/metrics.

Static configuration

The static configuration is meant to be used for non-Kubernetes environments. If you want to capture Prometheus metrics from local or remote endpoints, you can specify them using the customMetricSources configuration list.

com.instana.plugin.prometheus:
  # Global (all) endpoints username/password configuration (optional)
  username: ''
  password: ''
  customMetricSources:
  - url: 'http://localhost:8080/metrics'
    username: ''      # endpoint specific username/password configuration
    password: ''
    metricNameIncludeRegex: '^sample_app_request'
  - url: 'http://223.58.1.10:9100/prometheus'
    metricNameIncludeRegex: '^sample_app_request'

In case the metricNameIncludeRegex is not defined, the sensor collects all metric types up to the defined limit of 600 metrics per metric type.

Remote Write

Starting with Agent bundle 1.1.587, the Instana Agent includes support for the remote_write endpoint, meaning it is able to ingest metrics and these will be displayed as either a Prometheus Entity or part of the Process Custom Metrics.

To enable the remote_write endpoint, add the following to the agent's configuration.yaml:

com.instana.plugin.prometheus:
  remote_write:
    enabled: true

Consult the documentation for Prometheus' remote_write on how to set up the sender (the component that sends the metrics). The remote_write endpoint is available on port 42699 at the /prometheus/v1/receive path. Therefore, the URL you will need to configure in the prometheus configuration is going to be http://<agent_ip>:42699/prometheus/v1/receive.

Note: The optional Instana Agent Service provided on Kubernetes via the Instana Agent Helm Chart is very useful in combination with the remote_write API, as it ensures that the data is pushed to the Instana Agent running on the same Kubernetes node, ensuring the Instana Agent can correctly fill in the infrastructure correlation data.

Note: Currently authentication is not supported for the remote_write endpoint, so the basic_auth and bearer_token (including bearer_token_file) options of Prometheus should not be configured in the sender.

IMPORTANT: In order for Instana to be able to parse metrics correctly, the sender must send the metadata. This is the default in Prometheus, so be sure not to turn it off.

Infrastructure correlation

For detailed information on infrastructure correlation on Linux hosts, please review the documentation on the agent's HTTP API endpoint.

TLS Encryption for Remote Write

TLS encryption can be enabled on the Agent, so that all data send to the remote_write endpoint is TLS encrypted as well. See Enabling TLS Encryption for more information on how to set this up.

Metrics collection

Prometheus metrics appear as "Prometheus Apps" associated with the host or with the process from where they are collected (in case of using the remote_write endpoint). You can query Prometheus custom metrics using the "Dynamic Focus", "Event & Alerts", and the "Grafana Plugin" with entity.type:prometheus.

The sensor collects all core metric types, up to 600 metrics per type:

Counters
Gauges
Histograms
Summaries
Untyped

Alerting

Instana provides a Prometheus Alertmanager Webhook alert channel to send HTTP POST requests with the payload format of the Prometheus Alertmanager Webhook described at Prometheus Alertmanager Webhook Receiver configuration.

To configure a Prometheus Alertmanager Webhook alert channel, go to "Settings → Team Settings → Events & Alerts → Alert Channels → Add Alert Channel" and click on "Prometheus Alertmanager Webhook".

Instana sends alerts through this alert channel as HTTP POST requests to the configured Prometheus Alertmanager Webhook Receiver, for example Alert Snitch or SNMP Notifier.

SNMP Notifier

The SNMP Notifier project relays Prometheus alerts as SNMP traps to any configured SNMP receiver.

To ensure proper interaction with Instana alerts, we provide a Docker image for SNMP Notifier with a custom alert mapping in our private Docker registry, see Run with Docker.

Alert Channel Configuration

Prometheus Alertmanager Webhook Receiver URL: http://{SNMP-Notifier-Host}:9464/alerts

SNMP Notifier Configuration

To configure the SNMP Notifier service, please specify the following as container environment variables:

SNMP_VERSION = SNMP version. "V2c" and "V3" are currently supported. (Default: "V2c")
SNMP_DESTINATION = SNMP trap server destination. (Default: "127.0.0.1:162")
SNMP_RETRIES = SNMP number of retries. (Default: "1")
SNMP_TRAP_DEFAULT_OID = Trap OID to send if none is found in the alert labels. (Default: "1.3.6.1.4.1.98789.0.1")
SNMP_COMMUNITY = SNMP community (V2c only). (Default: "public")
SNMP_AUTHENTICATION_ENABLED = Enable SNMP authentication (V3 only). (Default: "false")
SNMP_AUTHENTICATION_PROTOCOL = Protocol for password encryption (V3 only). MD5 and SHA are currently supported. (Default: MD5)
SNMP_AUTHENTICATION_USERNAME = SNMP authentication username (V3 only).
SNMP_AUTHENTICATION_PASSWORD = SNMP authentication password (V3 only).
SNMP_PRIVATE_ENABLED = Enable SNMP encryption (V3 only). (Default: "false")
SNMP_PRIVATE_PROTOCOL = Protocol for SNMP data transmission (V3 only). DES and AES are currently supported (Default: "DES").
SNMP_PRIVATE_PASSWORD =  SNMP private password (V3 only).
SNMP_SECURITY_ENGINE_ID = SNMP security engine ID (V3 only).
SNMP_CONTEXT_ENGINE_ID = SNMP context engine ID (V3 only).
SNMP_CONTEXT_NAME = SNMP context name (V3 only).

Run with Docker

sudo docker login https://containers.instana.io -u="_" -p="${AGENT_KEY}"

Start Docker container

sudo docker run -p 9464:9464/tcp --env SNMP_DESTINATION=127.0.0.1:162 --env GROUP_ID=1001 --env USER_ID=1002 containers.instana.io/instana/release/product/snmpnotifier:2.164.224-0

Run in Kubernetes

Create secret for instana docker registry

kubectl create secret docker-registry instana \
 --docker-server=https://containers.instana.io \
 --docker-username=_ \
 --docker-password=${AGENT_KEY}

Deployment and Service Descriptor

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: instana-snmp-notifier-deployment
  labels:
    instana.io/component: instana-snmp-notifier
spec:
  replicas: 1
  selector:
    matchLabels:
      app: instana-snmp-notifier
  template:
    metadata:
      labels:
        app: instana-snmp-notifier
    spec:
      containers:
        - env:
            - name: GROUP_ID
              value: '1001'
            - name: USER_ID
              value: '1002'
            - name: SNMP_DESTINATION
              value: 127.0.0.1:162
          image: containers.instana.io/instana/release/product/snmpnotifier:2.164.224-0
          name: instana-snmp-notifier
          ports:
            - containerPort: 9464
              name: web-endpoint
              protocol: TCP
          resources:
            limits:
              cpu: 500m
              memory: 512Mi
            requests:
              cpu: 500m
              memory: 256Mi
      imagePullSecrets:
        - name: instana
---
apiVersion: v1
kind: Service
metadata:
  name: instana-snmp-notifier-service
spec:
  type: ClusterIP
  ports:
    - name: web-endpoint
      port: 9464
      targetPort: 9464
      protocol: TCP
  selector:
    app: instana-snmp-notifier

Troubleshooting

Remote Write High Metric Delay

Monitoring issue type: prometheus_remote_write_high_delay

Prometheus metrics that are being ingested from the remote_write endpoint, are being received with a high delay. This results in potentially delayed alerting and makes correlating metrics from other sources more difficult.

Please look at tuning the Prometheus remote_write configuration. Specifically adding the batch_send_deadline parameter as defined below, which limits the delay to 1 second:

remote_write:
  - url: "http://xxx.xxx.xxx.xxx:42699/prometheus/v1/receive"
    queue_config:
      batch_send_deadline: 1s

Full configuration options can be found in the Prometheus manual.