Prometheus
TABLE OF CONTENTS
Introduction
Collecting metrics data with Prometheus is becoming more popular. With Instana, it is easy to capture Prometheus metrics and correlate them using our extensive knowledge graph. A typical example is custom business metrics.
The Instana Prometheus sensor doesn't require a Prometheus server. The sensor captures metrics directly from the endpoints that are exposed by the monitored systems. For each Instana agent, specify which Prometheus endpoints you want to poll and which metrics should be collected from them using regular expressions.
Configuration
Scraping
Kubernetes configuration
The metric sources need to be defined in the agent configuration file <agent_install_dir>/etc/instana/configuration.yaml
as a list of endpoints.
The agent automatically recognizes when it is running in a Kubernetes environment, and gathers IPs and container ports from running pods.
For Kubernetes environments, the url
field should not contain a host and port information, only the metrics endpoint URL for example /prometheus/endpoint/metrics
.
With a regular expression, you can define which metrics should be captured for a specific metric endpoint in the metricNameIncludeRegex
.
Prometheus sensor has basic authentication support that can be defined on the global level (all endpoints) or per an endpoint.
com.instana.plugin.prometheus:
# Global (all) endpoints username/password configuration (optional)
username: ''
password: ''
customMetricSources:
- url: '/prometheus/endpoint/1' # metrics endpoint, the IP and port are auto-discovered
metricNameIncludeRegex: '.*' # regular expression to filter metrics
username: '' # endpoint specific username/password configuration
password: ''
- url: '/prometheus/endpoint/2'
metricNameIncludeRegex: '.*'
Note: the sensor is not going to monitor endpoints specified as full URLs, eg. https://golang-myapp.server.com/prometheus/metrics.
Static configuration
The static configuration is meant to be used for non-Kubernetes environments. If you want to capture Prometheus metrics from local or remote endpoints, you can specify them using the customMetricSources
configuration list.
com.instana.plugin.prometheus:
# Global (all) endpoints username/password configuration (optional)
username: ''
password: ''
customMetricSources:
- url: 'http://localhost:8080/metrics'
username: '' # endpoint specific username/password configuration
password: ''
metricNameIncludeRegex: '^sample_app_request'
- url: 'http://223.58.1.10:9100/prometheus'
metricNameIncludeRegex: '^sample_app_request'
In case the metricNameIncludeRegex
is not defined, the sensor collects all metric types
up to the defined limit of 600 metrics per metric type.
Remote Write
Starting with Agent bundle 1.1.587, the Instana Agent includes support for the remote_write
endpoint, meaning it is able to ingest metrics and these will be displayed as either a Prometheus Entity or part of the Process Custom Metrics.
To enable the remote_write
endpoint, add the following to the agent's configuration.yaml
:
com.instana.plugin.prometheus:
remote_write:
enabled: true
Consult the documentation for Prometheus' remote_write on how to set up the sender (the component that sends the metrics).
The remote_write
endpoint is available on port 42699
at the /prometheus/v1/receive
path.
Therefore, the URL you will need to configure in the prometheus configuration is going to be http://<agent_ip>:42699/prometheus/v1/receive
.
Note: The optional Instana Agent Service provided on Kubernetes via the Instana Agent Helm Chart is very useful in combination with the remote_write
API, as it ensures that the data is pushed to the Instana Agent running on the same Kubernetes node, ensuring the Instana Agent can correctly fill in the infrastructure correlation data.
Note: Currently authentication is not supported for the remote_write
endpoint, so the basic_auth
and bearer_token
(including bearer_token_file
) options of Prometheus should not be configured in the sender.
IMPORTANT: In order for Instana to be able to parse metrics correctly, the sender must send the metadata. This is the default in Prometheus, so be sure not to turn it off.
TLS Encryption for Remote Write
TLS encryption can be enabled on the Agent, so that all data send to the remote_write
endpoint is TLS encrypted as well.
See Enabling TLS Encryption for more information on how to set this up.
Metrics collection
Prometheus metrics appear as "Prometheus Apps" associated with the host or with the process from where they are collected (in case of using the remote_write
endpoint).
You can query Prometheus custom metrics using the "Dynamic Focus", "Event & Alerts", and the "Grafana Plugin" with entity.type:prometheus
.
The sensor collects all core metric types, up to 600 metrics per type:
- Counters
- Gauges
- Histograms
- Summaries
- Untyped
Alerting
Instana provides a Prometheus Alertmanager Webhook alert channel to send HTTP POST requests with the payload format of the Prometheus Alertmanager Webhook described at Prometheus Alertmanager Webhook Receiver configuration.
To configure a Prometheus Alertmanager Webhook alert channel, go to "Settings → Team Settings → Events & Alerts → Alert Channels → Add Alert Channel" and click on "Prometheus Alertmanager Webhook".
Instana sends alerts through this alert channel as HTTP POST requests to the configured Prometheus Alertmanager Webhook Receiver, for example Alert Snitch or SNMP Notifier.
SNMP Notifier
The SNMP Notifier project relays Prometheus alerts as SNMP traps to any configured SNMP receiver.
To ensure proper interaction with Instana alerts, we provide a Docker image for SNMP Notifier with a custom alert mapping in our private Docker registry, see Run with Docker.
Alert Channel Configuration
Prometheus Alertmanager Webhook Receiver URL: http://{SNMP-Notifier-Host}:9464/alerts
SNMP Notifier Configuration
To configure the SNMP Notifier service, please specify the following as container environment variables:
SNMP_VERSION = SNMP version. "V2c" and "V3" are currently supported. (Default: "V2c")
SNMP_DESTINATION = SNMP trap server destination. (Default: "127.0.0.1:162")
SNMP_RETRIES = SNMP number of retries. (Default: "1")
SNMP_TRAP_DEFAULT_OID = Trap OID to send if none is found in the alert labels. (Default: "1.3.6.1.4.1.98789.0.1")
SNMP_COMMUNITY = SNMP community (V2c only). (Default: "public")
SNMP_AUTHENTICATION_ENABLED = Enable SNMP authentication (V3 only). (Default: "false")
SNMP_AUTHENTICATION_PROTOCOL = Protocol for password encryption (V3 only). MD5 and SHA are currently supported. (Default: MD5)
SNMP_AUTHENTICATION_USERNAME = SNMP authentication username (V3 only).
SNMP_AUTHENTICATION_PASSWORD = SNMP authentication password (V3 only).
SNMP_PRIVATE_ENABLED = Enable SNMP encryption (V3 only). (Default: "false")
SNMP_PRIVATE_PROTOCOL = Protocol for SNMP data transmission (V3 only). DES and AES are currently supported (Default: "DES").
SNMP_PRIVATE_PASSWORD = SNMP private password (V3 only).
SNMP_SECURITY_ENGINE_ID = SNMP security engine ID (V3 only).
SNMP_CONTEXT_ENGINE_ID = SNMP context engine ID (V3 only).
SNMP_CONTEXT_NAME = SNMP context name (V3 only).
Run with Docker
Login to Instana Docker registry
sudo docker login https://containers.instana.io -u="_" -p="${AGENT_KEY}"
Start Docker container
sudo docker run -p 9464:9464/tcp --env SNMP_DESTINATION=127.0.0.1:162 --env GROUP_ID=1001 --env USER_ID=1002 containers.instana.io/instana/release/product/snmpnotifier:2.164.224-0
Run in Kubernetes
Create secret for instana docker registry
kubectl create secret docker-registry instana \
--docker-server=https://containers.instana.io \
--docker-username=_ \
--docker-password=${AGENT_KEY}
Deployment and Service Descriptor
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: instana-snmp-notifier-deployment
labels:
instana.io/component: instana-snmp-notifier
spec:
replicas: 1
selector:
matchLabels:
app: instana-snmp-notifier
template:
metadata:
labels:
app: instana-snmp-notifier
spec:
containers:
- env:
- name: GROUP_ID
value: '1001'
- name: USER_ID
value: '1002'
- name: SNMP_DESTINATION
value: 127.0.0.1:162
image: containers.instana.io/instana/release/product/snmpnotifier:2.164.224-0
name: instana-snmp-notifier
ports:
- containerPort: 9464
name: web-endpoint
protocol: TCP
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 500m
memory: 256Mi
imagePullSecrets:
- name: instana
---
apiVersion: v1
kind: Service
metadata:
name: instana-snmp-notifier-service
spec:
type: ClusterIP
ports:
- name: web-endpoint
port: 9464
targetPort: 9464
protocol: TCP
selector:
app: instana-snmp-notifier
Troubleshooting
Remote Write High Metric Delay
Monitoring issue type: prometheus_remote_write_high_delay
Prometheus metrics that are being ingested from the remote_write
endpoint, are being received with a high delay. This
results in potentially delayed alerting and makes correlating metrics from other sources more difficult.
Please look at tuning the Prometheus remote_write
configuration. Specifically adding the batch_send_deadline
parameter as defined below, which limits the delay to 1 second:
remote_write:
- url: "http://xxx.xxx.xxx.xxx:42699/prometheus/v1/receive"
queue_config:
batch_send_deadline: 1s
Full configuration options can be found in the Prometheus manual.