Prometheus
TABLE OF CONTENTS
Introduction
Collecting metrics data with Prometheus is becoming more popular. With Instana, it's easy to capture Prometheus metrics and correlate them using our extensive knowledge graph. A typical example is custom business metrics.
The Instana Prometheus sensor doesn't require a Prometheus server. The sensor captures metrics directly from the endpoints that are exposed by the monitored systems. For each Instana agent, specify which Prometheus endpoints you want to poll and which metrics should be collected from them using regular expressions.
Configuration
Static configuration
For the Prometheus sensor to be able to collect data, metric sources must be defined in the agent configuration file; <agent_install_dir>/etc/instana/configuration.yaml. The sensor picks up any type of metric. With a regular expression, you can define which metrics should captured for a specific metric endpoint.
# Static Prometheus Configuration
com.instana.plugin.prometheus:
customMetricSources:
- url: 'http://localhost:8080/metrics' # endpoint with exposed metrics
metricNameIncludeRegex: '^sample_app_request'
- url: 'http://localhost:9100/prometheus' # endpoint with exposed metrics
metricNameIncludeRegex: '^sample_app_request'Regex in the example above, configures a sensor to gather all metrics starting with sample_app_request e.g. sample_app_request_duration. In the case that metricNameIncludeRegex is not defined, the sensor collects all the exposed metrics until the defined limit.
Kubernetes configuration
Our Prometheus sensor detects if Kubernetes is being used and automatically gathers all the available pods on the node the sensor is running on. The URL field in the configuration is still required, but instead of specifying the whole URL, the path part is sufficient, e.g. /metrics.
# Kubernetes Discovery Prometheus Configuration
com.instana.plugin.prometheus:
customMetricSources:
- url: '/metrics' # url path part which expose metrics
metricNameIncludeRegex: '^sample_app_request'
- url: '/prometheus-metrics' # url path part which expose metrics
metricNameIncludeRegex: '^sample_app_request' Metrics collection
Prometheus custom metrics
Prometheus metrics appear as "Prometheus Apps" associated with the host from where they are collected. You can query Prometheus custom metrics using the "Dynamic Focus", "Event & Alerts", and the "Grafana Plugin" with entity.type:prometheus.
Metrics
- Counters (max. 700)
- Gauges (max. 700)
- Histograms (max. 700)
- Summaries (max. 700)
Alerting
To configure, head over to "Settings → Team Settings → Events & Alerts → Alert Channels → Add Alert Channel".
The Prometheus WebHooks are send as an HTTP PUT to the configured Prometheus Alertmanager Webhook Receiver.
SNMP Trap Notifier
The SNMP notifier sends SNMP traps to any given SNMP receiver. To ensure proper interaction with instana alerts, we provide a SNMP notifier with a custom alert mapping as docker image in our private docker registry.
Alert Channel Configuration
Webhook Url: http://{SMPT-Notifier-Host}:9464/alerts
Prometheus Alertmanager Configuration
To configure the Notifier Service, please specify the following as container environment variables (see below).
SNMP_VERSION = SNMP version. V2c and V3 are currently supported (Default: V2c)
SNMP_DESTINATION = SNMP trap server destination. (Default: 127.0.0.1:162)
SNMP_RETRIES = SNMP number of retries (Default: 1)
SNMP_TRAP_DEFAULT_OID = Trap OID to send if none is found in the alert labels (Defualt: "1.3.6.1.4.1.98789.0.1")
SNMP_COMMUNITY = SNMP community (V2c only). (Default: "public")
SNMP_AUTHENTICATION_ENABLED = Enable SNMP authentication (V3 only).
SNMP_AUTHENTICATION_PROTOCOL = Protocol for password encryption (V3 only). MD5 and SHA are currently supported. (Default: MD5)
SNMP_AUTHENTICATION_USERNAME = SNMP authentication username (V3 only). Passing secrets to the command line is not recommanded, consider using the (Default: USERNAME)SNMP_NOTIFIER_AUTH_PASSWORD = SNMP authentication password (V3 only). (Default: PASSWORD)
SNMP_PRIVATE_ENABLED = Enable SNMP encryption (V3 only).
SNMP_PRIVATE_PROTOCOL = Protocol for SNMP data transmission (V3 only). DES and AES are currently supported (Default: DES).
SNMP_PRIVATE_PASSWORD = SNMP private password (V3 only). (Default: SECRET)
SNMP_SECURITY_ENGINE_ID = SNMP security engine ID (V3 only)
SNMP_CONTEXT_ENGINE_ID = SNMP context engine ID (V3 only)
SNMP_CONTEXT_NAME = SNMP context name (V3 only)Run as Docker
Login to instana docker registry
docker login https://containers.instana.io -u="_" -p="${AGENT_KEY}"Start docker container
sudo docker run -p 9464:9464/tcp --env SNMP_DESTINATION=127.0.0.1:162 --env GROUP_ID=1001 --env USER_ID=1002 containers.instana.io/instana/release/product/snmpnotifier:2.164.224-0Run in Kubernetes
Create secret for instana docker registry
kubectl create secret docker-registry instana \
--docker-server=https://containers.instana.io \
--docker-username=_ \
--docker-password=${AGENT_KEY}Deployment and Service Descriptor
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: instana-snmp-notifier-deployment
labels:
instana.io/component: instana-snmp-notifier
spec:
replicas: 1
selector:
matchLabels:
app: instana-snmp-notifier
template:
metadata:
labels:
app: instana-snmp-notifier
spec:
containers:
- env:
- name: GROUP_ID
value: '1001'
- name: USER_ID
value: '1002'
- name: SNMP_DESTINATION
value: 127.0.0.1:162
image: containers.instana.io/instana/release/product/snmpnotifier:2.164.224-0
name: instana-snmp-notifier
ports:
- containerPort: 9464
name: web-endpoint
protocol: TCP
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 500m
memory: 256Mi
imagePullSecrets:
- name: instana
---
apiVersion: v1
kind: Service
metadata:
name: instana-snmp-notifier-service
spec:
type: ClusterIP
ports:
- name: web-endpoint
port: 9464
targetPort: 9464
protocol: TCP
selector:
app: instana-snmp-notifier