Apache Kafka Monitoring and Performance Management
Apache Kafka is a popular open source data streaming platform providing high-throughput data pipelines. As such, Instana has included comprehensive Kafka Monitoring in our Infrastructure and Application Monitoring solution. Instana discovers all Kafka instances present in the environment, understands the dependencies and interactions between the cluster and its servers, and automatically begins monitoring the health and performance of each instance.
To help DevOps optimize their Kafka clusters, Instana applies automation and artificial intelligence to Kafka performance and health monitoring, including predictive problem detection and AI-assisted root cause analysis. Instana reduces your need for explicit Kafka expertise within operations.
Kafka Performance and Health Monitoring
Instana monitors Kafka’s Key Performance Indicators (KPIs) at the individual component and application level for each instance. The Kafka KPIs are used for real-time health monitoring of each instance and cluster. The Service KPIs are used to understand Kafka’s performance within the overall environment.
Instana AI uses advanced statistical analysis, a set of specific Knowledge Base rules, and applied machine learning to determine the real-time health status of each instance and cluster.
Kafka Performance and Configuration Monitoring
Instana’s Apache Kafka sensor gathers all critical metrics to determine the health of each Kafka instance and the performance of all data pipelines on the cluster. Tracking Kafka configuration data allows Instana to monitor changes to the Kafka instances and correlate those changes to any overall performance issues.
The Kafka performance and configuration is summarized in a single dashboard that combines all information about setup, infrastructure, service levels and transactions. The dashboard allows IT Ops and DevOps to see all relevant Kafka data in a single place for easy-problem solving and performance optimization.
Kafka performance monitoring centers around metrics relevant to its interactions with the data pipelines and dependent applications that live in and around the Kafka cluster. Instana automatically identifies and collects all relevant metrics. Below are some sample screenshots of an Instana Kafka dashboard:
Kafka Performance and Health Dashboards show Kafka’s performance at the node, cluster, and broker levels, as well as Kafka configuration information.
Kafka Performance and Configuration Data
Instana’s Kafka Monitoring includes three types of data: Tracked Configuration, Performance Metrics, and Health Signatures:
|Tracked Configuration||Performance Metrics||Health Signatures|
|Version||Broker Traffic||Active Controller Count|
|Request Latency vs. Throughput||isrShrinks, isrExpansions|
|Broker Failures||Leader Elections|
|Broker State Metrics||Unclean Leader Elections|
|Average Idle Time||Network Processor Low Idle|
|Log Flushing||Request Handler Low Idle|
|Under Replicated Partitions|
Supported Kafka Environments
Thanks to the open source nature of Apache Kafka, there are numerous solutions to operate Apache Kafka in cloud and on-premise situations. Instana supports most common deployment strategies, including
- Installation on dedicated virtual or physical hardware (Windows, Linux)
- Installation in on-premise or managed Kubernetes environments
- Installation in on-premise or managed OpenShift environments
- Installation in on-premise or managed Pivotal Cloud Foundry environments
- Hosted Kafka services Amazon MSK (Amazon Managed Streaming for Apache Kafka)
Furthermore, Instana supports multiple distributions of Apache Kafka, including
- Vanilla Apache Kafka
- Confluent Kafka
- Cloudera Kafka
- Amazon MSK
Kafka Performance Management Sensor Installation: Getting Started
Ready to get better performance and availability from your Kafka services? Get started by signing up for an Instana Trial. If you already have an account, check out Instana’s Kafka Management Documentation.