Monitoring Java Virtual Machine
TABLE OF CONTENTS
Introduction
The Java sensor provides automated code instrumentation for supported technologies, zero configuration health monitoring of JVM instances, and end-to-end traces of requests across all systems.
Supported Languages
- Clojure
- Java 6 to 14; Java 15 is considered experimental
- Kotlin
- Scala
System requirements
Due to potential JVM bugs, we recommend using the most up to date JVM versions to make use of bug-free runtimes.
Note: Do not attempt to use agents from multiple vendors at the same time; this leads to unpredictable results and might crash your application. The Instana agent does not automatically monitor JVMs that also have the following agents running:
- AppDynamics
- Glow Root
- Java Flight Recorder
- Kanela
- New Relic
- Riverbed AppInternals
- Spring Loaded
- Wily Introscope, also known as CA Application Performance Management and Broadcom DX Application Performance Management
This is not supported. If you decide to monitor JVMs running any of the above agents, set the following value:
com.instana.plugin.javatrace:
trace-jvms-with-problematic-agents: trueJVM support
The following JVM distributions are supported.
| Name | Limitations |
|---|---|
| Amazon Corretto | - |
| Azul Zulu | - |
| IBM J9 | See IBM J9 limitations. |
| IBM OpenJ9 | - |
| OpenJDK | - |
| AdoptOpenJDK | - |
| Oracle HotSpot | - |
| SAP JVM | - |
| Sun HotSpot | - |
| BEA JRockit | - |
Java 8 or lower
On Amazon Corretto, Azul Zulu, OpenJDK, Oracle HotSpot, or SAP JVM, make the tools.jar file available in the runtime of each monitored JVM. The file is located at $JAVA_HOME/lib/tools.jar.
The Java 8 builds up to 1.8.0_40 have several known issues relating to the implementation of lambdas. Due to these issues, the Instana agent does not monitor Java 8 builds 1.8.0_40 or lower.
When using the G1 garbage collector, please update to 1.8.0_181 or later, as some crashes have been fixed in later releases.
Java 9 or higher
Java 9 introduces modularized builds based on the Jigsaw framework. If you use a custom runtime image, such as those created with jmod or jlink, make sure to include the following two modules in the instrumented JVM:
java.instrumentjdk.attach
TIP: To check if your Java 9 or higher runtime contains the required java.instrument and jdk.attach modules, run the following command:
java --list-modulesIf you try running the command on a JVM version 8 or lower, the Unrecognized option: --list-modules error message is displayed.
Running on Windows
A popular choice for running services, such as Apache Tomcat, is to use the in-process Server JVM.
In those cases the monitored processes do not run as C:\java12\bin\java.exe but as C:\Program Files\Apache Software Foundation\Tomcat 8.5\bin\Tomcat8.exe. This setup is supported, but before starting the service like this set PATH=%PATH%;C:\java12\bin, the bin directory on the JVM running the service must be on the PATH.
This is because the server jvm.dll needs to load the C:\java12\bin\instrument.dll. If it is specified on the path, it will not find it, and this results in an error:
com.sun.tools.attach.AgentLoadException: Failed to load agent library: instrument was not loaded.Can't find dependent librariesIBM J9 limitations
If -javaagent is not set as a command line argument, certain versions of J9 will not support all the required features. We recommend starting the JVM with the javaagent by running this command:
java -javaagent:instana-javaagent-1.0.0.jar -jar app.jar server
The JAR is available in maven central and for more information about the JVM agent, see the GitHub repository.
If using Java 8 - Service refresh 3 fix pack 22 (Dec 2016), the -XX:+EnableHCR option performs the same operation. This temporary option enables late attached agents to redefine or retransform classes. It might incur a performance penalty, however, and will be removed in a future update when no longer necessary.
To enable tracing, additional command line switches may be required:
-Dcom.ibm.tools.attach.enable=yesshould be the default setting.-Xshareclasses:none(class sharing can interfere with bytecode instrumentation) or if available-Xshareclasses:enableBCIto enable instrumentation despite class sharing.
Instrumented frameworks and libraries
Note: We explicitly list all supported major versions (e.g. >= 1.0 doesn't include 2.x versions or newer).
HTTP libraries
| Name | Versions |
|---|---|
| Akka HTTP | >= 2.4, >= 10.0 |
| Apache Async HttpClient | >= 4.0 |
| Apache Axis | >= 1.3 |
| Apache Axis2 | >= 1.5 |
| Apache CXF | >= 2.3, >= 3.0 |
| Apache HttpClient | >= 3.0, >= 4.3 |
| Apache Wicket | >= 6.0, >= 7.4, >= 8.5 |
| AsyncHttpClient (AHC) | >= 2.0 |
| AsyncHttpClient (NING) | >= 1.8 |
| Dropwizard | >= 0.9, >= 1.0, >= 2.0 |
| Feign | >= 9.0, >= 10.2, >= 11.0 |
| Finagle | >= 6.45 |
| GraphQL (Servlet) | >= 8.0, >= 9.0, >= 10.0 |
| GraphQL (Spring Boot) | >= 1.0 |
| GraphQL Java | >= 4.2, >= 5.0, >= 6.0, >= 7.0, >= 8.0, >= 9.7, >= 10.0, >= 11.0, >= 12.0, >= 13.0 |
| Grizzly | >= 2.1 |
| HTTP Kit | >= 2.2 |
| http4s1 | >= 0.19.0 |
| HttpUrlConnection | All supported JDKs |
| JAX-RS | |
| JAX-WS | |
| Jersey | >= 1.1, >= 2.20 |
| JSF | >= 2.0 |
| Micronaut | >= 1.1, >= 2.0 |
| NanoHTTPD | >= 2.2 |
| OkHttp | >= 2.0, >= 3.4, >= 4.0 |
| Play Framework (Play2) | >= 2.3 |
| Ratpack | >= 1.5 |
| RESTEasy | >= 3.0, >= 4.0 |
| Scalatra | >= 2.3 |
| Servlet | >= 2.0, >= 3.0, >= 4.0 |
| Spray | >= 1.3 |
| Spring Boot | >= 1.2.0.RELEASE, >= 2.0.0.RELEASE |
| Spring Cloud Gateway | >= 2.0.2.RELEASE |
| Spring REST | >= 4.0.0.RELEASE, >= 5.0.0.RELEASE |
| Spring Web | >= 3.0.0.RELEASE, >= 4.0.0.RELEASE, >= 5.0.0.RELEASE |
| Spring Webflux | >= 2.0.0.RELEASE |
| Vaadin | >= 7.0, >= 8.7 |
| Vert.x-Web | >= 3.0 |
| WebMethods Glue | >= 5.0 |
Databases
| Name | Versions |
|---|---|
| Amazon DynamoDB | >= 1.11.106 |
| Amazon Elasticache | >= 1.1 |
| Amazon S3 | >= 1.11.55 |
| Apache HBase | >= 1.1, >= 2.0 |
| Cassandra | >= 2.0, >= 3.0 |
| Couchbase | >= 2.5, >= 3.0 |
| Ehcache | >= 2.0 |
| Elasticsearch | >= 1.3, >= 2.3, >= 5.0, >= 6.0, >= 7.0 |
| FaunaDB | >= 1.2 |
| Google Cloud Bigtable (HBase 1.x) | >= 1.0 |
| Google Cloud Bigtable (HBase 2.x) | >= 1.1 |
| Google Cloud Storage | >= 1.2 |
| Hazelcast Java Client | >= 3.7, >= 4.0 |
| JDBC | >= 4 |
| JDBC (H2 Database Engine) | >= 1.1 |
| JDBC (IBM DB2) | >= 11.1 |
| JDBC (Microsoft SQL Server) | >= 6.2, >= 7.0 |
| JDBC (MySQL Connector/J) | >= 5.1, >= 6.0, >= 8.0 |
| JDBC (Oracle) | >= 12.1, >= 19.3 |
| JDBC (PostgreSQL) | >= 7.4, >= 8.4, >= 9.0 |
| JDBC (Sybase) | >= 1.3 |
| MongoDB (Java Driver) | >= 2.13, >= 3.0 |
| MongoDB (ReactiveMongo) | >= 0.12.2 |
| MongoDB (Spring Data MongoDB) | >= 2.0.0.RELEASE |
| Neo4j | >= 1.5 |
| Redis (Jedis) | >= 2.8, >= 3.0 |
| Redis (Lettuce) | >= 3.4, >= 4.1, >= 5.0, >= 6.0 |
| Redis (Redisson) | >= 3.0 |
| Redis (Spring Data Redis) | >= 2.0.0.RELEASE |
| Redis (Spring Redis Reactor) | >= 2.0.0.RELEASE |
| Redis (Spring Session Data Redis) | >= 2.0.0.RELEASE |
| Redis (Vert.x) | >= 3.1 |
| SpyMemcached | >= 2.10 |
| SpyMemcached (Amazon ElastiCache) | >= 1.1 |
| SpyMemcached (Google XMemcached) | >= 1.4, >= 2.0 |
| SpyMemcached (Netty) | >= 4.1 |
| SpyMemcached (Shade) | >= 1.8 |
Messaging services
| Name | Versions |
|---|---|
| Aerospike | >= 3.3, >= 4.0, >= 5.0 |
| Akka Remote | >= 2.3 |
| Amazon Kinesis | >= 1.11.106 |
| Amazon SNS | >= 1.11.79 |
| Amazon SQS | >= 1.11.79 |
| Apache Camel | >= 2.17, >= 3.0 |
| Executor Pools | All supported JDKs |
| Fork Join Pool | All supported JDKs |
| Google Cloud Pub/Sub (Client for Java) | >= 1.105 |
| Google Cloud Pub/Sub (Spring Cloud GCP) | >= 1.2.5.RELEASE |
| gRPC | >= 1.2 |
| HornetQ | >= 2.2 |
| IBM MQ | >= 8.0 |
| JMS | |
| JMS (Apache ActiveMQ) | >= 5.13 |
| Kafka (Apache)2 | >= 0.9, >= 1.0, >= 2.0 |
| Kafka (Reactor) | >= 1.0 |
| Mule | >= 3.3 |
| Netflix Hystrix | >= 1.0 |
| RabbitMQ | >= 3.6, >= 4.11, >= 5.7 |
| RabbitMQ (Spring) | >= 1.0.0.RELEASE, >= 2.0.0.RELEASE |
| Tibco ESB |
Logging frameworks
| Name | Versions |
|---|---|
| Java Util Logging | All supported JDKs |
| JBoss Log Manager (Log4j) | >= 1.0 |
| JBoss Logging3 | >= 3.0 |
| Log4j | >= 1.2 |
| Log4j 2 | >= 2.4 |
| Log4j over SLF4J | >= 1.5 |
| Logback | >= 0.9, >= 1.0 |
| SLF4J | >= 1.5 |
| SLF4J over Log4j | >= 1.1 |
LDAP frameworks
| Name | Versions |
|---|---|
| JNDI | All supported JDKs |
| Ldaptive (Apache) | >= 1.0 |
| Ldaptive (JLDAP) | >= 1.0 |
| Ldaptive (JNDI) | >= 1.0 |
| Ldaptive (Netscape) | >= 1.0 |
| Ldaptive (UnboundID) | >= 1.0 |
| Netscape | >= 4.1 |
| Novell | >= 2009 |
| UnboundID | >= 3.2 |
Web and application servers
- Apache Tomcat
- Eclipse Jetty
- Glassfish / Payara
- IBM Websphere
- JBoss AS / Wildfly
- Oracle / BEA Weblogic
- Pega Systems AS
- Sun ONE Server
Other instrumented technologies
| Name | Versions |
|---|---|
| Akka Actor | >= 2.3 |
| Corba (Sun) | |
| DistributeMe | >= 2.3 |
| EJB (Apache Open EJB) | >= 4.0 |
| EJB (Glassfish) | |
| EJB (JBoss / Wildfly) | >= 5, >= 7, >= 8, >= 11 |
| EJB (Weblogic) | >= 12 |
| FTP Commons | >= 3.5 |
| FTP JSCH | >= 0.1.38 |
| Google Cloud Store | >= 1.2 |
| Google GWT User | >= 1.5, >= 2.0 |
| Java Mail | All supported JDKs |
| Java RMI | All supported JDKs |
| JBoss Scheduler | >= 4.2 |
| Kotlin Coroutines | >= 1.0 |
| Lift Actor | >= 2.6, >= 3.3 |
| Quartz Scheduler | >= 1.7, >= 2.0 |
| Seam Mail (JBoss) | >= 3.1 |
| Spring Batch | >= 1.1.4.RELEASE, >= 2.0.0.RELEASE |
| Spring Context | >= 3.0.0.RELEASE, >= 4.0.1.RELEASE, >= 5.0.0.RELEASE |
| Spring Mail | >= 1.2.0.RELEASE, >= 2.0.0.RELEASE |
| Spring Scheduler | >= 1.1.4.RELEASE, >= 2.0.0.RELEASE |
| Sun ON/RPC | >= 1.1 |
| TABEX4 (BOI) | |
| Vert.x (Hazelcast Cluster Manager) | >= 3.3 |
Installation
The Java sensor is automatically deployed, configured, and installed by the Instana agent. To ensure your Java applications are instrumented, make sure your JVM distribution is supported.
Configuration
Although there is no configuration required for out of the box metrics and distributed tracing, individual components are configurable.
For more information on how to configure the sensor, see Java configuration.
Metrics collection
On the JVM dashboard, configuration and performance metrics for the JVM instance are displayed, along with any custom metrics.
Configuration data
| Configuration | Description |
|---|---|
| Java version | The Java version. |
| Java runtime | The Java Runtime Environment (JRE). |
| Maximum heap | Maximum heap size (GiB) allocated to the JVM. |
| Classpath | The classpath parameter set in the JVM. |
Performance metrics
Memory used
Total memory used by the JVM. The current measured KPI value is displayed.
Data point: Byte count is collected from java.lang.Runtime#totalMemory.
Threads
Number of threads that are in states: new, runnable, timed-waiting, waiting, or blocked. The values are displayed on a graph over a selected time period.
Data point:
- Current thread-ids are collected from
java.lang.management.ThreadMXBean#getAllThreadIds. - Thread states are collected from
ThreadMXBean#getThreadInfo.
Heap Memory
Total heap memory used by the JVM. The value is displayed on a graph over a selected time period.
Data point:
- Byte count is collected from
java.lang.Runtime#totalMemory. - The maximum displayed in the graph is either determined by parsing the
-Xmxcommand line parameter or is collected fromjava.lang.Runtime#maxMemory.
Memory Pools
Memory pool usage. Heap and non-heap pools are displayed on a graph over a selected time period.
Data point:
- Pool information is collected from
ManagementFactory#getMemoryPoolMXBeans. - Graph values are collected from each pools
java.lang.management.MemoryUsage. - The maximum value is collected from
getMax, the initial value is collected fromgetInit, and the current value is collected fromgetUsage.
Garbage Collection
Garbage collection activation and runtime values are displayed on a graph over a selected time period.
Data points:
- Garbage collection information is collected from
ManagementFactory#getGarbageCollectorMXBeans. - Graph values are collected from each collectors
java.lang.management.GarbageCollectorMXBean. - Garbage collection runtime is collected from
getCollectionTimewhich, according to JavaDoc, is the approximate accumulated collection elapsed time in milliseconds. - Invocation count is collected from
getCollectionCount. - Both the
getCollectionTimeandgetCollectionCountvalues are the calculated differential during 1 second.
Suspension
Calculated suspension of application execution displayed on a graph over a selected time period.
Data point: Calculated per the in-app Instana measurement thread.
Health signatures
For each sensor, there is a curated knowledge base of health signatures that are evaluated continuously against the incoming metrics and are used to raise issues or incidents depending on user impact.
Built-in events trigger issues or incidents based on failing health signatures on entities, and custom events trigger issues or incidents based on the thresholds of an individual metric of any given entity.
For information about the built-in events for the Java sensor, see the Built-in events reference.
Custom metrics
Dropwizard Metrics Library
If the JVM has the Dropwizard metrics library loaded, custom metrics are collected and displayed at the bottom of the JVM dashboard.
To prevent the backend from overloading, there is a default limit of 200 metrics.
To disable or change the limit of metrics being gathered, use the following configuration:
com.instana.plugin.java:
dropwizardMetricCollection:
enabled: false
limit: 200If the metrics are not displayed even though the library is loaded, register your MetricRegistry instance as the default. For example:
import com.codahale.metrics.SharedMetricRegistries;
import com.codahale.metrics.MetricRegistry;
MetricRegistry metricRegistry = new MetricRegistry();
SharedMetricRegistries.setDefault("default", metricRegistry);Micrometer Metrics Library
If the JVM has the Micrometer metrics library loaded, custom metrics are collected and displayed at the bottom of the JVM's dashboard.
To prevent the backend from overloading, there is a default limit of 200 metrics.
To disable or change the limit of metrics being gathered, use the following configuration:
com.instana.plugin.java:
micrometerMetricCollection:
enabled: false
limit: 200If the metrics are not displayed even though the library is loaded, consider registering your io.micrometer.core.instrument.MeterRegistry instance as the default one. For example:
import io.micrometer.core.instrument.Metrics;
Metrics.globalRegistry.add(meterRegistry)Other features
Live Thread Dump
To view a live thread dump for the JVM, click Get Thread Dump.
Heap Dump
To create a heap dump for the JVM, click Get Heap Dump. You will be prompted for a location local to the JVM to store the heap dump.
Instana AutoTrace™
By default, all requests are monitored, and a distributed trace is created automatically for each of them. This includes cross host and cross language tracing.
For more information, see Instana AutoTrace™.
Logging MDC populated with the Trace ID
When using Log4j, Log4j2, or Logback, to enable a more precise correlation of logging and tracing, Instana automatically populates the MDC with the trace ID. The MDC variable name is instana.trace.id.
For information on how to use it in format strings, refer to the documentation for your logging framework.
Custom tracing
Fully automated out-of-the-box tracing instrumentation is provided, but in some cases, you may prefer to send custom traces to your Instana dashboard. If so, there are are number of available methods.
Java Trace SDK
If you want to instrument a framework that is not yet supported by Instana, or monitor the requests of a custom application, we recommend using the Java Trace SDK and viewing the GitHub repo.
Before you implement custom tracing using the SDK, please read the tracing best practices.
Configuration-based Java Trace SDK
There may be situations in which is not feasible or desirable to use the Java Trace SDK, which requires manipulating the source code or get in touch with someone who can do that. In these cases the configuration-based Java Trace SDK can be used. Albeit less feature-rich than the programmatic Java Trace SDK, it allows a declarative declaration of spans and tags that cover a good amount of common use-cases.
Before you implement custom tracing using the configuration-based Java Trace SDK, please read the tracing best practices.
Java OpenTracing API
To collect traces that are described via the OpenTracing API, you must use Java OpenTracing. For more information, see OpenTracing.
IMPORTANT: Disable automatic tracing (Instana AutoTrace™) before using the Java OpenTracing API. For more information, see Disabling automatic instrumentation.
OpenCensus Instana Trace Exporter
Instana provides an OpenCensus Trace Exporter for applications written in Java. Using the Instana agent processes as proxy, Instana forwards traces that are exported by applications instrumented with Census to its backend.
For more information, see the OpenCensus Exporters website.
IMPORTANT: Please disable automatic tracing (Instana AutoTrace™) if the agent is used on the same host as Census. For more information, see Disabling automatic instrumentation.
Instana AutoProfile™
Profiles are essential for locating performance hot spots and bottlenecks at the code level. They are instrumental in reducing resource consumption and improving performance.
AutoProfile™ generates and reports process profiles to Instana. Unlike development-time and on-demand profilers, where a user must manually initiate profiling, AutoProfile™ automatically schedules and continuously performs profiling appropriate for critical production environments.
For more information, see our Instana AutoProfile™ docs.
Troubleshooting
Unsupported Java Virtual Machine version 8
Monitoring issue type: java_8_unmonitored_version
Your application is running on an unsupported Java 8 Hotspot Virtual Machine with a version earlier than 1.8.0_40.
Instana will not monitor Java applications running in such a Hotspot Virtual Machine due to severe bugs in the latter that lead to incompatibilities and crashes.
We strongly advise to update the Java Virtual Machine to a more recent version, preferably a JDK 8 version 252 or above, or to another Java Long Term Support release.
For more information on the JVMs supported by Instana, please refer to the JVM support documentation.
Incompatible agent detected
Monitoring issue type: jvm_incompatible_agent_detected
The Instana agent has detect another agent attached to this Java Virtual Machine that is known to be incompatible. To prevent issues from occurring with this Java Virtual Machine, the Instana agent is not collecting tracing data from the application running inside the Java Virtual Machine. This may result in Application Perspectives, Services and Endpoints missing data, as well as spurious alerts about drop in calls to Application Perspectives, Services and Endpoints.
To resolve the issue, remove the incompatible agent from your Java Virtual Machine.
Restart needed
Monitoring issue type: jvm_attach_socket
The attachment facility of this Java Virtual Machine is stuck, which is known to happen from time to time due to long-standing bugs of the Java Virtual Machine. The Java Virtual Machine is unable to connect to a communication socket used during dynamic attach. Unfortunately, this cannot be fixed without restarting the Java Virtual Machine.
Network visibility issues
Monitoring issue type: jvm_attach_network
The instrumentation of this Java Virtual Machine failed due to issues with the network visibility from this Java Virtual Machine and the host agent. Specifically, the host agent must be reachable from this Java Virtual Machine as described in the Network Requirements documentation. In containerized environments, network policies and configurations may prevent this from happening.
Attach tools missing
Monitoring issue type: jvm_attach_tools
The JDK used during dynamic attach does not ship the required attach code. Please consult the Java version 8 or lower and Java version 9 or higher sections on this page for details.
Container Attachment failed
Monitoring issue type: jvm_attach_container_command
The command used to dynamically attach to a containerized Java Virtual Machine did not succeed. See the exit code from the specific event description for more details.