How to Isolate a Service HotSpot with Continuous Production Java Profiling

Continuous Application Profiling in Production

One look at the Instana website lets you know that Instana is in the Observability (and APM) business — providing the nascent Enterprise Observability Platform, especially useful for Cloud-native and containerized microservices applications. A critical element of Instana’s Enterprise Observability Platform is continuous production profiling with AutoProfile™. AutoProfile attaches itself to every running process** automatically (no restarts, no code changes, no re-configuration), allowing users to analyze the performance of their code in production ALL THE TIME — including, of course, when it’s necessary to solve a problem.

Profiling examines application code internals running within a container, VM, or server and displays any performance slowdowns or disruptions. Instana’s continuous production profiling supports Java, Node.js, Go, Python and Ruby. Always on, Instana’s production profiler monitors CPU usage and Contention metrics, which delineate performance hotspots. Instana automatically profiles every process running any of the supported code stacks throughout the entire distributed network mesh.

The CPU usage and especially the Contention measurements give an indication of potential code issues that independently or together can create or do create a performance issue. You can use CPU profiles to pinpoint which methods are CPU intensive. Meanwhile, Contention profiles show which synchronized directives are causing long wait times, inefficient use of the CPU’s multithreading capabilities, and leading to bad overall performance.

Instana can visualize profiles as a flame graph or a tree graph, making it easy to see exactly where resources are being consumed. The tree graph bars are sized according to the amount of resources consumed. The graph is also ordered chronologically from the top down. Instana augments the profiles with other valuable metrics like CPU usage and Garbage Collector (GC) activity to provide you with context for understanding the impact of issues identified by the profile.

In this screenshot, the flame graph simplifies finding Java hotspots by clearly indicating the worst anomalies in red.

The flame graph simplifies finding Java hotspots by clearly indicating the worst anomalies in red.

The flame and tree graphs simplify finding Java hotspots by clearly indicating the worst anomalies in red. From the dashboard it’s possible to drill down to quickly find the code hot paths. When hotspots are shown in the Flame and Tree graphs, you can dig into the details, isolate the problem, and solve them with little or no application disruption.

The key steps for isolating a service hotspot with continuous Java profiling are:

  1. Turn on AutoProfile by editing the Instana agent configuration file
  2. Let Instana automatically discover all the JVMs used in the service
  3. Discover issues using the Flame and Tree graph and drill down to the line of code to find any Java code issues
    1. Identify CPU, Memory, and Wait Time Hotspots
    2. Determine which method or code segment is CPU intensive and can be optimized
    3. Investigate inefficient CPU multithreading

Solving a Production Problem in Java Code with the Instana Profiler

Instana’s production profiler supports multiple JVM platforms, Node.js, Go, Python, and Ruby. It monitors CPU and Memory use and Thread Contention metrics and shows performance hotspots. It deploys automatically once enabled and profiles code stacks throughout the distributed network mesh. The information gathered is displayed on Flame and Tree Graphs, from which you can drill down to the hotspot Java code.

In this screenshot, the user hovers over the Instana logo, which automatically opens the sidebar menu.

To examine a Profile, hover over the Instana logo, which automatically opens the sidebar menu.

To examine a Profile, hover over sidebar menu Instana logo which automatically opens the sidebar menu.

In this screenshot, the user selects Analytics in the sidebar menu to explore metrics and analytics more deeply.

Select Analytics in the sidebar menu to explore metrics and analytics more deeply.

Then, select Analytics in the sidebar menu.

In this screenshot, the user selects "Profiles" to display all the applications that have active profiles generated by the Instana profiler.

On the Analytics screen, select ‘Profiles” to display all the applications that have active profiles generated by the Instana profiler.

From there, drop down to the Analytics selection and click on it to open the Analytics screen. On the Analytics screen, navigate to the top level Analytics menu and select ‘Profiles”.

The Profiles page displays all the applications that have active profiles generated by the Instana profiler.

In this screenshot, Instana displays the profile page for the selected process – in this case, Instana Demo – Discount Application 2.0.

Instana displays the profile page for the selected process – in this case, Instana Demo – Discount Application 2.0.

In this screenshot, Instana displays a timespan graph of the CPU and Wait Time hotspots summary.

Instana displays a timespan graph of the CPU and Wait Time hotspots summary.

In this case, we select the Instana Demo – Discount Application 2.0 link above. Instana then displays the profile page for the selected process. The page shows a timeline graph for CPU activity for the process as well as Hotspot charts for the applicable metrics that have measurements that could contain Hotspots. In this example, the page displays CPU and Wait Time Hotspots.

In this screen we see a timespan graph of the CPU and Wait Time hotspots summary for the Instana Demo – Discount Application 2.0. Below it are break out windows that display the most serious issues for each category.

In this screenshot, the flame graph shows CPU consumption entries in color coded blocks that contain the monitored Java code. The orange bars show moderately high CPU consumption time, and the red very high CPU consumption.

The flame graph shows CPU consumption entries in color coded blocks that contain the monitored Java code. The orange bars show moderately high CPU consumption time, and the red very high CPU consumption.

Next we select the CPU link to display the flame graph (below) that shows CPU consumption entries in color coded blocks that contain the monitored Java code. The orange bars show moderately high CPU consumption time and the red very high CPU consumption.

We switch to the Tree View (below) to view Java code shown in the Flame graph in a hierarchical tree structure with hyperlinks to the code within the process methods along with the portion of the CPU consumption percentage of each.

In this screenshot, the Flame graph shows Java code shown in a hierarchical tree structure with hyperlinks to the code within the process methods along with the portion of the CPU consumption percentage of each.

The Flame graph shows a hierarchical tree structure with hyperlinks to the code within the process methods, along with the portion of the CPU consumption percentages.

We select a method that is consuming 28.7% of the CPU time to determine where it’s located and to enable us to change how the method is used, if desired. The profiler displays the code, in context, within the .java file in which it’s located.

In this screenshot, the profiler displays the code, in context, within the .java file in which it’s located.

The profiler displays the code, in context, within the .java file in which it’s located.

Summary

Instana’s production profiler provides “set it and forget it” continuous application code profiling. When turned on (which is done via a single config line), Instana profiling delivers a continuous stream of important code execution information to your dashboard – the SAME dashboard containing all your other observability data. It couldn’t be easier, but we’ll keep on improving it because we believe that manual instrumentation is something that truly belongs in a symphony orchestra.

** ALL processes running supported languages are automatically profiled continuously in production!

Play with Instana’s APM Observability Sandbox

Announcement, Engineering, Product
Instana Adds Production Ruby Profiling In the latest Instana release (Instana Release 201), AutoProfile for Ruby is generally available. This allows teams developing and supporting Ruby based applications to continuously collect and...
|
Announcement, Developer, Product, Thought Leadership
Instana has previously announced automated tracing of AWS Lambda applications in Node.js and Python. We are extending our best-in-class distributed tracing to include AWS Lambda functions written in Java and Go. The...
|
Profiling
There are multiple reasons why a program will consume more CPU resources than excepted. In the case of a high computational complexity of an algorithm, the amount of data it operates on...
|

Start your FREE TRIAL today!

Instana, an IBM company, provides an Enterprise Observability Platform with automated application monitoring capabilities to businesses operating complex, modern, cloud-native applications no matter where they reside – on-premises or in public and private clouds, including mobile devices or IBM Z.

Control hybrid modern applications with Instana’s AI-powered discovery of deep contextual dependencies inside hybrid applications. Instana also gives visibility into development pipelines to help enable closed-loop DevOps automation.

This provides actionable feedback needed for clients as they to optimize application performance, enable innovation and mitigate risk, helping Dev+Ops add value and efficiency to software delivery pipelines while meeting their service and business level objectives.

For further information, please visit instana.com.