Making Continuous Delivery Actually Work with Instana’s Continuous Discovery Technology

Dynamic APM and Continuous Discovery

One of the Six Pillars of Dynamic APM is Continuous Discovery. Continuous Discovery is the effortless discovery of your application’s structure: its infrastructure, middleware and code as well as the messages flowing across the application. In other words, Instana continuously discovers the complete technical stack of any API service (aka microservice) AND the request flows across the microservices that deliver the business application.

Why did we invest in this capability? Because modern operations has institutionalized the ability to continually evolve business applications. Agile development, continuous delivery, continuous integration, microservice architecture, clouds and the use of containers are all strategies that create constant change of systems, architecture, code and ultimately business process.

A key ingredient to the Instana Dynamic APM solution is our Agent architecture and specifically our use of Sensors. Sensors are mini agents – small programs specifically designed to attach to and monitor one thing. Instana’s Sensors are automatically managed by our SINGLE Agent (one per host). The Agent is responsible for discovering the operating technology running on that host, including the containers and whatever is running in those containers, and then applying the appropriate Sensor. And all of this happens continuously and automatically.

Sensors Are Active NOT Passive

Instana uses a single Agent with multiple Sensors. Currently, Instana supports over fifty sensors. These Sensors are NOT extensions. They are completely updated, loaded, and unloaded by the Agent, humans never touch them. There is even an optional command line interface which provides access to the agent state, individual sensors and agent logs.

A Sensor is designed to DISCOVER and MONITOR a specific technology and pass its data to the Agent. The Agent manages all communication to the Instana Service Quality Engine. After discovery, the Sensor collects the details and metric data to provide an accurate representation of the state of the component. A specific sensor will gather specific types of data about their respective technologies, which will vary depending on the technology. Sensors collect the following:

  • Configuration – catalogues current settings and states in order to keep track of any change
  • Events – Initial Discovery and all state changes (online, offline)
  • Traces – Traces are captured based upon the programming language platform
  • Metrics – qualitative attributes of the technology that indicate performance

In addition, discovery is recursive within a sensor. For example, the Java Machine sensor continues up the stack and discovers frameworks running on it (like Tomcat or Springboot), then assists the Agent to load the appropriate additional sensors.

Instana’s Sensor vs. Traditional APM Agent Approach

In my previous life, a customer asked me to install “APM” on what was, at the time, a “non-standard” stack. It was comprised of the following components:

  • Java SpringBoot
  • Kafka
  • Redis
  • Zookeeper
  • MySQL

Let’s compare the two installation processes:

 

Instana
Other APM
  • Install the single Instana Agent on each host
  • Instana Agent automatically discovers
    • Springboot
    • JVM
    • Kafka
    • Redis
    • Zookeeper
    • MySQL
  • And You’re Done!

By the way, you can deploy the Agent via containers if that is more convenient.
  • Copy the host-based Agent to each host. It was a host monitor which collected CPU and Memory.
  • This host-based Agent was also the “container” for any and all extensions.
  • Copy the Java based APM Agent to each host.
  • Find the location in the Spring Boot Applications requiring the -javaagent switch.
  • Decision Time: run Kafka in a JVM. Install a Java Agent into the Kafka runtime, OR add and configure an extension to monitor the Kafka process.
  • Install and configure the Redis Extension into the server-host Agent mentioned above.
  • Another Decision Time: run ZooKeeper in a JVM. Install a Java Agent into the ZooKeeper runtime, OR add and configure an extension to monitor the ZooKeeper process.
    • Install the ZooKeeper Extension.
  • Install another database monitoring Agent.
  • If they are running in Docker, install the “Docker Extension.”
  • Create a Redis Custom Dashboard.
  • Create a Zookeeper Custom Dashboard.
  • Create a Kafka Custom Dashboard.
  • Configure Docker Dashboard.
  • Create Custom Alerts for Docker.
  • Create Custom Alert for Zookeeper.
  • Create Custom Alert for Kafka.
  • Create Custom Alerts for Redis.

 

Now let’s ask an interesting question: What happens when you change something?

For example, you need to move Kafka to its own machine. With Instana, you do nothing. The Agent automatically cleans up the sensor when Kafka is removed, and adds the sensor on the new host. All you have to do is make sure there is an Agent running on that new host. With other APM tools? Well,… you get the picture.

This is a screenshot from the our user interface that shows the Agent and Sensors deployed and collecting performance metrics after installation with ZERO human intervention:

The reality is that initial setup and ongoing configuration is an issue with almost every other monitoring and APM vendor. They all utilize an “extension” approach. It might look like the following:

  • Use your Java Agent as a data collection platform for one or many extensions.
  • Collect metrics from processes.
  • Send metics to collection server or services.
  • Parse the data into a complete Relational Database Schema.
  • Data is a REST API in that Relational Database Schema.
  • User Interface application pulls data from complex data schema and reconstitutes it into a JSON, or some other data format.

Why does Instana implement Sensors?

APM is not the same as it was 15 years ago. More data is not necessarily a good thing. We don’t need to collect 200 metrics for every extension. Collecting a few focused metrics with the correct “health signatures” is a better approach.

As an example, take a look at our Redis Sensor Documentation which includes: Tracked Configuration, Metrics, and Health Signatures:

Redis Monitoring Documentation Screenshot

Why is all this Essential for Continuous Delivery?

To summarize, the Sensor approach automates continuous discovery and change detection, metrics collection and even the notion of health determination via health signature based monitoring.

  1. Nobody has time to configure tools. The speed of continuous delivery is the whole point, why slow the process down configuring extensions or setting up health rules?
  2. You don’t know in advance where what pieces will run. The point of abstracting the infrastructure is to not care where components or services actually run. The monitoring tool must figure this out automatically.
  3. Time to feedback is essential. Instana’s Sensor approach discovers new structure and immediately reports the ramifications of the recent push in 3 seconds. With Instana, no matter how your continuous delivery system has organized the middleware and code, we discover it and monitor the performance of your services.

The Moral of the Sensor Story

  1. Change happens. Instana’s Continuous Discovery automates and captures the continuous state changes in current environments.
  2. Confidence! You need immediate understanding of the quality and performance of your application after any change. Instana automates that also!
  3. One automated Agent with Sensors is SUPERIOR to many Agents. Deploy the Agent and let it do the work. Get out of the business of configuring monitoring tools.

Instana is fundamentally designed for the new world of constant change and enables the goals of Continuous Delivery.

Learn more about Instana Dynamic Application Performance Management or contact me at hugh.brien@instana.com