Are your observability tools as clunky as a hybrid DVD-VHS player?
It’s time to upgrade to a best-in-breed solution for handling your log management and monitoring.
Listen to The Hoot podcast (or read the transcript below) as Instana CTO Pavlo Baron, and Humio Developer Advocate PJ Hagerty discuss the growing need for observability, and how the Humio-Instana integration can help point you to the highest quality data.
If you’ve listened and want to find out more, sign up for our upcoming Humio-Instana webinar on November 5th at 11:00 AM EDT / 5:00 PM CEST, which dives deep into how you can partner these two services to reach the deepest levels of observability.
Best-of-Breed Observability
PJ Hagerty: Observability has evolved as a mission-critical property when it comes to modern systems, and there’s still a lot of attention on it. What do you think are the driving forces behind the evolution of observability?
Pavlo Baron: The topic itself is not new, but it gets more attention these days due to the fact that the implementation and delivery and operations of software these days, especially around microservices oriented architectures, have become so complex. Since everything is continuously changing, the new challenges that the human being, no matter how many of them there are, will face an overstrained need to understand all of the implications of just a simple deployment of a simple piece of software. And this brings us to the necessity to build software in a way that we can feel the pulse, and have the ability to ask “Okay, is this application doing okay? How is the entire application doing, if this one thing is broken?”, without a deep understanding of what is in there.
PJ Hagerty: Because of the complexity of what we’re building, there are so many abstraction layers that have got away from knowing what’s going on in the underlying infrastructure. It seems that a lot of people have gotten away from knowing what’s going on in the underlying infrastructure and application, but observability gives us a window into what’s happening, and what’s absolutely crucial to what we’re doing.
Pavlo Baron: Totally. There is no one person who knows everything about a piece of software that is operating anymore. It’s all distributed, all over the team, where people have only partial knowledge of what they’re responsible for. Everything is a polyglot, with all of these different technical aspects that contribute to high-speed delivery. And depending on how independently the team operates, an additional challenge is created where there is no single expert on the whole stack.
PJ Hagerty: So the next question I have is, what do ITOps, DevOps, and SecOps need to know in this new, fast-paced environment? Because with observability, we’re bringing in so much information and there’s a lot going on.
Pavlo Baron: Well, you mentioned the word “information”. The way we see the world at Instana is that they are dealing with a lot of data, which is by far, not information. And that’s exactly the challenge because even if they have the expertise, they really don’t have time to dig into problems. Any kind of security incident or any kind of outage needs to be addressed immediately with the highest possible quality, which is good enough to get the service back up and running. They need a higher order of information, derived from all of the underlying noise. This is an additional challenge for observability, for the areas that you just mentioned. Because they move fast, and there is a lot of change, and they don’t have time to deal with low-level chunks of data. They need a clear statement provided by someone else, for example,
“Here’s what’s happening, here’s what’s broken, and here’s how much is impacted. Boom.”
And then from there, you can go ahead and use your expertise to fix the problem.
PJ Hagerty: Right, and that’s kind of key. Timing is everything in this game, especially when it comes to technology. Considering both Humio and Instana are platforms that focus on immediate real-time feedback, can you speak to how this helps ITOps, DevOps, and SecOps work more efficiently?
Pavlo Baron: Absolutely. If you’re operating in an ever-changing world, there are just a few moments of time that you can accept when recognizing the problem. This means that you need a modern tool that processes the data and stores it in a very efficient way, that allows you to access the information without any significant delay. So let’s look at logs first. Logs normally contain the latest information, of all possible data sources. Normally, it takes minutes, maybe hours until you really get anything reasonable extracted out of flux, due to technical reasons, such as processing. Humio addresses this problem in real-time, effectively extracting the information from totally chaotic data sources. And that’s exactly what Instana is doing with other data sources as well.
Telemetry might be reliable because normally it’s just numbers. But there’s a lot of information that hides inside of configuration changes. And traces are even more chaotic because they are like a huge distributed log, with anything that a developer could provide. So this real-time aspect is extremely important, and our integration goal with Humio not to just a show log entries to you, but to also give you a view of where it’s happening. For example, for a service or an application, we can translate the logs into our own understanding of the environment.
PJ Hagerty: That makes sense. And it doesn’t go without mentioning that it’s possible to integrate the data sources between the two. How important is that to people?
Pavlo Baron: It’s extremely important. There are so many tools out there that claim to be the single pane of glass. If you remember, there used to be hybrid devices that were both DVD and VHS players. And what you had was this shiny thing that couldn’t do it’s two jobs perfectly, and you were never fully satisfied with it. This is effectively what the typical observability tools out there are. They give you a little bit of everything, but as a human being, you need to be deeply involved and making sense of all the data sources that are disconnected and bring them together.
Humio and Instana’s integration is different. It’s the best-of-breed for both worlds.
We’ve married these two worlds into one perfect semantic understanding. So it’s not only an integration based on the data source, it’s also an integration that gives you an immediate understanding if there is a broken service, which is information derived from traces, which reference the logs that extracted the data. Nobody out there can give you this visibility on the trace scope.
PJ Hagerty: You mentioned the DVD/VHS player, a double feature, where it did the job good enough, but not really that well. And maybe for our younger listeners, we should mention the early flip camera phones that were not very good phones and not very good cameras. But they were what we had at the time. On the opposite end, Humio and Instana working together is actually the best of both worlds.
Pavlo Baron: It’s best-of-breed, and the semantics on top of the data processing is key.
PJ Hagerty: Absolutely, I agree. We talked a bit before about how dynamic applications, and how things are becoming more and more complex. What do you think are the pain points people are going to see with microservices, and increasing complexity as things move forward? What problems will people experience in the future?
Pavlo Baron: In the modern microservices world, it’s difficult to understand what is in an application. Let me give you an example. When you have a bunch of services, the services contribute to some kind of business case that you are in. This is exactly the application you’re looking at. Not much out there is modeled like this, but it has to be because this is the only thing that matters. When you see 10 services in front of you and five of them are broken, you need to understand what the impact is on the business. And you either do this in your head, through someone who knows what business applications are addressed by the services. Or, you have a tool or a set of tools that give you the ability to understand, for example, if the login service doesn’t work for application payment, but it works for application billing, how important is it?
This is what I think is the biggest challenge in the modern world. Working with applications as the only means for an observability and monitoring solution to help understand,
“Do I need to get up at 2 am and fix this right now? Can I postpone until the next morning?”
PJ Hagerty: Right. An idea of how many backups to backups we have. How stable is our environment? How stable is our so-called application?
Pavlo Baron: Yeah, “this environment” and “this application” thing is a huge source of confusion. I don’t know why this happened. But in the modern world, the abstraction of business applications is secondary because people tend to look at the environments, etc. If you have an environment that operates 20 business cases, how do you know what is important when something is broken? If you don’t understand the business case, then how do you understand the environmental components that are effectively supporting these business cases?
PJ Hagerty: I think it’s interesting because these are kind of the positive and negative trends that have been feeding the world of software and infrastructure for a while. A lot of people think that things are going to continue to focus on observability, but also with the ideas of chaos and cloud complexity. What trends are we going to see in software infrastructure over the next few years?
Pavlo Baron: The majority of modern software is going to be delivered as plain workloads into some kind of an environment, where they can run as containers in some kind of orchestrated manner. If you look at systems like AWS Fargate and the like, it goes into that same direction. You just drop a random container into the infrastructure that you don’t own, and it executes according to your expectations and specifications. The flow of operations will be even more abstract, and developers will get back to writing business applications, instead of playing with the technology. I think this is what’s going to happen, which immediately emphasizes the importance of having observability and treating applications as first-class citizens.
