Software and system engineers today are on the front line of a battle in trying to effectively and efficiently manage complexity and change while supporting ongoing organizational agility and adaptiveness. With this in mind, many teams are turning to Observability as a vital tool in a much bigger toolkit consisting of concepts, processes, models, and technologies, depending on your interpretation of this now revamped term. To add more to the confusion surrounding the difference between monitoring and observability, I would like to introduce a new higher-order way of thinking about these using the acronym APM with A standing for attention, P for perception, and M for memory.
Attention, perception, and memory are the basis for human cognition. Since a solution like Instana is designed to support and extend the cognitive capacities of teams and individual engineers, it stands to reason that these three elements are considered foundational in our pursuit of excellence, innovation, and industry leadership.
In the rest of this article, I will focus on attention in the context of observability and monitoring, which I regard as the overarching process to the effective management of the former. That said, I would like to emphasize that while it is possible to discuss attention in isolation in practice, it is very much interweaved and interrelated with perception and memory. Unless we attend to something, we neither perceive it nor remember it. All APM parts are active and influence each other. Our expectation (from memory) of what is sensed will impact how we perceive and attend to it. Cognition involves largely the interaction between stimuli and stored knowledge within a frame.
Attention comes in many forms, but in general, it involves the ability to select a subset of information (or data) processed by other cognitive processing parts. Attention selection is required due to the vast amount of stimulus that comes in via sensory channels. Selection ensures that both machines (in the automated service sense like Instana) and minds (operational users) do not become overloaded. Attention acts much like a spotlight; it can be made larger or smaller. Some would argue that it is the key to intelligence: the ability to focus on what is significant and to control how the focus shifts over space and time based on an inner attendance to attention itself.
This sounds very much like cybernetics, a role I continue to envisage the next iteration of application performance monitoring solutions playing today in a new world of rebranded observability. From my many years of performance troubleshooting large scale distributed systems, what distinguishes experts from novices is the effectiveness of attention – what to attend to and what to ignore or discard when everyone else around you is pushing data and reports into your vision. A sort of executive control of attention is paramount here and I suspect in successful products in the observability and monitoring space in the long term. Are we not all trying to build intelligence over data sinks, pipelines, or platforms? Everything else such as wide events or deep traces is just pretentious noise.
Attention can be top-down or bottom-up. In top-down, it is steered by intentions or policies of surveillance. In bottom-up, it is that novel sensory stimulus that surfaces into our consciousness. One of the most critical tasks for an APM solution, whether for observability or monitoring purposes, is to make sure that information is communicated correctly and that important information is not just seen but also processed at the identification level. Attention is present at the biological level of the brain with neurological signals competing with each other in firing, boosting, or dampening other adjacent or associated equivalents. Signals and boundaries are everywhere.
Attention is not so much a spotlight but a filter – one that is adaptive and reflective of the operational context and source of signaling. Intelligence, in the form of an appropriate response (action) to a perceived and recognized stimulus, is highly dependent on how attention is directed (overtly or covertly, top-down or bottom-up, exogenously or endogenously), in particular when humans and machines coordinate work cooperatively.
Today’s Observability is much like the yesteryear web with a focus on content overproduction, and mass duplicated distribution with little due consideration given to semantics or significance, leading to information overload. Intelligence in the form of selective suggestions (attention), as opposed to rudimentary storing and searching, is all but absent in many of the niche players with no pedigree in application performance monitoring or distributed systems control. User attention is distracted by obese data dashboards devoid of signals and states. Users are also being encouraged to wander around aimlessly in an unknown data fog without much in the way of intelligent guided supervision and direction.
We need to prioritize attention over data as much as attention itself prioritizes signals over noise. We can’t simply focus on everything. We need to simplify every layer of the stack with attention playing an active role in all layers and stages in the transformation of sensory stimuli to signals and states that must be attended to. Intelligence is not data nor is it doing. Intelligence is adaptive and active attention followed by action. Today Instana has many features, such as dynamic focus, dynamic graph, and application perspectives, that go a long way to being a solution for tomorrow’s complex world. APM once again rises to the challenge in a new form.