Observability is a hot topic in the industry at the moment. Looking at various blog postings and conference tracks, it would seem that observability has superseded monitoring. The shift in attention is unfortunate as observability in its multiple guises including tracing, metrics, and logging, is just the first step along a road to more effective application monitoring. The far more difficult challenge is making sense of the multisensory data that observability technologies capture and send over different channels with varying time scales and rates of change, as well as the multiple means of identity and classification of those elements and types observed within a system.
At Instana most of the complex engineering effort ongoing is devoted to solving this difficult problem – delivering a model of a system that is consistent and coherent at different points in time. The challenge is not unique to application performance monitoring. The human brain has had to evolve ways to resolve the problem of multisensory integration, also known as multimodal integration, where information from the different sensory modalities, such as sight, sound, touch, smell, self-motion, and taste, is fused together to facilitate the construction of a valid and useful mental representation of objects of concern within an environment. During this integration, the brain much like our solution has to decide how best to integrate and segregate structural and behavioral groupings. Both need to determine whether there is a sufficient degree of permanence for a source of a signal or stimulus to warrant an entity representation as distinct from a change between two observation points. It is a question of scale, but not just of the system under observation and control but the observer and the tooling employed in becoming aware of and fulling understand dynamics and behaviors.
The problem for application monitoring is not just one of the many different sensory technologies applied. Even with the same observability technology, such as distributed tracing, there are hurdles to overcome in making sense of trace span data that can arrive out of order or partial (incomplete) across different execution nodes (microservices). It is impractical, if not impossible, to expect change, occurring in many locations and measured in various ways from diverse instrumentation points, and on multiple layers of a stack, can somehow be magically snapshotted in an instant to capture a corresponding consolidated model.
Application monitoring much like newer enterprise-scale systems must deal with temporal inconsistency and employ various reconciliation techniques to insulate, to some degree, the measurement gap and opaque windows of state transitioning that exist within multiple spaces and time. The reality that is constructed by the mind is an illusion, but it is an effective one for our survivability. An overriding objective for engineering at Instana is to search for, discover, and construct novel models and modes of perception that serve to more effectively understand and manage systems by directing attention and targeting operational (re)action – survivability being the availability and reliability of the utility our customers deliver to their customers.
One could argue that application monitoring solutions have had to deal with multiple data collection technologies for a long time already – logging, tracing, and metrics are nothing new. What is new and making engineering life far more challenging as well as exciting for all are some key industry trends – micronization of systems, components, and interactions as well as the increasing rate of change of such. Things that concern IT operations are becoming increasingly smaller and in turn much more prolific. The scope of a change to a thing of concern is even shrinking. Coupled with this miniaturization and decentralization is the shortening of the life-cycle for such things.
Change is the nature of our physical existence, but humans and the software machines they have created has brought forward a whole new plane of virtual existence where we are far more consciously aware of the ongoing process of change that is now much more explicitly directed and accelerated.
There is an ever-increasing awareness that much of what we acknowledge, attend to, and address within and across systems and domains is fleeting. Impermanence is now pervasive in the design of systems, services, workflows, stores, and flows, yet the need to sense and identify remains.
The application monitoring industry is heading towards sensory overload, reduced capability, and impaired cognition unless we develop new models and processes that push down what was before manageable and reclassify it as low-level. At Instana we are seeking to redefine a new level of sensing and consciousness for developers and IT operations staff that surfaces only those appropriate signals, states, and actions most effective at this new scale and in this era of rapid cycles of change. The future of monitoring is fusion, focus, and foresight – not data!