With the term Observability taking on new meaning and application daily, it seems next to impossible to be able to compare projects, products, solutions, and offerings. One vendor will claim application performance monitoring is merely a pillar or subset of Observability. Another will counter that with it being a stepping stone to Controllability. Others that it is the new Testing or the basis for descriptive intentional application-level workflow. There is so much marketing noise; at times, it seems to come from a few who seem to follow an even smaller few who kind of stumbled along into this particular engineering space while shouting and swearing off at those already present. I don’t think it helps anyone to make a distinction based on the time of arrival to the party that is Observability as there is much overlap in the past and present within and across companies and products. We are all here to bear some influence on the future. It is here (or yonder) where I believe there is a clear distinction in the direction Observability will take over the coming year. Do we drown in data and detail or swim in semiotics and systems?
The most vocal voices in the Observability space, or at least those who get to shout the most from a conference podium, are those who seem solely preoccupied with data – dataism meets atomism. This vendor group reduces all instrumentation and sensory experience down to flat and fat events with hundreds of tags, labels, baggage items, and attributes (or is it properties these days…who knows…it is data anyway) – all dressed up as high dimensionality. This is the ultimate in the democracy of data – all data points and values are equal. No data item is irrelevant and none more significant than another. Just like the voices, data collected screams out for your attention; it is you who must decide whether to attend to or ignore by way of databoards (dashboards). The noise dwarfs all signals. Vendors in this group are either glorified time-series database stores in the cloud or search engines sitting atop a large data sink of semi-structured log or metric-based data – all with dashboarding, ad hoc querying, and reporting capabilities. Data is king, and a process, such as monitoring, is just another transformation applied to data itself.
The second Observability vendor group, which I like to believe Instana leads, comes from a systems perspective. This group does not entirely discount data(ism); instead, it gives greater importance and prospective focus to the process of understanding and steering of complex systems crossing and encompassing both machine and human organization. Data is a means to a much higher order ends. This group gives prominence to situations over events with all cognition, whether internalized within a machine or human, being situated and contextualized. Entities and classifications supplant tags and records. Knowledge, understanding, and prediction are formed and reinforced by a conversation between users, solution, and embedded agents (of introspection and intervention). Learning is a collective and continuous process. The dynamics of a system are probed and prodded, instead of data just being painted and presented. Data, information, and action are all framed within multiple and layered scopes and boundaries of state inference. An overriding goal is to self-regulate change and the rate of which it accelerates or otherwise – to bring back some sense of control (guiding/steering) within an era of growing complexity and change.
The information technology landscape is experiencing a moment of unprecedented levels of micronization and modularization at all layers in the stack and along many stages in the service supply pipeline. The question for companies in choosing an Observability path is whether the resulting complications need to be tackled in the same manner and thinking as the data-driven crowd offers or whether communication and control wins the day bringing back delight and wonder to systems engineering on a grand scale beyond our individual capabilities and capacities.