In my last post, I discussed the data fog of observability. In this post, I would like to draw attention to one important aspect lost in the data fog that is not given enough attention by those observability vendors not coming from a monitoring background and that is system and subsystem boundaries.
Without boundaries, it is near impossible to recognize objects (or events) within our perception. Boundaries are how the structure of a complex system is derived or defined, and then tracked. When boundaries are not perceived directly, they tend then to manifest in models of a system. Application monitoring vendors know this all too well. In contrast, observability vendors up to this point have been more concerned with collecting data and piping it towards a data sink of records devoid of context such as boundaries. One reason for this I suspect is that much of the design of Open Source instrumentation libraries gives little consideration to this aspect of sensory perception – most libraries are very data (collection) centric with tags thrown in to alleviate modeling failings. Observability has all but flattened the richness (structure, state, and signals) of the computing world down into tables of records. The shapes and forms offered up in charts are just renderings of quantities, not objects of interest and significance. Much noise is made of multidimensional analysis yet recognition of contextual boundaries is mostly amiss.
In my post, The Abstraction of Observability, I discussed the importance of being able to reduce and compress measurements, which is much helped by representations extracted from the environment via some form of hierarchical boundary determination. When this is not done automatically, what happens then is that the custom dashboard capabilities of the observability solution need to be used to reconstruct some form of structure that mirrors the boundaries all but lost in the data fog. Naturally, this is extremely costly and inefficient for an organization. While it might be fun and creative for the first one or two dashboards, it does not scale and is extremely hard to sustain in practice outside of ad hoc use cases – this should be the exception, not the rule. Dashboards are not a realistic replacement for a model. Companies who go down this route will invariably end up with needing additional tooling to catalog and manage the proliferation of dashboards. Use wisely and sparingly.
Hierarchies are everywhere in computing, especially modular systems. Without boundaries, there can be no discernible structure and in turn hierarchies (or nested enclosures and scopes). Hierarchies and the layers they can imply are a necessity in meeting the challenges that come with complexity. Irrespective of the distribution of software, there is always a spatially defined hierarchy. But boundaries don’t just describe what constitutes an object of some permanence; they also demarcate events that in themselves can be composed or decomposed.
While a simplistic data sink table-like view is useful at the beginning of an Observability initiative, especially when painstakingly instrumenting code manually with open source libraries such as OpenTelemetry or the now deprecated OpenTracing and OpenCensus libraries, it is important to keep in mind that the effective perception and cognition of systems relies heavily on discerning structures and boundaries – both spatial and temporal.