One of the biggest challenges facing effective continuous delivery of new and updated software services is complexity – in the form of many interconnected and interdependent software components interacting across different spaces, times, and in turn contexts.
Simplicity via Abstraction
When faced with complexity, one that is growing, the typical engineering reaction is to strive for simplicity by way of abstraction upwards and outwards. But what exactly is abstraction? What is the right level of abstraction? How does abstraction function as a process rather than as an outcome as most see it? These are some of the questions related to abstraction that have yet to be fully resolved across many disciplines.
Abstraction is a critical component in knowledge acquisition and cogent reasoning. It is a bridge between perception and cognition and is everywhere in computing in its many forms. In many ways, abstraction is at the core of human intelligence that has still to be replicated over into machines as a process. Many consider abstraction to be an essential tool in managing complexity. That said, abstraction seems to defy a clear and concise definition being invariably entangled with generalization, categorization, and information reduction or hiding. Maybe abstraction is all of these, much like how observability as a concept has changed in scope and application.
Reduction and Compression
At Instana, there are two primary approaches to dealing with the complexity that our customers face. The first is reducing the information that needs to be processed by users in needing to instantly understand a space (spatial) or situation (temporal) under observation. Here we discard or push down details to other layers and navigations paths. The second approach is where information is condensed (or aggregated) – this can happen, if required, at various stages in the data pipeline, all the way from the process through to the storage and up into the visualization. Abstraction is central to observability from measurement through to model and the memory of such.
Boundaries of Significance
When considering abstraction, many imagine simpler, straightforward representations of physical and conceptual objects in terms of state (properties), structure (relationships), and signals (communication). The selection of what is significant is the predominate focus here. With microservices, highly interconnected interdependent networks, abstraction is a necessity. Without some degree of abstraction, it would be near impossible to comprehend such environments. Comprehension is only realistic with compression. The difficulty is choosing the appropriate level of abstraction (and detail) exposed across layers, levels, and views in which boundaries create more manageable and permanent aggregates. Below is one of the original diagrams used in a series of workshops before the inception of Instana in 2014 – abstraction was front and center and continues to be going forward in both short and long-term.
Abstraction is Attention
At this point in the Observability movement, abstraction seems to have taken a backseat to data collection. In dealing with significantly more uncertainty and complexity the tendency, and somewhat emotional reaction, has been to collect more and more of the same at even finer granularities and to offer up this raw data in the form of dashboards that act as magnifying lenses on vast amounts of pixel-level data. Lost in all of this is automatic extraction and representation of objects – now instead served up by way of an ever-growing list of tags.
A flatland overpopulated with now meaningless signposts has replaced the rugged and dancing landscape of a complicated and complex system. Individual pixels in a scene are named and tagged. There is hardly any feature extraction and, in turn, object or behavioral pattern recognition because the measurement approaches employed come from a period of far less complexity and change. Abstraction is not a loss; it is a gain of deeper understanding.
Abstraction is attention. In the domain of application performance monitoring and now observability, it becomes a process of selecting observables that bring attention to some critical aspects of the system. But if every aspect of a system is deemed significant, then simplicity flies out the window. Focusing on everything is a recipe for failure.
Signals over Noise
Much of the abstraction employed in the observability space today centers around the aggregation of metrics. But is this useful and sustainable in the long run? Can observability initiatives at companies offer real value to operations and development teams while continuing to experience reality at the pixel level?
The Observability, DevOps, and Site Reliability Engineering communities need to be extremely judicious in what is measured and how such things are named (and tagged). Data diversity, in practice more like data duplication, is fine up to a limit until there is nothing that can be decoded, delineated, discerned, and derived. Observability needs an over-arching steering process of simplification, synthesis, and significance otherwise it is likely to lead to waste.
At Instana, we are turning to another definition of abstraction to help us more effectively assist customers, and that is the reformulation of the problem – a fundamental change of representation, a revised model and method that uniquely addresses simplicity, significance, scale, and signaling. Not a complete disconnect from the data world that is still predominant in the IT landscape and the many layers that are relatively unchanging compared to the microservices. But something far more appropriate and applicable to complex conversational service-to-service interactions requiring contextualized and scaled status assessment. We are redrawing the landscape and designing new methods of observation and modeling that dramatically improve the operational effectiveness of teams in instantly making sense of ever-changing situations, structures, states, services, and systems.