This is the third post in a series of posts alphabetically listing and defining important Observability terms. If you have just jumped in then please consider starting from the beginning to get a better understanding of my selection.
Today’s Observability is all about instruments – software, and occasionally hardware, devices used to measure, collect, and record events (actions, outcomes, and state changes) and their associated properties and enclosing contexts. Instrumentation is mostly the manual or dynamic injection of instruments into the flow of code and communications of calls across thread, process, and machine boundaries. Instruments respond to event callbacks, much like the tripping of a wire, where the surface on which the wire rests is in source or binary form. Instruments allow us to see inside a computational process. Unfortunately, many of the instruments employed come from a past of little complexity and change, and do not hold up well to challenges faced by developers and operational staff. Today’s instrumentation toolkits still require developers to hardcode the type of instrument to be used at interception points along with the model (trace, metric, log, event) collected and transmitted, severely limiting the ability to adaptively dial-up and dial-down workflow introspection and state capture. In addition, very little due consideration is given to the value, the significance, of measurements. All too often, instruments are manually added because it is so cheap to do so at source, with little regard for cost elsewhere. Instruments must evolve.
Alternatives: Introspection, Instrumentation, Injection, Interceptor, Incident, Isolation, Interface
It is essential for successful monitoring and management of systems (of services) that behavioral signals and qualitative states are delivered and derived just in time for an operator, human or machine, to reason and respond appropriately where intervention or further introspection is deemed necessary. Timeliness will become increasingly important as the focus shifts from passively observing and querying data to automatically and adaptively reacting. Controllability is the future of Observability, when will it arrive is still mostly an open question for the industry.
Knowledge is a precious stock to be carefully managed – created, read, updated, and deleted when invalidated by change. It does not exist without a knower, such as a collective of humans and machines. Knowledge is the link between a knower and fact, a subject and truth. With regard to Observability, knowledge is when sensory data, transformed into information, is accepted as being a reliable observation or inference of the state of affairs. Being able to explain an observation (operation or outcome) or the current situation by way of prior knowledge and learning is crucial for effective monitoring and management of systems involving multiple parties. Perception, patterns, and predictions need to be explained in terms of facts. Facts need to be framed within composite contextual bounds – both spatial and temporal. Knowledge acquisition is not only a passive process of accepting facts by way of feeds. It can be constructed and generated with experimentation – the deliberate probing and prodding of the system, such as the tweaking of flow routes and resource capacities. Knowledge is a guide as well as an anchoring causal chain. It is important not to confuse a communicated observation (or testimony) with knowledge or fact in the truth sense. A challenge for tooling in a world of ever-increasing complexity and higher rates of change is contextualizing knowledge – the infamous “it depends on” attached to many propositions.
Alternatives: Kubernetes, Knob
Learning is the acquisition of information from experience or communication and the subsequent change of behavior or improved understanding of a subject – this must be one of the primary goals of any organizational Observability effort. Both humans and machines must learn in unison from the phenomena sensed, transmitted, and recorded, and more importantly, during the period of converse in the performance of tasks involving interactions with data, inferencing of states, and detection of patterns of behavior. There is far more insightful information to be acquired, as well as behavioral changes to adopt, far above the simple task of querying of sensory data. Humans and machines need to learn to understand and appreciate how the other perceives and reason about what is observed. Learning is continuous, and the feedback loop between humans and (software) machines must be extended beyond the parts of the system under observation to encompass more of the operational objectives and constraints each aligns work with. Both parties need to complement and amplify the strengths of the other. Training a machine should not stop with feeding sensory data into a data sink for post-processing. The semantics required for deeper perception and reflection must be an ongoing process of dialog, development, and deployment.
Alternatives: Layer, Linking