Introduction to Modern Observability

Abstract

When we are building software and services, we are building something for others. This necessitates creating and expressing (shared) understanding about what we are trying to. This includes anything and everything from the purpose of the service to our specific work aimed at expressing this understanding – which all work is. For example, when we write a simple Create-Read-Update-Delete request handler logic, this is part of expressing understanding about the service itself. Understanding is often expressed through many abstraction layers. It can be testing, documentation, and the code itself and so on. This paper describes a particular form of abstraction for expressing understanding where we observe the software and its users from the outside in – service acts as a black box with given user inputs and outputs generated by what is called “observability”.

The topic is motivated by the idea that most complete understanding of the software should be on the most curious person. Observability should allow for the most curious persons to dive deep into the service, asking arbitrary questions from it, questions it was not designed to answer explicitly. Software developers desire to understand in technical shapes and colors. For example, asking such questions from a given service and data like “how many items do the 99th percentile users usually have in their cart on average?”, or “how do cart sizes correlate with page load times?” to generate understanding about the most important customers for the service, and from this technical angle, observability is approached in this study.

Before we can start interrogating systems, we have to fit them for this new way of looking at them. This requires new approaches in the context of distributed systems - any software that cannot reason about asynchronicity and timing assumptions turns into a distributed system. These systems are usually modeled as event-based systems, where each interaction creates an event that is sent to downstream services for further processing.

This paper applies this event-based design to logging, and models logging as merely another event in this system - an expression of an atomic unit of work just like any other. We first build a context between monoliths and distributed systems, and then sketch a model of traditional and modern observability fundamentals. To highlight the differences, we proceed to build an example over this theory of a practical issue (logging) through a on-hands comparison of string and custom data structure based logging, in order to show the benefits of aligning the logging type with the event based model.