Observability

The first time I got paged at 3am for a production system I didn't write, I learned what observability actually is. It's not the dashboards. It's whether, at the moment your phone goes off, the system can answer the question "what is broken, where, and why?" in the next ten minutes — or whether you have to SSH into machines and read application logs by hand while the customer-facing graph keeps falling.

That distinction — between monitoring (alerting on known failure modes) and observability (the system can answer questions you didn't know to ask) — is the real subject of this chapter. Monitoring tells you the disk is full. Observability tells you which deploy caused checkout latency to double last Tuesday and which downstream service it correlates with.

The three pillars, and what each one is bad at

The textbook says observability has three pillars: logs, metrics, traces. The textbook is correct and underestimates how much of the job is choosing which pillar to reach for in each situation. They're not interchangeable; each one is great at one thing and embarrassing at the others.

Logs are text the application writes when something noteworthy happens. They're high-cardinality (each line can carry arbitrary context: user_id, request_id, error message), cheap to emit, and expensive to query at scale. Use them when you need to understand what specifically happened in one request. The failure mode is volume — a chatty app at 10K RPS produces hundreds of GB of logs per day and grep takes minutes. Most production teams sample or structured-log to keep this in check. Loki, Splunk, Elasticsearch, CloudWatch Logs are the usual storage backends.

Metrics are numbers, aggregated over time, with low cardinality labels. http_request_duration_seconds{method="GET",route="/checkout",status="200"} is one metric. You don't get to ask "which user saw the slow request" — that's a logs question — but you do get cheap dashboards and fast alerts. The failure mode is cardinality explosion: add a label with high cardinality (user_id, request_id) and your metrics database melts. Prometheus, Datadog, and CloudWatch Metrics are typical.

Traces show the path a single request took through your system, with timing for each step. GET /checkout → auth.verify (4ms) → cart.load (18ms) → payment.charge (340ms) → order.save (8ms). Without traces, in a microservice system you cannot answer "why was that request slow" — the latency is hiding in some span you can't see from any one service's logs. The failure mode is volume: tracing every request at 10K RPS is too expensive, so you sample (1%, 0.1%) and live with the fact that most individual requests are not in your trace store. Tempo, Jaeger, Zipkin, Honeycomb, Datadog APM.

The three are designed to compose. A typical investigation starts with a metric (latency p99 spiked), drills to a trace (this one request shows the slow span), and ends in logs (the slow span's log line tells you exactly what went wrong).

A metric without a trace is "something is broken". A trace without a log is "the slow part lives here". Logs without metrics is grep at 3am. You want all three, wired so each one is one click from the next.

SLO, SLI, SLA — the language of "good enough"

Observability without an SLO is a hobby. The SLO (Service Level Objective) is the number that tells you whether the system is currently good enough. Everything else — dashboards, alerts, postmortems — is in service of moving the SLO in the right direction or noticing when it isn't.

SLI (Service Level Indicator) — the measurement. "Fraction of checkout requests that complete in under 500ms and return 2xx."

SLO (Service Level Objective) — the target. "99.9% of checkout requests complete in under 500ms and return 2xx, measured over a rolling 28-day window."

SLA (Service Level Agreement) — the contractual promise to a customer, usually weaker than your SLO so you have headroom. "99.5% uptime, refund if we miss it."

The trick is to alert not on raw SLI breaches but on error budget burn rate. If your SLO is 99.9% over 28 days, you have 0.1% of budget — about 40 minutes of downtime — to spend in that window. If you're burning the entire budget in 1 hour, page immediately. If you're burning it over 6 hours, page during business hours. If you're burning it slowly, file a ticket. This is Google SRE's burn-rate alert pattern and it has replaced most of the brittle threshold- based alerts every team starts with.

The error budget, in actual minutes

The reason burn-rate alerts beat raw thresholds is that they're denominated in a budget you can count down. Here's what each "nine" buys you over a 28-day window — worth memorizing, because product managers always want one more nine without knowing what they're asking for.

The jump from 99.9% to 99.99% looks like one decimal place. It is the difference between "a human can wake up, read the page, and respond" and "you need automated rollback and zero-downtime deploys or you will miss it." Each nine roughly 10×'s the engineering cost. Burn rate turns this table into alert thresholds: a "2× burn" means you'll exhaust the whole window's budget in 14 days instead of 28 — fast enough to ticket, too slow to page.

The four golden signals

Most metrics dashboards are noise. The four signals that actually matter for a serving system are small and you can fit them on one panel:

• Latency — how long a request takes. Distinguish successful latency from failed latency; a 500 returned in 2ms is not "fast".
• Traffic — requests per second. The base unit for everything else; if traffic is unusual, that explains a lot.
• Errors — rate of failed requests. Define "failed" precisely; 4xx is the user's fault, 5xx is yours, partial successes are judgment calls.
• Saturation — how full is the resource that will run out first. CPU, memory, disk, connection pool. Saturation predicts the latency spike before it happens.

For batch and queueing systems the signals shift: latency becomes queue depth and time-in-queue, errors becomes DLQ rate. The shape stays the same.

High cardinality is the modern fight

Old-style metrics ("CPU utilization on host db-7") were low- cardinality and easy. Modern services want per-tenant, per-route, per-user metrics, and you cannot put those in Prometheus without melting it (each unique label combination is a separate time series). The industry has split into two camps:

• Sample-based observability (Honeycomb, OpenTelemetry tracing, ClickHouse-backed APM) — store every event with all its attributes, query at read time. Expensive storage, infinite cardinality, slow scan-based queries. Great for "show me all requests by user X with latency > 2s last hour".
• Pre-aggregated metrics (Prometheus, StatsD) — decide labels at write time, aggregate by them, cheap storage, fast query, capped cardinality. Great for "p99 latency by route over the last 24h".

Production teams usually run both — Prometheus for the cheap fast dashboards and alerts, an event-store for the deep investigations.

Sampling and cost

Tracing 100% of requests at meaningful scale is rarely affordable. The standard pattern is head-based sampling at ~1% — decide at the trace's first span whether to keep it, and that decision propagates downstream. The problem is you miss the rare failure traces, which are the ones you most needed.

Tail-based sampling fixes this by buffering all spans for a trace and deciding to keep it after the fact — keep all errors, keep all latency outliers, sample the boring ones. Costs more (you buffer every trace temporarily) but the kept set is dramatically more useful. OpenTelemetry's tail sampler and most modern APMs do this.

What the bill actually looks like

Sampling isn't a nice-to-have; it's the difference between a line item and a budget meeting. Run the back-of-envelope for a service at 10K RPS that emits one trace per request, each trace ~3 KB serialized once spans and attributes are flattened:

The 100× drop from 2.6 TB to 26 GB is the whole reason head sampling exists — at vendor ingest prices (commonly $0.10–$1.50 per GB), full fidelity is a five-to-six-figure monthly line and 1% is a rounding error. Tail sampling buys back the outlier coverage you actually needed for ~20% more bytes than plain 1% head, because the rare slow and failed traces are a tiny fraction of total volume. The trap teams hit: they pick 1% head to save money, then during an incident discover the failing requests they need to debug were in the 99% they threw away.

What good looks like in production

A senior team's observability story is depressingly mundane: pick one Grafana dashboard for each service that shows the four golden signals. Wire alerts off the burn rate, not raw thresholds. Make every log line, metric, and span carry the same trace_id so the three pillars compose. Define SLOs with the team that owns the service, not the SRE team — the owners decide what "good enough" means.

Run a game day every quarter where you deliberately break something and time how long it takes to find. If "find" is over ten minutes the observability story has a hole, and the hole is almost always in traces — somebody added a service two months ago and forgot to propagate context.

When to walk away

• Single-process scripts that run for seconds — print to stderr, exit, move on. You don't need OpenTelemetry to debug a cron job.
• Truly low-traffic systems — keep all the logs, skip the metrics pipeline, you'll find issues by reading.
• Compliance-only observability — if the goal is audit trail, not debugging, you want a different shape: signed append-only logs, not Prometheus.

[CONCEPT]circuit-breaker and [CONCEPT]rate-limiting are what observability tells you to put in once you can see the failure shape. [CONCEPT]latency-numbers grounds what your SLO targets should look like.