Distributed clocks

On one machine, "what happened first?" is free — the CPU executes instructions in order and a single clock stamps them. The moment two machines are involved, that question stops having an easy answer, and most data-corruption bugs in distributed systems are someone assuming it still does.

Here's the trap in one line: a timestamp from node A and a timestamp from node B are not comparable. If A stamps an event 12:00:00.000 and B stamps one 12:00:00.300, you cannot conclude B's event happened later. Their wall clocks are independent, each drifting on its own NTP schedule, and 300ms is well inside the skew two cloud VMs routinely carry. You've compared two rulers that don't agree on where zero is.

Why wall clocks lie

Failure mode	What happens	How often
NTP drift	Clocks slowly diverge between syncs	Constant (<100 ms/day typical)
Leap second	Clock jumps 1 second back	Rare (~yearly)
VM pause	Process freezes; clock seems to jump forward on resume	Hourly under load / GC
Manual admin date -s	Anything	Once is enough to lose data
NTP source fails	Drift grows unbounded	Whenever sync breaks
Skew between AZs/regions	Different time servers, different drift	Routine 50–500 ms

The honest model is not "the clock is slightly off." It's "the clock can jump backwards, can stall, and can disagree with its neighbour by hundreds of milliseconds — at any time, without warning."

A war story: the NTP correction that ate writes

A team ran an active-active key-value store across two regions with last-write-wins conflict resolution: whichever write carried the higher wall-clock timestamp won. Simple, and it worked for months.

Then region A's NTP daemon, after a network blip, stepped the clock forward by ~4 seconds to catch up. For the next few seconds, every write originating in A carried timestamps from the future. A user in region B updated their profile; moments later (in real time) that record was silently overwritten by an older write from A that happened to carry a larger timestamp. Last-write-wins faithfully kept the "newer" write — newer by the clock, older in reality.

No error fired. The store did exactly what it was told. The fix wasn't a better NTP config; NTP corrections are normal. The fix was to stop trusting wall-clock order for causality and move to a clock that encodes happened-before directly.

Nearly every "I assumed timestamps would order my events, but…" outage is one of these clocks being wrong for the workload. The wall clock answers "roughly when," never "in what order." Pick the right clock before the incident, not during it.

The happened-before relation

The only ordering you can trust is causal. Event X happened-before event Y (written X → Y) if any of these hold:

• Same process, X executed before Y.
• X is a message send and Y is its matching receive.
• Transitive: X → Z and Z → Y.

If neither X → Y nor Y → X, the events are concurrent — nothing connects them causally, and any order you impose is arbitrary. Logical clocks exist to capture exactly this relation without trusting any wall clock.

The three answers

1. Lamport clocks — partial order, one integer

Each node holds a counter. Every local event increments it. Every message carries the sender's counter, and on receive the node does counter = max(local, received) + 1.

The guarantee: if X → Y then L(X) < L(Y). The catch is the converse fails — L(X) < L(Y) does not mean X caused Y; they might be concurrent and just happened to land on different counters. Lamport gives you a consistent total order to break ties, but it cannot tell you whether two events were actually related.

2. Vector clocks — they detect concurrency

Each node holds a vector of counters, one slot per node. You increment your own slot on each event; messages carry the whole vector; on receive you take the element-wise max, then bump your own slot.

Now comparison is rich: V₁ "dominates" V₂ if every slot is ≥ and at least one is strictly greater. If neither dominates the other, the events are concurrent — a genuine conflict. Worked example on three nodes:

A: [1,0,0]  local event on A
A → B msg, B receives:  B = max([0,0,0],[1,0,0]) then +own = [1,1,0]
C: [0,0,1]  local event on C, never talked to A or B

Compare B=[1,1,0] and C=[0,0,1]:
  neither dominates  ->  B and C are CONCURRENT (a conflict to resolve)
Compare A=[1,0,0] and B=[1,1,0]:
  B dominates A      ->  A happened-before B (no conflict)

That "concurrent" verdict is exactly what AP stores like DynamoDB and Riak need to surface a conflict to the application (or a CRDT) instead of silently dropping a write. The cost: the vector grows with the number of nodes, so it doesn't scale to thousands of ephemeral clients without pruning.

3. Hybrid logical clocks (HLC) and TrueTime

Logical clocks lose touch with real time — a Lamport value of 5000 tells you nothing about when. HLC fixes that by packing a wall-clock reading and a logical counter into one value: it tracks physical time when clocks agree, and falls back to incrementing the logical part when skew would otherwise reorder events. You get causality and a timestamp that's close to real time, in 8–16 bytes. CockroachDB and MongoDB run on HLC.

Google Spanner's TrueTime goes further by being honest about uncertainty. Instead of returning a point, it returns an interval — "now is somewhere between 12:00

.000 and 12:00.007." To commit a transaction with external (linearizable) ordering, Spanner waits out that interval — commit-wait — so that by the time it releases, every other node's clock is guaranteed past the commit timestamp. That uncertainty window is ~1–7ms in practice, backed by GPS and atomic clocks in every datacenter. The price of true global ordering is a few milliseconds of deliberate waiting on every commit — and hardware you can't replicate without Google's clock infrastructure.

Comparison

Clock	Captures causality	Detects concurrency	Size	Real systems
Wall clock	❌ (drifts, jumps)	❌	8 B	Almost everyone (often wrongly)
Lamport	✅	❌	8 B	Tie-breaking, academic
Vector clock	✅	✅	O(N) nodes	DynamoDB, Riak, CRDTs
HLC	✅ + ~real time	partial	8–16 B	CockroachDB, MongoDB
TrueTime	✅ + bounded real time	partial	16 B + commit-wait	Spanner (GPS/atomic clocks)

What to actually reach for

• Single-node append log: wall clock is fine — one clock, one order. Add a node ID as a tiebreaker if you'll ever merge logs.
• Two services writing the same record (AP): vector clocks, so you can detect the conflict instead of losing a write to last-write-wins.
• Cross-region transactions needing real ordering: HLC if you control the stack; TrueTime if you're on Spanner.
• You just want "roughly when it happened" for humans: wall clock, and never use it to decide causality.

[CONCEPT]consensus-raft solves a different problem — agreement on a single value — but uses logical clocks (election terms) internally. [CONCEPT]cap-theorem explains why this matters: CP systems serialize through consensus, AP systems lean on vector clocks to keep concurrent writes instead of dropping them.