CDN design: routing, cache correctness, and failure boundaries

This design separates four concerns that are often collapsed into one box: DNS answers, anycast routing, network-layer protection, and HTTP caching. A DNS answer supplies an address. If that address is anycast, BGP selects a site according to routing policy; it is not a per-packet latency oracle and does not guarantee the geographically closest or lowest-RTT PoP. Route convergence and withdrawal are operational processes, not an instant failover primitive.

Request path and trust boundary

1. The client resolves the hostname, then connects to the advertised service address.
2. The anycast ingress and L3/L4 shield handle packets and connections. HTTP policy is applied only after TLS termination.
3. The WAF admits the request to the HTTP cache. The cache key starts with method and target URI and includes every configured Vary dimension. Authorization, cookies, locale, compression, and device variants must not be accidentally coalesced.
4. Only a miss or required revalidation goes through edge policy and a per-key request-collapsing gate to the private origin.
5. The origin is restricted to authenticated CDN traffic. A public origin address that bypasses the CDN invalidates the protection model.

Responses containing private or per-user data are not shared unless an explicit, reviewed policy makes that safe. Cache-Control: private, no-store, authorization rules, and Vary are correctness controls, not tuning hints. Conditional validation with ETag or Last-Modified can turn an expired entry into a small 304 exchange, but it still reaches origin.

Capacity model

Let peak demand be 1,000,000 requests/s, mean public response size be 200 decimal kB, and reusable-cache hit ratio be 95%.

• End-user payload rate is approximately 1,000,000 * 200,000 * 8 = 1.6 Tb/s before protocol overhead.
• Requests that need an origin decision are 1,000,000 * (1 - 0.95) = 50,000 requests/s before conditional validation and request collapse.
• Their upper-bound payload rate is 50,000 * 200,000 * 8 = 80 Gb/s if every miss returns a full body.

The origin load is not zero. Hit ratio is workload- and cache-key-specific; personalized traffic, one-hit objects, churn, purges, and revalidation can dominate. A global percentage also hides hot PoPs and hot keys, so capacity planning uses per-PoP and per-key distributions plus origin shielding.

Concurrency and freshness

Concurrent misses for the same cache key are collapsed to one fill. Collapse scope must include all variant dimensions, otherwise one representation can be delivered to another caller. The fill leader is not a lock on business state: the origin still needs its own concurrency controls.

Freshness follows HTTP cache semantics. stale-while-revalidate and stale-if-error are bounded permissions, not permission to serve arbitrary old data. Security-sensitive or personalized responses fail closed when validation cannot complete. Immutable, content-addressed asset URLs reduce dependence on global purge completion; HTML and other mutable indexes keep shorter, explicitly chosen freshness windows.

Purge events are authenticated, idempotent, sequenced, and observable. Delivery can be partial, so the control plane tracks acknowledgements and lag. Versioned URLs make a delayed purge a bounded cleanup problem instead of a correctness dependency.

Failure and abuse scenarios

• Origin timeout: reuse stale content only inside the response's allowed stale window; otherwise return an error rather than leak or invent data.
• L3/L4 flood: distributed network filtering and capacity act before TLS. A SYN or packet flood is not an HTTP WAF event.
• L7 abuse: after TLS termination, WAF and rate limits can use HTTP identity and request cost.
• TLS 1.3 0-RTT: early data is replayable. Unsafe or non-idempotent operations such as checkout are rejected as early data and sent only after the handshake. Even an idempotent endpoint still needs an explicit replay policy.
• PoP or route failure: routing reconverges according to the network; applications must tolerate connection loss and retry safely. No fixed sub-second withdrawal or latency improvement is assumed.

Decisions visible in the animation

The normal miss is not rendered as an error. Only actual timeout, partial control-plane delivery, abusive traffic rejection, and unsafe early-data rejection use failure styling. Responses traverse the same physical connections in reverse; there are no synthetic reverse edges or origin-bypass shortcuts.

Related material

DNS resolution is an explore diagram, so it is linked as a normal viewer route rather than a course directive.

Primary sources

• RFC 9111: HTTP Caching — cache keys, Vary, authorization, freshness, validation, and stale response rules.
• RFC 8446: TLS 1.3 — 0-RTT replay properties and application obligations.
• RFC 4786: Operation of Anycast Services — routing selection, load distribution limits, and route stability.
• Cloudflare cache documentation — operational cache controls and tiered cache behavior.
• Cloudflare purge documentation — supported invalidation mechanisms and operational trade-offs.
• Cloudflare DDoS protection overview — distinct network and application-layer protections.
• Amazon CloudFront request and response behavior for custom origins — request collapsing and origin behavior.
• Amazon CloudFront restrict access to origins — preventing direct origin bypass.

Interactive scenarios