System Design
Defense in Depth
Why one firewall is never enough. Layered security — each layer assumes the others failed.
Defense in depth
In 2013, attackers walked into Target's network through an HVAC contractor. The contractor had remote access for monitoring refrigeration units. That access sat on the same flat network as the point-of-sale systems. Once inside, the attackers pivoted from a thermostat vendor's stolen password all the way to 40 million credit card numbers — because nothing between the HVAC login and the card data assumed the HVAC login might be the attacker.
That's the whole subject in one sentence. Every individual security control eventually fails. A WAF rule misses a novel payload. A library ships a 0-day. A credential leaks in a screenshot pasted into Slack. A pull request adds a regression nobody caught. The only system that survives a single failure is one where each layer assumes the layers in front of it have already been breached — and does its own job anyway.
The opposite of this is perimeter security: one strong wall, soft everything inside. It worked when "inside the network" meant "inside the building." It stopped working the moment your services ran across three clouds, your engineers SSH'd in from coffee shops, and your build pulled code from npm at 2am.
Perimeter security is one strong door with nothing behind it. Defense in depth is many small doors, each of which assumes the one in front has already been kicked in.
The layers
A modern production request crosses five to seven independent controls before it touches state. Each has a narrow job and zero trust in the layers ahead of it. The key column is the last one — not what the layer does, but what catches the attacker when this layer fails.
| Layer | Job | What fails | What catches it next |
|---|---|---|---|
| Edge (WAF/CDN) | Block obvious junk, absorb floods | Novel payload slips through | TLS + per-IP rate limit |
| Transport (TLS) | Encrypt + verify server identity | Cert mis-issuance | Pinning, mTLS |
| AuthN | Prove who the caller is | Stolen token | Short TTL, MFA, refresh rotation |
| AuthZ | Allow this action? | Logic bug, missing check | Row-level filter, deny-by-default |
| Mesh (mTLS) | Verify service-to-service identity | Compromised pod | Signed workloads, SPIFFE IDs |
| Storage RBAC | Limit DB blast radius | DB credential leak | Encrypted at rest, scoped grants |
| Audit | Detect after the fact | Late detection | Anomaly alerts on the log |
Read it as a chain of "even if." Even if the WAF misses it, TLS still encrypts it. Even if the token is stolen, AuthZ still checks the action. Even if the service is owned, the DB user it talks through can only read one tenant's rows. No single line in that table saves you. The composition does.
Why one strong wall is not enough
The seductive thing about a perimeter is that it's measurable. One firewall, one ruleset, one thing to audit. Defense in depth is messier: more certs to rotate, more grants to review, more logs to retain. Teams reach for the single wall because it's cheaper to operate, and it works right up until the first pivot.
There is no perimeter when:
- Services live across regions and clouds — "internal traffic" crosses the public internet.
- A vendor library pulls transitive dependencies from npm during the build, executing arbitrary install scripts on your CI runner.
- A vulnerability in one container's libc lets an attacker break out and reach every pod on the node.
- An engineer's laptop, with a cached
kubeconfig, gets stolen at a conference.
Each of these is inside the wall by the time it matters. The wall never sees them.
A war story: the flat network and the $292M cleanup
Target's breach is the canonical version, so use it as the worked example. Reconstructing the public post-mortems, the chain went roughly:
- Phishing email lands a credential-stealer on Fazio Mechanical, the HVAC contractor.
- Stolen vendor credentials log into Target's external vendor portal — AuthN succeeds, because the credentials are real.
- From the portal, the attackers reach the internal network. There is no segmentation between the vendor zone and the payment zone.
- They push memory-scraping malware onto point-of-sale terminals, which read card numbers from RAM in the half-second between swipe and encryption.
- Card data is exfiltrated to an internal staging server, then out.
Count the layers that should have existed and didn't. The vendor portal should never have had a route to the POS network — network segmentation is the missing AuthZ-at-the-network-level layer. Service-to-service calls should have required identity the stolen vendor password couldn't supply — mTLS / least privilege. Outbound traffic to an unknown staging host should have tripped an alert — egress monitoring / audit. Each missing layer turned a contractor's bad day into a national headline.
The numbers: ~40 million card numbers and ~70 million records of personal data. Target's own filings put the gross breach costs around $292 million before insurance, the CEO and CIO both lost their jobs, and the settlement with banks and states ran for years. The root cause wasn't a clever zero-day. It was a flat network — one wall, nothing behind it.
The attackers never broke encryption, never cracked a cipher, never found a memory-corruption exploit in the POS firmware. They walked through an HVAC vendor's front door and found every other door propped open. Every control they bypassed was one that didn't exist, not one that failed.
What "breached" actually buys an attacker
The useful mental exercise is to assume each layer is gone and ask: what does that get them? In a flat system, breaching the edge gets them everything. In a layered system, each breach buys one small thing.
| Compromised | Flat network outcome | Layered outcome |
|---|---|---|
| Edge/WAF | Direct path to every backend | Still need a valid TLS session + token |
| One service token | Acts as any user | Acts as that user only; AuthZ still checks each action |
| One pod (RCE) | Pivot to all pods on the node | mTLS identity is pod-scoped; can't impersonate others |
| DB credential | Read/dump the whole database | Read only the rows the grant allows |
| Backup file | Plaintext dump | Encrypted at rest; useless without the key |
The right-hand column is the entire product of doing this work. You will never stop all breaches. You can make every individual breach worth almost nothing.
The failure modes of defense in depth itself
Layering has its own ways to go wrong. Three that bite real teams:
Theatre instead of depth. Seven layers that all check the same thing aren't depth — they're one control wearing seven hats. If your WAF, your gateway, and your service all validate the same JWT signature and nothing checks authorization, you have one layer and six speed bumps. Depth means each layer catches a different failure class.
The "even if" you never tested. A team writes "even if the token is stolen, the DB user can only read one row" in a design doc, ships it, and never verifies the grant. Two years later an audit finds the service runs as a superuser because a migration needed it once and nobody walked it back. The backup layer was fiction.
Cost spiral and alert fatigue. Every layer adds operational weight: cert rotation that pages at 3am, RBAC reviews nobody has time for, audit logs so noisy the one real anomaly drowns. Past a point, adding a layer makes you less safe because it starves the layers that matter. The correct amount of depth is "one more layer than the attacker is willing to peel back" — banks have many, a static marketing site has almost none.
Someone disables an "annoying" layer to ship a hotfix — turns off mTLS enforcement, or grants the service broad DB access "temporarily." The flag never gets flipped back. Months later that exact layer is the one that would have contained an incident, and it's been off the whole time. Layers you don't verify in CI are layers you don't actually have.
How much depth is enough? A back-of-envelope
You can reason about layers probabilistically. Suppose each independent control has a 10% chance of failing to stop a given attacker on its own — a deliberately pessimistic 0.1. If the layers are genuinely independent, the attacker must beat all of them, and the probabilities multiply.
| Layers | Chance attacker beats all | Interpretation |
|---|---|---|
| 1 | 0.1 (1 in 10) | A single bad ruleset = breach |
| 2 | 0.01 (1 in 100) | Already an order of magnitude harder |
| 3 | 0.001 (1 in 1,000) | Diminishing-but-real returns |
| 4 | 0.0001 (1 in 10,000) | Most attackers move to an easier target |
The catch hides in the word independent. If two layers share a root — same SSO provider, same secret in the same vault, same libc — they fail together, and the math collapses back toward a single layer. Target's "layers" weren't independent; they were one network. The whole game is buying layers that fail for different reasons.
When you've actually achieved it
You have defense in depth when you can answer "what happens if X is compromised?" for every X in your system — and the answer is always "the attacker still can't reach Y, because of Z."
A team that says "we use TLS" has one control. A team that says "we use TLS, and even if TLS were broken the mesh signs every call, and even if the mesh were broken the DB user can only read one tenant's rows, and even if that user dumped everything the data is encrypted at rest and the key lives in a separate vault" has defense in depth — and, just as important, has named the chain, which means they can test each link.
The cheapest layer to add first is usually at the edge: [CONCEPT]rate-limiting costs almost nothing and blunts both brute-force and volumetric attacks. The most overlooked one is [CONCEPT]idempotency — when a security event forces a client to retry a dropped request, idempotency is what keeps that retry from double-charging a card or double-applying an action.