distributed-file-system.js·5 scenarios
Distributed File System case study covering HDFS, GFS, and Ceph patterns: master/metadata server (NameNode HA via QJM + ZooKeeper) plus DataNode tier with rack-aware 3x replication. POSIX-like file API contrasted with object storage. Block-based storage (128MB), pipelined writes, streaming reads, append-only semantics. Five scenarios: pipelined chunk write to 3 replicas, parallel multi-block read with Master out of data path, append to existing file, DataNode failure with background re-replication, Active NameNode crash with Standby promotion via fence tokens. Includes 2 ADRs (single master vs sharded vs masterless Ceph CRUSH; 3x replication vs Reed-Solomon EC) and capacity hints for all nodes.
Distributed file system is the storage case behind big-data platforms such as GFS, HDFS, and many analytics clusters. The system is optimized for petabytes of data, large files, streaming reads, append-heavy writes, high throughput, and frequent hardware failure. It is not a normal POSIX filesystem stretched over the network, and it is not exactly object storage either.
A realistic design target is 100 PB logical data, 3x replication, thousands of commodity nodes, hundreds of millions of chunks or blocks, aggregate read throughput in hundreds of GB/s, and several node or disk failures per day. The platform must keep data available while machines fail continuously. This case trains ::concept{slug="replication"}, ::concept{slug="consensus-overview"}, ::concept{slug="sharding"}, rack-aware placement, metadata management, leases, edit logs, and recovery.
The key lesson is that metadata and data have different paths. Clients ask the master where chunks live, but they stream actual bytes directly from DataNodes. If the master is in the data path, the design cannot reach cluster throughput.
Split the file system into metadata plane and data plane. The metadata plane stores namespace tree, file-to-chunk mapping, chunk-to-replica mapping, permissions, leases, versions, and recent operations. In HDFS/GFS-like designs this metadata lives in the master’s memory and is persisted through checkpoints and edit logs.
The data plane stores large chunks on DataNodes or chunkservers. A file is divided into blocks such as 64 MB or 128 MB. Each block has replicas, often three, placed across racks. Clients read from the nearest healthy replica. Writes are pipelined: the client sends data to the first DataNode, it forwards to the second, the second forwards to the third, and acknowledgements travel back.
The system assumes failures are normal. Heartbeats and block reports tell the master which DataNodes are alive and what they store. When a node disappears, the master marks its blocks under-replicated and schedules background copy from existing replicas to new targets.
The diagram shows clients such as Spark readers and Flink writers talking to an Active NameNode. A Standby NameNode and JournalNodes provide high availability. JournalNodes store a replicated edit log so the standby can catch up and promote if the active master fails. ZooKeeper or a similar coordination service can hold failover state and fencing tokens.
The DataNode tier contains many storage servers. Only a few are drawn, but the scale assumption is thousands of boxes with many disks. Edges from clients to DataNodes represent direct data transfer. Edges from DataNodes to the master represent heartbeats and block reports. Edges through the pipeline represent write replication.
The diagram also contrasts design families. A single metadata master is simple and fast for large-file analytics but limited by metadata memory and failover complexity. Sharded metadata or federation can scale namespaces. Masterless placement, as in Ceph-style CRUSH, removes the central placement table but changes the operational and semantic model.
Pipelined chunk write teaches the write path. The writer asks the master to create a file and gets a lease plus replica targets. It streams a 128 MB block to DataNode 1, which forwards to DataNode 2, which forwards to DataNode 3. When all replicas acknowledge, the client can treat the block as durable and the master records metadata.
Streaming read teaches that the master is not a byte proxy. The reader asks for chunk locations, then reads large blocks directly from DataNodes, ideally local or rack-near. Large scans can prefetch and parallelize across chunks. This is why HDFS works well for Spark and MapReduce style workloads.
Append scenario teaches limited mutation semantics. Distributed file systems like this usually support create, append, and delete more naturally than random writes. Appends use leases or a primary replica to serialize the end of file. Random small writes would create too much coordination and checksum complexity.
DataNode failure teaches self-healing. The master misses heartbeats, marks the node dead, finds blocks that lost replicas, and schedules background re-replication. Recovery must be throttled so it does not steal all bandwidth from foreground reads.
Master failover teaches metadata durability. The active master writes edit-log operations to a quorum of JournalNodes. The standby reads the log, catches up, fences the old active, and begins serving. Without fencing, split brain could corrupt namespace metadata.
Single master vs sharded master vs masterless placement is the central architecture trade-off. A single master gives simple consistency and fast in-memory decisions, but metadata memory limits the number of files and chunks. Sharded masters scale metadata but make cross-namespace operations harder. Masterless systems reduce central bottlenecks but push placement and recovery logic into clients and cluster maps.
Three-way replication vs erasure coding is another major trade-off. Replication is simple, fast to read, fast to repair for small failures, and expensive in storage: 100 PB logical becomes 300 PB raw. Erasure coding can reduce overhead dramatically, but read-modify-write, repair, and small writes become more CPU and network intensive. Many systems use replication for hot data and erasure coding for cold data.
Large block size improves throughput and reduces metadata count. It is bad for many tiny files. If you store billions of 4 KB files, master metadata explodes and every file wastes block-level assumptions. This is why big-data file systems often need compaction formats such as Parquet, ORC, SequenceFile, or object bundles.
Strong POSIX semantics vs throughput is also a trade-off. Full random writes, rename guarantees, file locking, and close-to-open consistency across thousands of nodes are expensive. HDFS/GFS choose simpler semantics that fit analytics: write once or append, then read many times.
Google File System introduced many of these patterns for large-scale internal workloads. HDFS brought them to the Hadoop ecosystem with NameNode, DataNode, blocks, rack-aware replication, and MapReduce/Spark locality. Ceph uses a different model with CRUSH placement and OSDs, better suited to object/block/file storage under one cluster but operationally more complex. Amazon S3 and cloud object stores solve adjacent problems with different APIs and consistency/latency expectations.
Modern data lakes often combine object storage with table formats such as Iceberg, Delta Lake, or Hudi. They are not HDFS, but they inherit the same concerns: large immutable files, metadata scaling, compaction, partitioning, and recovery from partial writes.
Do not route file bytes through the master. The master should answer metadata questions; DataNodes should serve data.
Do not design for millions of tiny files without changing the metadata model. Tiny files overload the master, waste block accounting, and destroy scan throughput. Compact them or use a database/object store designed for small objects.
Do not place all replicas on the same rack. A rack switch or power failure can remove all copies. Rack-aware placement should spread replicas across failure domains.
Do not re-replicate at unlimited speed after a node failure. Recovery traffic can take down the cluster if it competes with production scans. Use throttling and prioritization.
Do not ignore fencing during failover. If both active and standby masters accept mutations, the namespace can split brain. A standby must fence the old active before serving writes.
Do not promise full POSIX behavior if the system is optimized for append and streaming. Product and API semantics must match the storage engine.
Do not use HDFS/GFS-style storage for low-latency transactional data, user profiles, shopping carts, leaderboards, or random small-object reads. Use a database, cache, key-value store, or object store depending on access pattern.
Do not build this architecture for a small team storing a few terabytes. Managed object storage is simpler, more durable operationally, and usually cheaper once staffing is included.
Do not choose three-way replicated DFS for cold archival data if erasure-coded object storage meets latency needs. Replication wastes too much raw capacity for rarely read data.
Do not use it when the dominant workload is many concurrent small updates to the same files. The lease and append model is not a substitute for a transactional database.
Read ::concept{slug="replication"} for replica placement and repair, ::concept{slug="consensus-overview"} for master failover and quorum logs, ::concept{slug="sharding"} for namespace and block distribution, ::concept{slug="partitioning-strategies"} for data locality, and ::concept{slug="pacelc-theorem"} for latency/consistency trade-offs. Then compare HDFS/GFS with S3-style object storage and Ceph-style CRUSH placement: the APIs look similar only until you inspect metadata, consistency, and failure recovery.