# ClickHouse with S3-Backed Storage ## Overview This architecture demonstrates **ClickHouse with disaggregated storage**, where compute nodes store metadata and caches locally while all persistent data resides in Amazon S3 (or S3-compatible object storage). The design separates compute from storage, enabling independent scaling, cost optimization, and simplified operations for large-scale analytical workloads. ## Why Separate Storage from Compute Traditional ClickHouse deployments store data on local SSDs attached to each replica. This works well for low-latency queries but introduces significant operational challenges at scale: - **Storage costs grow linearly** with replica count. Adding a replica for read throughput also doubles storage. - **Rebalancing shards** requires physically moving terabytes of data between nodes. - **Capacity planning** must account for peak storage and peak compute simultaneously. S3-backed storage decouples these concerns. Replicas become stateless compute units that share a single data lake. Adding a replica costs only the compute resources; storage is paid once. ## Architecture Components ### SharedMergeTree Engine ClickHouse Cloud and recent open-source builds support `SharedMergeTree`, a table engine designed for shared storage. Unlike `ReplicatedMergeTree`, which replicates data parts between nodes, SharedMergeTree writes data parts directly to S3 and registers their metadata in a coordination layer. Each part is written exactly once to S3 under a unique prefix. All replicas read from the same set of parts, eliminating replication traffic between nodes entirely. ### Per-Replica S3 Prefixes A common pattern in production is to assign each replica its own S3 prefix for ingestion. This avoids write contention on the object store and simplifies debugging: ```xml s3 https://s3.amazonaws.com/my-bucket/clickhouse/shard1/replica1/ ... ...

s3_disk

``` All replicas can read parts written by any other replica because the metadata layer tracks every part's S3 location regardless of the prefix that wrote it. ### Local SSD Cache S3 object retrieval latency (50-150ms per GET) is too high for interactive queries. ClickHouse addresses this with a **local SSD cache** that stores frequently accessed data parts and index files. The cache is populated on read and evicted by LRU policy. In practice, queries against recent data (the typical hot working set) hit the local cache and achieve latencies comparable to local-SSD deployments. Queries scanning cold historical data fall through to S3 with higher latency but without requiring any local storage provisioning. ### ZooKeeper / ClickHouse Keeper Even with SharedMergeTree, a coordination service is required for: - **Part registration**: tracking which data parts exist and their S3 locations. - **Merge coordination**: ensuring that background merges do not conflict across replicas. - **DDL replication**: propagating schema changes to all nodes. ClickHouse Keeper (a Raft-based replacement for ZooKeeper) is the recommended choice. A 3-node Keeper ensemble provides fault tolerance with automatic leader election. ## Data Flow 1. **Ingestion**: Client sends INSERT to any replica. The replica writes data parts to its S3 prefix and registers them in Keeper. 2. **Background merges**: A replica picks registered parts, merges them on local SSD, uploads the merged result to S3, and updates Keeper metadata. 3. **Queries**: A replica receives a SELECT, consults Keeper for the current set of active parts, reads them from local cache or S3, and returns results. ## When to Use This Architecture This design is well suited for: - **Observability platforms** ingesting logs and metrics at high volume, where storage costs dominate. - **Analytics data lakes** where the same data is queried by many consumers (dashboards, ad-hoc analysts, ML pipelines). - **Elastic compute** environments where replica count changes frequently based on query load. ## Trade-offs | Advantage | Trade-off | |-----------|-----------| | Storage cost paid once, not per replica | Cold query latency depends on S3 (100-200ms per part fetch) | | Replicas are stateless and disposable | Requires a healthy Keeper ensemble for coordination | | No inter-replica replication traffic | Local SSD cache needed for interactive latency on hot data | | Independent compute and storage scaling | Slightly more complex configuration than local-only setup | For workloads where sub-10ms p99 latency is required across the entire dataset (not just the hot working set), local SSD storage remains the better choice. For cost-sensitive, high-volume analytical workloads, S3-backed storage reduces operational burden and infrastructure costs significantly.