ClickHouse with S3-Backed Storage

Overview

This architecture demonstrates ClickHouse with disaggregated storage, where compute nodes store metadata and caches locally while all persistent data resides in Amazon S3 (or S3-compatible object storage). The design separates compute from storage, enabling independent scaling, cost optimization, and simplified operations for large-scale analytical workloads.

Why Separate Storage from Compute

Traditional ClickHouse deployments store data on local SSDs attached to each replica. This works well for low-latency queries but introduces significant operational challenges at scale:

• Storage costs grow linearly with replica count. Adding a replica for read throughput also doubles storage.
• Rebalancing shards requires physically moving terabytes of data between nodes.
• Capacity planning must account for peak storage and peak compute simultaneously.

S3-backed storage decouples these concerns. Replicas become stateless compute units that share a single data lake. Adding a replica costs only the compute resources; storage is paid once.

Architecture Components

SharedMergeTree Engine

ClickHouse Cloud and recent open-source builds support SharedMergeTree, a table engine designed for shared storage. Unlike ReplicatedMergeTree, which replicates data parts between nodes, SharedMergeTree writes data parts directly to S3 and registers their metadata in a coordination layer.

Each part is written exactly once to S3 under a unique prefix. All replicas read from the same set of parts, eliminating replication traffic between nodes entirely.

Per-Replica S3 Prefixes

A common pattern in production is to assign each replica its own S3 prefix for ingestion. This avoids write contention on the object store and simplifies debugging:

<storage_configuration>
  <disks>
    <s3_disk>
      <type>s3</type>
      <endpoint>https://s3.amazonaws.com/my-bucket/clickhouse/shard1/replica1/</endpoint>
      <access_key_id>...</access_key_id>
      <secret_access_key>...</secret_access_key>
    </s3_disk>
  </disks>
  <policies>
    <s3_main>
      <volumes>
        <main><disk>s3_disk</disk></main>
      </volumes>
    </s3_main>
  </policies>
</storage_configuration>

All replicas can read parts written by any other replica because the metadata layer tracks every part's S3 location regardless of the prefix that wrote it.

Local SSD Cache

S3 object retrieval latency (50-150ms per GET) is too high for interactive queries. ClickHouse addresses this with a local SSD cache that stores frequently accessed data parts and index files. The cache is populated on read and evicted by LRU policy.

In practice, queries against recent data (the typical hot working set) hit the local cache and achieve latencies comparable to local-SSD deployments. Queries scanning cold historical data fall through to S3 with higher latency but without requiring any local storage provisioning.

ZooKeeper / ClickHouse Keeper

Even with SharedMergeTree, a coordination service is required for:

• Part registration: tracking which data parts exist and their S3 locations.
• Merge coordination: ensuring that background merges do not conflict across replicas.
• DDL replication: propagating schema changes to all nodes.

ClickHouse Keeper (a Raft-based replacement for ZooKeeper) is the recommended choice. A 3-node Keeper ensemble provides fault tolerance with automatic leader election.

Data Flow

1. Ingestion: Client sends INSERT to any replica. The replica writes data parts to its S3 prefix and registers them in Keeper.
2. Background merges: A replica picks registered parts, merges them on local SSD, uploads the merged result to S3, and updates Keeper metadata.
3. Queries: A replica receives a SELECT, consults Keeper for the current set of active parts, reads them from local cache or S3, and returns results.

When to Use This Architecture

This design is well suited for:

• Observability platforms ingesting logs and metrics at high volume, where storage costs dominate.
• Analytics data lakes where the same data is queried by many consumers (dashboards, ad-hoc analysts, ML pipelines).
• Elastic compute environments where replica count changes frequently based on query load.

Trade-offs

Advantage	Trade-off
Storage cost paid once, not per replica	Cold query latency depends on S3 (100-200ms per part fetch)
Replicas are stateless and disposable	Requires a healthy Keeper ensemble for coordination
No inter-replica replication traffic	Local SSD cache needed for interactive latency on hot data
Independent compute and storage scaling	Slightly more complex configuration than local-only setup

For workloads where sub-10ms p99 latency is required across the entire dataset (not just the hot working set), local SSD storage remains the better choice. For cost-sensitive, high-volume analytical workloads, S3-backed storage reduces operational burden and infrastructure costs significantly.