Redis Cluster with Sentinel Failover

Overview

This architecture demonstrates a production Redis Cluster deployment with automatic failover. The cluster is sharded across 3 master nodes, each with a dedicated replica for high availability. A Sentinel ensemble monitors all masters and orchestrates automatic promotion of replicas when a master fails. This design provides horizontal write scalability, read scaling through replicas, and sub-second failover without manual intervention.

Redis Cluster Architecture

Hash Slot Sharding

Redis Cluster divides the keyspace into 16,384 hash slots. Each master node owns a contiguous range of slots. When a client writes a key, Redis computes CRC16(key) mod 16384 to determine which slot (and therefore which master) owns that key.

In a 3-shard cluster, the typical slot distribution is:

• Master 1: slots 0-5460
• Master 2: slots 5461-10922
• Master 3: slots 10923-16383

This deterministic mapping means clients can compute the target node locally, avoiding a central routing layer. When a client sends a command to the wrong node, Redis returns a MOVED redirect, and smart client libraries cache the slot-to-node mapping to minimize redirects.

Master-Replica Topology

Each shard consists of one master and one (or more) replicas:

• Masters accept both reads and writes for their slot range.
• Replicas asynchronously replicate from their master and can serve read traffic when configured with READONLY mode.

The replication is asynchronous by default, which means a master acknowledges writes before the replica confirms receipt. This provides lower write latency at the cost of potential data loss during failover (the replica may be slightly behind the master). For workloads that require stronger guarantees, WAIT can be used to block until a specified number of replicas acknowledge the write.

Why 3 Shards

Three shards is the minimum for a production Redis Cluster because:

• The cluster requires a majority of masters to agree on the state of the cluster. With fewer than 3, losing a single master means losing the majority.
• It provides a meaningful distribution of the keyspace, preventing hotspots from concentrating on a single node.
• It balances operational simplicity with capacity. Each shard handles roughly 5,400 slots.

Sentinel for Automatic Failover

What Sentinel Does

Redis Sentinel is a distributed monitoring system that runs alongside the cluster. Sentinel instances continuously ping all master and replica nodes. When a master becomes unreachable, Sentinel coordinates a failover:

1. Subjective Down (SDOWN): A single Sentinel marks a master as down after down-after-milliseconds of missed heartbeats.
2. Objective Down (ODOWN): When a quorum of Sentinels agree the master is down, it is marked objectively down.
3. Leader Election: Sentinels elect a leader to perform the failover.
4. Promotion: The leader selects the most up-to-date replica and promotes it to master.
5. Reconfiguration: All other replicas and clients are notified of the new master's address.

Sentinel Ensemble

A minimum of 3 Sentinel instances is required for reliable failover. The quorum (typically 2 out of 3) prevents split-brain scenarios where network partitions cause conflicting failovers. Sentinel instances should be deployed on separate hosts or availability zones to tolerate infrastructure failures.

sentinel monitor shard-1 master-1-host 6379 2
sentinel monitor shard-2 master-2-host 6379 2
sentinel monitor shard-3 master-3-host 6379 2
sentinel down-after-milliseconds shard-1 5000
sentinel failover-timeout shard-1 15000

Data Flow

Write Path

1. Client computes the hash slot for the key.
2. Client sends the write command to the correct shard master (or follows a MOVED redirect).
3. Master executes the command and responds to the client.
4. Master asynchronously replicates the command to its replica(s).

Read Path

For read-heavy workloads, clients can direct read queries to replicas:

1. Client sends READONLY to a replica node.
2. Subsequent read commands are served from the replica's local dataset.
3. This distributes read load across masters and replicas, effectively doubling read throughput per shard.

Note that reads from replicas may return slightly stale data due to asynchronous replication. This is acceptable for caching use cases but not for operations that require read-after-write consistency.

Failover Process

When Master 1 crashes:

1. Sentinel detects missed heartbeats after 5 seconds.
2. Sentinel quorum (2/3) agrees Master 1 is down.
3. Sentinel leader promotes Replica 1 to become the new Master 1.
4. Replica 1 begins accepting writes for slots 0-5460.
5. Clients receive notification of the topology change and update their slot mapping.
6. Total failover time: typically 5-15 seconds depending on configuration.

When to Use Redis Cluster vs. Alternatives

Redis Cluster (this architecture)

Choose Redis Cluster when:

• You need horizontal write scalability beyond what a single master can handle.
• Your dataset exceeds the memory capacity of a single node.
• You require automatic sharding with minimal application-level complexity.

Redis Sentinel-Only (no Cluster)

Choose Sentinel without Cluster when:

• Your dataset fits in a single node's memory.
• You only need high availability (automatic failover) but not sharding.
• You want simpler operations with a single master handling all keys.

Key Differences

Production Considerations

• Memory overhead: Reserve 30-40% of each node's memory for fragmentation, replication buffers, and fork overhead during persistence.
• Persistence: Use appendonly yes with appendfsync everysec for durability without sacrificing throughput.
• Network: Deploy masters and replicas in different availability zones for infrastructure-level redundancy.
• Monitoring: Track replication lag (info replication), memory usage, and command latency per shard. Alert on replication lag exceeding 1 second.
• Scaling: To add a 4th shard, use redis-cli --cluster reshard to migrate slots from existing masters to the new one with zero downtime.