Design a distributed cache

Question 1

Who is calling the cache?

Accepted Answer

Another service, on its own hot path — which is why a cache lookup has to cost less than the query it is replacing, or it is not worth having.

Question 2

A thousand nodes hold the data. How does a caller know which one has its key?

Accepted Answer

Something has to turn a key into a node. Whether it lives in a smart client library or a proxy in front of the fleet, it has to be on the board — this is where consistent hashing actually runs.

Question 3

Ten billion keys at 1KB is 10TB. Where does it sit?

Accepted Answer

On no single machine. The keyspace is partitioned across the fleet, each node holding a slice in memory. Sharding is not an optimisation here — it is the only reason the data fits at all.

Question 4

One node dies holding a thousandth of the keyspace. What do its callers see?

Accepted Answer

With a single copy, a cold miss storm straight onto the origin store — which is how a cache node failure becomes a database outage. A replica per shard turns it into a failover instead.

Question 5

A node is added at 3am. How does every caller find out?

Accepted Answer

The ring has to be agreed on. Either a config service everyone reads or gossip between the nodes — but something owns the membership list, or two callers hash the same key to different nodes and the cache quietly splits in two.

Question 6

A key is not in the cache. Then what?

Accepted Answer

A cache is a copy of something. Drawing the origin is what makes the miss path — and the thundering herd onto it — visible.

Question 7

How are keys placed, and what happens when memory runs out?

Accepted Answer

Hashing modulo the node count remaps almost every key the moment the count changes — one node added and the whole cache is cold. Consistent hashing moves about 1/N instead. And a cache must evict: LRU plus a TTL is the usual answer, and saying so is what shows you know a cache is allowed to lose data.

Blueprint — system design