Design an LLM inference API

Question 1

Who is calling, and what do they get back?

Accepted Answer

A streaming response, not a single reply — the connection stays open for seconds while tokens arrive. That alone rules out treating this like an ordinary request/response API.

Question 2

Where are per-customer quotas enforced?

Accepted Answer

At the edge, before a request reaches a GPU. Inference is the most expensive request your system will ever serve, so authentication, quota and admission all have to happen where rejecting is still cheap.

Question 3

Demand spikes to five times capacity. What happens to the extra requests?

Accepted Answer

You cannot autoscale GPUs in seconds — they may not even be available to rent. A queue in front of the pool turns a spike into a wait instead of a wall of errors, and it is where you decide who waits: priority, per-tenant fairness, and shedding load that has already timed out.

Question 4

One request at a time on a GPU wastes most of the card. What fixes that?

Accepted Answer

Batching many sequences into one forward pass — and doing it continuously, so a finished sequence leaves the batch and a queued one joins at the next step rather than at the next batch. This component is the whole performance story of an inference system, and it is not the model server.

Question 5

What actually runs the model?

Accepted Answer

A pool of GPU workers, each holding the weights in memory. They scale on a completely different curve from the API layer — and cost hundreds of times more — which is why they sit behind their own queue rather than one per request.

Question 6

Generating token 300 shouldn’t re-read tokens 1 to 299. What stops it?

Accepted Answer

The KV cache: the attention keys and values for the tokens so far, kept in GPU memory so each new token is one step rather than a re-run. It is also the memory budget that decides how many requests fit in a batch — and caching shared prefixes across requests is free throughput when many prompts start the same way.

Question 7

How do you bill for this, and how do you know a tenant is over quota?

Accepted Answer

Tokens in and out, per request, per customer. It is small structured data written constantly and queried by account — a database — and it is the part candidates skip even though it is how the product makes money.

Question 8

How many requests a second is 20M a day?

Accepted Answer

20,000,000 / 86,400 is about 231 a second. Trivially small as a request rate — which is the point: at roughly 1,300 tokens each that is around 300,000 tokens a second, and tokens, not requests, are the unit this system is sized in.

Question 9

Bigger batches mean better GPU utilisation. What do they cost?

Accepted Answer

Latency, and specifically the latency of whoever arrived first. Waiting to fill a batch adds time to first token; padding short prompts to the longest in the batch wastes compute; and the KV cache means memory, not compute, is usually what caps batch size. Naming which of throughput and latency you are optimising, and why, is the answer.

Blueprint — system design