LLM System Design Interview: A 45-Minute RAG Chatbot Walkthrough

The prompt is usually one line: "Design a chatbot that answers questions over our company's documents." Naming the most tools won't carry you through the next forty-five minutes. Asking what a wrong answer costs, putting numbers on the design, and saying how you'd know it works will.

This is a structure for the LLM system design interview on a RAG chatbot, with timings, the sentences worth saying aloud, and a cost and latency calculation I ran as a script rather than guessing. It's a practice guide, not a leak of real questions. I haven't taken or given these interviews at any named company, and nothing here is claimed to be an actual question from one. For shorter Q&A style prep, use AI engineer interview questions on RAG, agents and evals. If you need the pipeline basics first, read how RAG works.

The 45-minute plan for an LLM system design interview

Say this plan out loud in the first two minutes. It shows structure, and it lets the interviewer redirect you early.

If you run out of time, the evals and failure modes blocks are the ones to protect. They are easy to squeeze out, and they are what separates a diagram from a system.

Clarify requirements before drawing anything

Spend five minutes on questions. A good set:

• Who asks, and what happens when the answer is wrong? An internal HR bot and a bot that answers customers about refunds have different tolerances.
• How big is the corpus, what formats (PDF, wiki, tickets), and how often does it change?
• Do users need citations? Do different users have different document permissions?
• What are the latency and cost targets? "Under 3 seconds to first token" and "under half a cent per question" are very different designs.
• What is the traffic? Queries per day, and how bursty.

Then write your assumptions on the board. For the rest of this post I use these, all of them invented for practice: 20,000 documents averaging 1,500 tokens, 5,000 questions a day, answers of about 250 tokens. Say "I'll assume" every time. Never present an assumption as a fact about the company.

What to say out loud: "The thing I'd most like to avoid is a confident wrong answer, so I'll bias toward refusing when retrieval is weak."

Ingestion and chunking

Parsing is where silent damage happens. A PDF with two columns can come out interleaved, and a table can turn into a stream of numbers with no headers. Mention that you'd inspect a sample of parsed output by eye before building anything on top of it.

For chunking, give a default and a reason to deviate:

• Start with 300 to 500 tokens per chunk and 10 to 15 percent overlap. It's a starting point you tune, not a law.
• Prefer boundaries the document already has: headings, sections, list items. Split on structure first, size second.
• Attach metadata to every chunk: source, title, section path, updated date, access group. Filtering by metadata is cheaper and more reliable than hoping the embedding captures it.
• Prepend the document title and section heading to the chunk text before embedding. Many chunks are meaningless alone ("It must be renewed every 12 months").

Also cover refresh. Re-embed only changed documents, keyed by a content hash, and delete chunks of removed documents. Candidates forget deletion, and then a retired policy keeps getting cited.

Embedding model choice is a trade-off between quality, dimension (storage), and price. The site has a comparison at embedding models compared; in the interview, say you'd pick two candidates and measure them on your own labelled queries rather than trust a leaderboard.

Retrieval and reranking

The default answer that holds up: hybrid retrieval, then rerank.

1. Dense (embedding) search catches paraphrases.
2. Keyword search (BM25) catches exact strings: error codes, product names, SKUs, clause numbers. Embeddings are often weak here.
3. Merge the two ranked lists, for example with reciprocal rank fusion.
4. Take roughly 30 candidates and rerank them with a cross-encoder or a small model, then keep the top 5.

Why rerank at all? First-stage retrieval is fast but coarse. A reranker reads the query and each candidate together, so it is better at ordering, but too slow to run over the whole corpus. That is the whole argument.

Say the failure you're guarding against: "If the right chunk isn't in the top 30, nothing downstream can save the answer. So I measure recall at 30 for the first stage, and precision at 5 after reranking."

Two refinements worth a sentence each: query rewriting for follow-up questions that depend on chat history ("what about for contractors?"), and a similarity threshold below which the bot says it doesn't know instead of generating from noise. For a deeper take on agent-driven retrieval, see agentic RAG, but don't open with it. Start simple and say when you'd add complexity.

Generation: prompt shape, citations, refusal

Keep this block short. The design points:

• Stable instructions first (cacheable), retrieved chunks next, chat history, then the question.
• Number the chunks and require the answer to cite chunk numbers. Then verify in code that every cited number exists in the context. This catches fabricated citations cheaply.
• Tell the model what to do when the context doesn't contain the answer, and test that instruction, because it is the one that quietly fails.
• Treat retrieved text as data. A chunk that says "ignore your instructions" is an injection attempt, which is a real risk for any corpus users can edit. See prompt injection in tool-using agents.

If the interviewer asks which model: name a tier, not a brand. "A small fast model for most questions, with a larger one for hard or low-confidence cases" is a better answer than a product name that will be stale next quarter.

Evals: the block most candidates skip

You need two layers, and you should say so.

Retrieval evals. Build a golden set of 100 to 200 real questions, each labelled with the chunk or document that answers it. Measure recall at k and mean reciprocal rank. This layer is cheap, deterministic, and runs on every change to chunking or embeddings. Building evaluation datasets covers how to assemble the set.

Answer evals. Given the retrieved context, is the answer faithful to it (no claims outside the sources), and does it address the question? An LLM judge is acceptable if you say how you'd check the judge: agree it against human labels on a sample, watch for length and position bias, and avoid grading a model with itself.

Then the habit that wins interviews: when a user reports a wrong answer, pull the retrieved chunks first. If the right chunk wasn't retrieved, it's a retrieval bug. If it was retrieved and the answer ignored it, it's a generation bug. If the chunk itself is wrong or stale, it's a data bug. Three different fixes.

Online, track thumbs up and down, "no answer" rate, citation click-through, and escalation to a human. Log the query, chunk IDs, scores and prompt version for every request, or you can't debug anything.

Cost and latency: do the arithmetic out loud

This is where you earn trust. I wrote a script with the assumptions above and ran it on 2026-10-08. The LLM prices come from Anthropic's pricing page, read that day: Claude Sonnet 5.5 at $2 input and $10 output per million tokens, cache reads at $0.10; Claude Haiku 5.5 at $0.10 and $0.50 for prompts up to 100,000 tokens, cache reads at $0.01. Everything else (corpus size, traffic, stage times) is an assumption I made up. Latency numbers in particular are placeholders: measure your own.

Real output:

corpus tokens: 30,000,000
chunks (size 400, overlap 50): 85,715
tokens to embed once: 34,286,000 (1.14x corpus)
vector storage at 1024 float32 dims: 0.35 GB

input tokens per query: 2940 (system 600, context 2000, history 300, question 40)
output tokens per query: 250

Claude Sonnet 5.5
  per query, no caching:      $0.00838
  per query, prefix cached:   $0.00724
  per day (5,000 q):        $41.90 plain, $36.20 cached
  per 30 days:                $1257.00 plain, $1086.00 cached

Claude Haiku 5.5 (prompt <=100k)
  per query, no caching:      $0.00042
  per query, prefix cached:   $0.00036
  per 30 days:                $62.85 plain, $54.75 cached

latency (assumed stage times)
  user sees first token at: 920 ms
  full answer at: 5087 ms

top_k sensitivity, Sonnet 5.5, no caching
  k= 3:  2140 input tokens, $0.00678 per query, $1017.00 per 30 days
  k= 5:  2940 input tokens, $0.00838 per query, $1257.00 per 30 days
  k=10:  4940 input tokens, $0.01238 per query, $1857.00 per 30 days
  k=20:  8940 input tokens, $0.02038 per query, $3057.00 per 30 days

What the numbers tell you, and what to say:

• Storage is trivial. 85,715 chunks at 1024 dimensions is about 0.35 GB. At this scale the vector database choice is not a cost story.
• The one-time embedding bill is small and one-time. About 34 million tokens, then only changed documents. Multiply by your embedding provider's price, which I did not look up for this post.
• Generation dominates the running cost, and here input dominates generation. On Sonnet 5.5, the 250 output tokens cost $0.0025 and the 2,940 input tokens cost $0.00588, even though output is priced 5x higher per token, because the context is large. Cut context before you cut answer length.
• top_k is a cost knob. Going from 5 to 20 chunks roughly 2.4 times the monthly bill in this model, and more context also tends to hurt precision. Rerank to a small k.
• Caching saves less than people expect here, about 14 percent, because only the 600-token system prompt is stable. Retrieved chunks change each query. Caching matters far more for long fixed prefixes.
• Model choice is the biggest lever. The same traffic on the small model is about $63 versus $1,257 a month. Whether the small model is good enough is an eval question, not an opinion.
• Latency is mostly decoding. With my assumed 60 tokens per second, 250 output tokens take over four seconds, while retrieval and reranking add under 250 ms. Stream the answer so the user sees text at about 0.9 seconds, and say the perceived and total latency separately.

Sanity-check yourself aloud: "5,000 questions at roughly a cent each is about $40 a day. That's the right order of magnitude, so I trust it." For a longer treatment of per-request pricing, see LLM API cost per 1,000 requests, worked examples.

Cost levers to name, in order of payoff: a smaller model with routing for easy questions, smaller context through reranking, a semantic cache for repeated questions (semantic caching), prompt caching for stable prefixes, and the batch API only for offline work like re-summarizing documents.

Failure modes to bring up before they ask

Interviewers reward candidates who attack their own design. A short list with the mitigation attached:

The permission row is the one worth stressing. Putting "only answer from documents the user may see" in the prompt is not access control. Filter in the retrieval query.

What to say out loud, and what to avoid

Phrases that work:

• "I'll state my assumptions, and you tell me where they're wrong."
• "Before I touch the prompt, I'd look at what was retrieved."
• "I can't tell whether the small model is good enough without the golden set, so that's the first thing I'd build."
• "Here's the version I'd ship in week one, and here's what I'd add once the metrics justify it."

Habits to drop:

• Naming five vector databases without a reason to pick one.
• Saying "fine-tune it" before measuring a baseline.
• Giving numbers without saying they're assumptions.
• Adding agents, graphs or multi-hop retrieval to a problem that didn't ask for them.

An illustrative exchange (written by me, not a transcript):

Interviewer: Users say the bot is wrong about our refund policy. You: I'd start with one failing query. Do we log the retrieved chunk IDs? If the current policy page isn't in the top 5, I check whether it's in the top 30, which tells me if reranking or first-stage search is at fault. If it is retrieved, I look at the prompt and whether an older policy chunk outranked it.

That answer has a sequence, a decision point at each step, and no tool names. It is the pattern for the whole interview.

Practice drill

Run this on a friend or alone with a timer. Pick a corpus you know (your team's runbooks, a product manual). Write assumptions for size, traffic and budget in two minutes. Then copy the script logic into a spreadsheet and change one input at a time: chunk size, k, model tier. Predict the effect before you change it. If your prediction is off by more than 2x, find out why. That gap is the thing interviews actually probe.

If you want to prototype instead of just talk, build a RAG chatbot over your own PDFs gives you a working baseline to measure.

Prices were read on 2026-10-08, re-checked on 2026-10-09, and change often. Check the vendor page before you quote a figure in a real conversation.