🎥 FDE Mock Interview
RAG System Design Mock Interview
A health insurer's AI told a staff member an MRI was covered. It cited the plan document, and it was wrong. Watch every decision a Forward Deployed Engineer makes to get from that request to a system the client can trust.
10 minutes 44 seconds. Watch on YouTube
What this mock interview covers
Most RAG interview prep starts with the architecture: chunk the documents, embed them, store them in a vector database, retrieve the top results and pass them to a model. That pipeline matters, but it's rarely where a client deployment goes wrong. This mock starts earlier and ends later. It plays through a real Simul FDE scenario, talking through client discovery, the choice between RAG, tools and machine learning, a wrong answer in production-style testing, and how to prove accuracy before launch.
It's useful if you are preparing for a Forward Deployed Engineer or AI engineer loop, or if you already know the RAG vocabulary and want to see how the decisions connect on a client project.
Video chapters
- 0:00
Confident, cited, wrong
The failure that opens the video: an AI answer that cites a plan document and is still wrong.
- 0:30
The mandate: 1.2M calls, 6 weeks
The client, the call volume and the deadline.
- 1:15
Step 1: Client discovery
What to ask before deciding what to build.
- 2:30
Step 2: RAG vs tools vs ML
Choosing the design from where the data lives.
- 4:00
Step 3: Debugging a wrong RAG answer
Chunking, metadata filtering and rerankers.
- 5:45
Step 4: RAG evaluation & golden sets
Proving accuracy with a golden set and a launch bar.
- 7:00
How it's scored: 4 FDE principles
What the scenario rewards, and why.
- 7:50
Try it yourself
Where to practice next.
The problem: 1.2 million calls and six weeks
Meridian Health is a fictional large US health insurer. Its call centres take 1.2 million member calls a month, and members wait on hold while staff search plan PDFs. Priya Shah, the VP of Customer Support, wants AI to reduce call volume, and you have six weeks to show it works.
The strong first move is to ask questions, not propose a chatbot: which calls make up most of the volume, what happens on those calls today, and whether “reduce” means fewer calls, shorter calls or both. That reveals that coverage questions are 35% of calls and that Priya wants shorter calls with no more wrong answers. It's a different brief from the one she walked in with.
RAG vs tools vs machine learning
Take the most common question: “Is an MRI covered under my plan?” The rules live in about 60,000 pages of plan PDFs that vary by plan, state and year. Each member's plan sits in a separate member system. Neither source can answer alone, so let the data pick the pattern:
| Where the answer lives | Pattern | Example |
|---|---|---|
| Rules and policies in documents | RAG | What the plan says about MRI scans |
| Live data about one person | Tools (API or database lookup) | Which plan this member is on, and this member's claims |
| Predicting an outcome from history | Machine learning | Forecasting something no document or record states |
The design uses both: look up the member's plan, state and year, then run RAG only over that plan's documents and cite the section used. Indexing all 60,000 pages misses whose plan it is. Querying the database alone misses the exceptions in the PDFs. Training a model on past calls would also learn the wrong answers staff gave, and it would go stale when the plans change.
How to debug a wrong RAG answer
The first version tells a staff member an MRI is covered and cites a plan document. But the member's 2026 plan needs approval before the scan, or the claim is denied. Being confident, cited and wrong is worse than giving no answer. Work through it in this order:
- 1.
Read the retrieval log first
Before changing anything, look at the exact text the system retrieved for the failing question. In the Meridian case, the log shows text from three different plans and years. The model can only answer from what it's given.
- 2.
Check your metadata filters
If rules differ by plan, state and year, similarity search alone will happily mix them, because the wording is almost the same. Look up the member first, then filter retrieval to that plan, state and year.
- 3.
Check your chunking
In the Meridian case, the rule that an MRI needs approval had been cut away from the coverage text. Splitting by document section keeps a rule together with the exceptions printed under it.
- 4.
Only then consider a reranker or a bigger model
A reranker can only sort the text it was handed, and a larger model can't read a rule that was never retrieved. Both may help later, but neither fixes this failure.
Proving accuracy: golden sets and launch bars
Staff will read these answers to members over the phone, so Priya asks how she can know the system is accurate before her team relies on it. A demo with hand-picked questions proves little. Neither does a test set written and graded by another AI model, because nobody at the client agreed those answers are right. And thumbs-down ratings only tell you something went wrong after a member heard it.
The stronger answer is a golden set: real call questions, with answers approved by the client's plan experts. Score search, answers and refusals separately, because each one fails for different reasons. Then agree a launch bar in advance. In this scenario it's at least 90% correct, every answer cited and zero wrong-plan answers, signed off by Meridian's experts.
The four principles it's scored on
Problem first
Find out which calls, why, and what success means before choosing what to build.
Pattern fit
Let the data choose between RAG, tools and machine learning. Most real questions need two of them.
Diagnosis first
Check what was retrieved before changing the model. Wrong text in means a wrong answer out.
Proof rigor
Prove accuracy before launch, with real questions and answers the client has approved.
Now make the decisions yourself
Knowing the answer on video is different from choosing it under pressure. Simul's FDE scenarios are scripted and branching: you make each call, the client reacts, and you get a grade at the end.
Frequently asked questions
What does a RAG system design interview test?
It tests whether you can take a vague business request to a system a client will trust. Knowing what embeddings and vector databases are is only part of it. You also need to find the real problem, decide whether RAG is the right pattern, debug bad answers and prove the system's accuracy. Formats vary by company, so treat this mock as practice for the thinking rather than a script.
Why did the cited answer in this mock turn out wrong?
A citation only shows where the text came from, not that it was the right text. In the scenario, retrieval mixed text from different plans and years, and the approval rule had been cut off from the passage. The system cited a real document and still gave the wrong answer for that member's plan.
When should you use tools instead of RAG?
Use RAG for knowledge that lives in documents, and tools for live data about a specific person or record. Many real questions need both. Here, the member system knows which plan someone is on, and the plan documents explain that plan's rules, so the design looks up the plan first and then reads only that plan's documents.
What is a golden set in RAG evaluation?
A golden set is a test set of real questions with answers approved by people who are qualified to judge them, which here means the client's plan experts. Scoring search, answers and refusals against it shows how often the system is wrong before any customer hears an answer. A demo with hand-picked questions can't show that.
Is this video a spoiler for the Meridian Health scenario?
Yes. It walks through every decision and the best move at each step, and so does this page. If you want a real read on your instincts, play the scenario first. It is part of Premium access, and the Logistics Contradiction scenario is free to play if you want to try the format first.