AI · Intermediate

RAG chatbot over documents

Hybrid retrieval, reranking and a cited LLM answer, with background chunking and embedding of documents.

Every part, and why it is there

Employees (chat UI)

Ask questions like “what is our parental leave policy?” and expect a correct answer with a link to the source, not a confident guess.

Doc sources (wiki, drive)

Wikis and shared drives send change notifications whenever a document is created, edited or deleted.

Load balancer

Separates the chat traffic from the ingestion webhooks so a bulk re-import never slows down answers.

Chat service

Orchestrates one answer: rewrite the question using the conversation, retrieve passages, build the prompt, call the LLM and stream tokens back. Most of its time is spent waiting on the model.

Conversations DB

Chat history, so follow-ups like “and for contractors?” make sense, plus thumbs-up/down feedback for evaluation.

Retriever

Runs a vector search and a keyword search in parallel, merges the results and drops anything the user is not allowed to read. Permission filtering here is what stops the bot leaking HR files.

Vector index

Chunks stored as embeddings, so “time off for new parents” finds a passage titled “parental leave” even without shared words.

Keyword index (BM25)

Classic keyword search. It catches exact terms embeddings blur, like error codes, product names and policy numbers.

Reranker

A cross-encoder that reads the question and each candidate passage together and re-orders them. Sending the LLM 5 great passages beats 50 mediocre ones.

LLM API

Writes the answer from the retrieved passages and cites them. The prompt tells it to say “I don’t know” when the passages do not cover the question.

Ingest API

Receives change notifications and queues a job per document. Ingestion is asynchronous because parsing and embedding a large PDF can take minutes.

Ingestion jobs

Smooths out bursts such as a first-time import of 100,000 documents, and retries failures.

Chunk & embed workers

Split each document into overlapping chunks, prefix each chunk with a short note on where it sits in the document, embed it, and write it to both indexes with its permissions.

Embedding model

Turns text into vectors. Queries and chunks must use the same model, so changing it means re-embedding everything.

Asked in interviews at

  • openai
  • plaid
  • microsoft
  • qualified-health-ai
  • scale-ai
  • atlassian
  • intuit

Learn more

All system design examples