Retrieval-augmented generation (RAG) explained
How RAG lets AI answer from your own documents: chunking, embeddings, vector search and grounded answers, with design tips and common failure modes.
By the AiTrain editorial team · Reviewed under our editorial standards
Retrieval-augmented generation, usually shortened to RAG, is the most common way to make an AI assistant answer questions from your own documents — policies, manuals, contracts, knowledge bases — instead of from general training data.
Key takeaways
- RAG = search your documents for relevant passages, then give those passages to the model with the question.
- It reduces made-up answers and lets the model cite sources.
- Most quality problems come from the retrieval step, not the model.
- Always show users which sources an answer came from.
The pipeline
- Collect documents you trust.
- Chunk them into passages (often a few paragraphs each), keeping titles and section headings with each chunk.
- Embed each chunk: convert it into a list of numbers (a vector) that represents its meaning.
- Store the vectors in a searchable index.
- Retrieve: when a question arrives, embed it and find the chunks with the most similar meaning.
- Generate: send the question plus the top chunks to the model with an instruction such as "Answer using only these sources and cite them."
Why embeddings work
Embeddings place text with similar meaning close together in a high-dimensional space. "How many vacation days do I get?" lands near a policy paragraph about "annual leave entitlement" even though they share few words. That is why RAG search often beats keyword search for natural questions. Many systems combine both — semantic search plus keyword matching — for the best results.
Design tips
- Chunk by meaning. Split on headings and sections rather than fixed character counts where possible.
- Keep metadata. Store document title, date, owner and link with each chunk to enable citations and filtering.
- Retrieve a few, not many. Too many chunks dilute the relevant ones and raise cost.
- Instruct grounding. "If the sources don't contain the answer, say so."
- Keep documents current. Outdated policies produce confidently outdated answers. Re-index when documents change.
Common failure modes
| Symptom | Likely cause | Fix |
|---|---|---|
| Says "not found" for answers that exist | Chunks too small or poor search | Larger chunks, hybrid search, better titles |
| Mixes up two similar policies | Missing metadata | Add document names and dates; filter by them |
| Cites the wrong section | Too many retrieved chunks | Retrieve fewer; re-rank results |
| Answers from general knowledge | Weak instruction | Explicit "use only the sources" rule |
Evaluate it
Build a test set of 30–50 real questions with known correct answers and the source each should come from. Re-run it whenever you change chunking, search or prompts. Our guide on evaluating AI outputs shows how.
Security
Only index documents users are allowed to see, and enforce permissions at retrieval time. Treat text inside documents as data, not instructions, to reduce prompt-injection risk.