"Can we just put our documents into an AI and ask it questions?" is the most common request we get. The honest answer is yes, in a day, and then no, for the next three months, because the first version will confidently answer questions wrongly and your team will stop trusting it. Here is how to build retrieval-augmented generation (RAG) that people keep using.
The failure modes are boring and predictable
RAG systems go wrong in four ways. The right document was never indexed. The right document was indexed but not retrieved. The right passage was retrieved but the model ignored it or blended it with something else. Or the answer was correct but the user could not tell, so they did not trust it. Every design decision below exists to address one of these.
Chunking is a product decision
Splitting documents every 500 tokens is the default and it is usually wrong. Policies want to be chunked by clause. Support articles want to be chunked by heading with the title prepended to every chunk. Contracts want section numbers kept intact. Spend an afternoon reading your own documents and ask "what is the smallest piece that answers a question on its own?" That is your chunk, and attaching the document title, section and date to each chunk improves retrieval more than any embedding model upgrade.
Retrieval quality beats model quality
A better language model cannot fix bad retrieval. Measure retrieval on its own: for 100 real questions, is the passage that contains the answer in the top five results? If not, no amount of prompting will help. Hybrid search, combining keyword matching with vector similarity, is the single biggest improvement we see on business documents, because people search for product codes, invoice numbers and exact names that embeddings handle poorly. Add a reranking step once hybrid search is in place.
Make the model show its sources
Every answer should cite the passages it used, with a link to the original. This does two things. Users can verify, which builds trust. And when the answer is wrong, you can see whether retrieval or generation was at fault. Instruct the model to say "I could not find this in the documents" rather than guess, and test that it actually does so with questions you know are not covered.
Permissions are not optional
If your documents have access rules, your retrieval must respect them before the model ever sees a passage. Filtering after generation leaks information through the answer. Index documents with their permission metadata and filter at query time. This is the step most tutorials skip and the one your security team will ask about first.
Keep the index fresh
A RAG system that answers from last quarter's price list is worse than none. Decide up front how documents get added, updated and removed, and show the "last updated" date on every source. For most teams a nightly sync from the document store is enough; for anything operational, index on change.
Evaluate with real questions, not synthetic ones
Collect the questions people actually ask, in the words they use, and build a test set with expected sources and answers. Run it before every change. A set of 150 questions takes two days to build and will save you from shipping regressions for the life of the system. Track three numbers: retrieval hit rate, answer correctness and "answered when it should have abstained".
Takeaway
RAG is a search problem with a language model on top. Treat the search part with the seriousness it deserves, be honest about what the system does not know, and put citations in front of users. Do that and the "AI over our documents" idea becomes a tool people reach for every day instead of a demo that got a polite nod.
