RAG Explained: Building a Chatbot That Only Answers From Your Documents
Updated
A chatbot that invents your refund policy is worse than no chatbot. Here is how retrieval-augmented generation keeps answers grounded in your actual documents.
Ask a general-purpose model about your company's warranty terms and it will produce a confident, well-written, entirely fictional answer. It has never seen your documents. Retrieval-augmented generation — RAG — fixes this by retrieving the relevant passages from your own content first, then asking the model to answer using only those passages.
That single architectural change is what separates an internal tool your team trusts from a demo that gets quietly disabled after the first embarrassing answer.
How it works, in four steps
- Ingestion: your documents — contracts, policies, product specs, support tickets, past proposals — are split into passages of a few hundred words each.
- Indexing: each passage is converted into a vector that encodes its meaning, and stored in a searchable index alongside its source metadata.
- Retrieval: an incoming question is converted the same way, and the closest passages are pulled back, usually combined with a keyword search to catch exact terms like product codes.
- Generation: the model receives the question plus those passages, with an instruction to answer only from them and to cite the source.
Where these projects actually fail
Messy source documents
Scanned PDFs without text layers, five versions of the same policy with no indication of which is current, tables that lose all meaning when flattened into text. Cleaning and structuring the corpus is typically sixty percent of the work, and skipping it guarantees a system that confidently cites a document superseded in 2023.
Naive chunking
Splitting every 500 characters cuts sentences in half and separates a table header from its rows. Chunk on structure — sections, headings, logical units — and keep enough overlap that context survives the boundary.
No refusal path
If retrieval returns nothing relevant, the correct behaviour is to say so and offer a human contact. A system permitted to fall back on general knowledge will eventually give legal-sounding advice you never wrote, in your brand voice, to your customer.
Measuring whether it works
Build an evaluation set before you build the system: fifty real questions, with the correct answer and the document it comes from. Then measure retrieval accuracy — did the right passage come back — separately from answer quality. Without this split, you cannot tell whether a bad answer is a retrieval failure or a generation failure, and you will tune the wrong component.
- 90%+ target retrieval accuracy on your eval set
- 0 answers invented outside the corpus
- 100% answers carrying a source citation
What it costs to run
For a small to mid-sized US business, a well-built internal RAG assistant typically runs in the low hundreds of dollars per month: embedding costs are one-off per document, vector storage is inexpensive at this scale, and generation is billed per question. The dominant cost is not infrastructure — it is the initial effort to get your documents into a state worth indexing.
Keeping the index fresh
A RAG assistant is only as current as its last ingestion, and a confidently cited obsolete policy does more damage than no answer at all. Decide from the start how documents get in, and how they get out.
- Re-index on change rather than on a schedule, so an updated price list is live in minutes instead of at the end of the month.
- Delete properly. Removing a document from a folder must remove its passages from the index, or the assistant keeps quoting a contract you terminated.
- Carry the effective date in the metadata and show it in the answer. Readers forgive an old document they can see the date of.
- Keep one canonical version per subject. Two contradictory policies in the corpus produce an assistant that picks one at random, which is worse than either.
Permissions, in practice
Access control has to happen during retrieval, not after generation. If the model sees a passage it should not, filtering the wording afterwards does not help: the answer still carries the content, and the citation points at the document. Attach the audience to every passage at ingestion, which is the same question as what an AI system may hold about a customer, pass the asker's groups with the query, and let the index return only what that person could already open. A salaried assistant that answers HR questions to the whole company is a data incident waiting for a screenshot.
When RAG is the wrong tool
- Questions about numbers in a database. How many orders shipped late last month is a query, not a retrieval. Point the assistant at the data through a proper query layer instead.
- Live status. Order tracking, stock levels and open tickets change hourly, so call the system that owns them rather than indexing yesterday's copy.
- Arithmetic and calculations. Ask the model to reason over the retrieved figures at your peril; compute it in code and let the assistant present the result.
- A corpus of five documents. If a person could find the answer with a search box in ten seconds, ship the search box.
Most of the AI assistants we build end up as a hybrid: retrieval for anything written down, a handful of tools for anything that is stored as data, and a clear refusal path when neither applies.
A grounded assistant that answers eighty percent of questions with citations beats an ungrounded one that answers a hundred percent, because the second requires you to verify every answer anyway.
Frequently asked questions
Is RAG better than fine-tuning a model?
For factual company knowledge, almost always. Fine-tuning teaches style and format; retrieval supplies facts. Updating a RAG system means adding a document, whereas updating a fine-tuned model means retraining it — and neither approach makes a model reliably recite specifics it was merely trained on.
How many documents do I need for RAG to be worthwhile?
Value appears once search stops being trivial, typically somewhere past a few dozen substantial documents. Below that, a well-organized shared drive is genuinely the better answer, and we will tell you so.
Can a RAG assistant leak confidential information?
Yes, if permissions are not enforced at retrieval time. Access control must filter the index per user before generation, not afterwards in the prompt. A model instructed to hide information it has already been shown is not a security boundary.
How long does a RAG project take to build?
A focused internal assistant over a clean corpus is typically four to eight weeks, and the split surprises people: a couple of weeks of engineering and the rest on documents, permissions and the evaluation set. If the source material is scanned PDFs and contradictory drafts, the document work dominates the schedule. That effort is not wasted, because a corpus worth indexing is also a corpus your team can finally search.
Should it answer customers or only staff first?
Start internal, almost always. Your team tolerates an imperfect answer, tells you when it is wrong and gives you the questions the evaluation set was missing. After a few weeks you have real accuracy figures and a refusal path that has been tested by people who cannot leave a bad review. Open it to customers once retrieval accuracy is stable and the escalation route to a human is genuinely quick.
More on this topic: Artificial Intelligence.
Keep reading
Want this built for your business? See what we do.