RAG Explained: Building a Chatbot That Only Answers From Your Documents

A chatbot that invents your refund policy is worse than no chatbot. Here is how retrieval-augmented generation keeps answers grounded in your actual documents.

Ask a general-purpose model about your company's warranty terms and it will produce a confident, well-written, entirely fictional answer. It has never seen your documents. Retrieval-augmented generation — RAG — fixes this by retrieving the relevant passages from your own content first, then asking the model to answer using only those passages.

That single architectural change is what separates an internal tool your team trusts from a demo that gets quietly disabled after the first embarrassing answer.

How it works, in four steps

1. Ingestion: your documents — contracts, policies, product specs, support tickets, past proposals — are split into passages of a few hundred words each.
2. Indexing: each passage is converted into a vector that encodes its meaning, and stored in a searchable index alongside its source metadata.
3. Retrieval: an incoming question is converted the same way, and the closest passages are pulled back, usually combined with a keyword search to catch exact terms like product codes.
4. Generation: the model receives the question plus those passages, with an instruction to answer only from them and to cite the source.

Where these projects actually fail

Messy source documents

Scanned PDFs without text layers, five versions of the same policy with no indication of which is current, tables that lose all meaning when flattened into text. Cleaning and structuring the corpus is typically sixty percent of the work, and skipping it guarantees a system that confidently cites a document superseded in 2023.

Naive chunking

Splitting every 500 characters cuts sentences in half and separates a table header from its rows. Chunk on structure — sections, headings, logical units — and keep enough overlap that context survives the boundary.

No refusal path

If retrieval returns nothing relevant, the correct behaviour is to say so and offer a human contact. A system permitted to fall back on general knowledge will eventually give legal-sounding advice you never wrote, in your brand voice, to your customer.

Measuring whether it works

Build an evaluation set before you build the system: fifty real questions, with the correct answer and the document it comes from. Then measure retrieval accuracy — did the right passage come back — separately from answer quality. Without this split, you cannot tell whether a bad answer is a retrieval failure or a generation failure, and you will tune the wrong component.

• 90%+ target retrieval accuracy on your eval set
• 0 answers invented outside the corpus
• 100% answers carrying a source citation

What it costs to run

For a small to mid-sized US business, a well-built internal RAG assistant typically runs in the low hundreds of dollars per month: embedding costs are one-off per document, vector storage is inexpensive at this scale, and generation is billed per question. The dominant cost is not infrastructure — it is the initial effort to get your documents into a state worth indexing.

A grounded assistant that answers eighty percent of questions with citations beats an ungrounded one that answers a hundred percent, because the second requires you to verify every answer anyway.