RAG explained for business teams, retrieval, grounding and refusals

A language model on its own answers from what it learned in training, which knows nothing about your contracts. RAG adds a step before the answer: find the passages in your documents that are relevant to the question, and give only those passages to the model to answer from. Done well, the answer is grounded in your documents and can point to the exact page it used.

Before any question is asked, each document is read and split into passages. For a PDF that means extracting the text of every page (running OCR on scanned pages), keeping track of which page each passage came from. Spreadsheets keep their sheet and cell ranges; slide decks keep their slide numbers.

Why it matters to you. If the page number is lost at this stage, no later step can give you a page citation. Poor OCR here means the right clause can never be found.

Step two, indexing

Each passage is stored in a search index. Modern systems combine two kinds of search: keyword matching, which is good at exact terms like "Section 9.3" or "indemnification", and semantic matching, which finds passages that mean the same thing in different words, like "end the agreement early" and "termination for convenience".

Why it matters to you. Keyword search alone misses paraphrases. Semantic search alone can miss exact clause numbers and defined terms. Contracts and policies need both.

Step three, retrieval

When you ask a question, the system searches the index and ranks the passages most likely to contain the answer. Only the top passages are passed on.

Why it matters to you. If retrieval picks the wrong passages, the answer will be wrong no matter how capable the model is. Most poor answers trace back to this step, not to the writing.

Step four, grounded generation

The model receives your question and the retrieved passages, with an instruction to answer only from those passages and to tag each sentence with its source.

Why it matters to you. This is where "grounding" happens. A well-built system forbids the model from adding facts that are not in the passages. A loosely built one lets general knowledge leak in.

Step five, checking the citations

This step is often skipped, and it is the one that makes answers safe to use. After the answer is written, the system checks each cited passage: does it exist on the cited page, and does it contain the quoted text? Sentences that fail the check are removed.

Why it matters to you. Models occasionally attach a plausible page number to a sentence. A code check catches this before you see it. In ChatDocument every cited passage goes through this check before the answer is shown.

Step six, refusing when nothing fits

If retrieval finds nothing relevant, or every sentence fails the citation check, the right output is not a best guess. It is a plain statement that the documents do not answer the question, with the closest passages that were checked.

Why it matters to you. A refusal tells you something true about your documents. A guess hides it.

• Chunk or passage. A piece of a document, usually a paragraph or a few, stored with its location.
• Embedding. A numerical representation of a passage's meaning, used for semantic search.
• Context window. How much text the model can read at once. RAG exists partly because whole libraries do not fit.
• Grounding. Restricting the answer to the retrieved passages.
• Hallucination. A fluent statement not supported by any source. Grounding and citation checks are the defences.
• Space or collection. The set of documents a question is allowed to search. Restricting it is how permissions work in practice.

When a vendor says their tool "uses RAG", these questions tell you how well:

1. 1Does every sentence of an answer cite a document and page, or only the answer as a whole?
2. 2Are citations checked against the page text before the answer is shown?
3. 3What happens when the documents do not answer the question? Ask a question you know is not covered.
4. 4Is search keyword plus semantic, or semantic only? Try a question that uses an exact clause number.
5. 5How are scanned PDFs, spreadsheets and slide decks handled, and what do their citations point to?
6. 6Can answers be restricted to a selected set of documents for each team?
7. 7Are your documents used to train models, and what do the model provider terms say?
8. 8How are documents and the index deleted, including from backups?

Engineering teams can build a RAG pipeline from open components. The first version is quick; the hard parts come later: OCR quality, keeping page locations through every step, combining keyword and semantic search, citation checking, permissions, syncing drives and deleting data properly. A service that already does these lets a team focus on its documents instead of the plumbing.

ChatDocument is RAG as a finished product: upload or connect documents, ask in a team space, get cited answers. Business and Enterprise plans add a REST API for upload, ask and extract, and a widget restricted to one space. More on that on the RAG as a service page and the document question answering API page.

• Test retrieval with paraphrased questions and exact clause numbers.
• Test refusals with questions your documents do not cover.
• Click citations and read the highlighted passages.
• Read why AI answers need citations for a checking routine your team can adopt.

Suppose an HR space holds the employee handbook, the travel policy and the benefits guide, and someone asks "How many days of parental leave do we offer?" The handbook was read page by page when it was added. Retrieval finds three passages mentioning parental leave, two in the handbook and one in the benefits guide. The model writes a two-sentence answer from those passages, tagging each sentence with its page. The citation check confirms both passages exist and contain the quoted words. The person sees the answer, clicks the citation and reads the policy paragraph on page 22.

If the same person asks about a sabbatical, and no document mentions one, retrieval returns only loosely related passages about leave. None supports an answer, so the reply says the documents do not cover it and lists the passages checked.