RAG AI: how artificial intelligence works on company knowledge and when it pays off

RAG explained in simple terms

A large language model (a program that understands and writes text) knows a lot about the world, but nothing about your company: your current price list, warranty terms or what was agreed with a particular customer. Asked about them, it will either refuse or guess. The latter is worse. RAG AI solves this problem simply: before the model answers, the system searches the company’s documents for passages matching the question and gives them to the model as material.

The name comes from a research paper by Patrick Lewis and co-authors published in May 2020. The terms RAG LLM and LLM RAG mean the same thing, i.e. RAG built on a large language model (LLM). In companies people also simply talk about an AI knowledge base or AI on company knowledge.

RAG is often part of an AI agent. An agent that answers customer questions or prepares a briefing note before a meeting treats the knowledge base as one of its tools, just like a search engine or a spreadsheet. We describe how an agent uses such tools and what it costs in our article AI agent in marketing and PR, and we have gathered the other sections of the topic in guide to artificial intelligence for business.

How RAG works under the bonnet

You can leave the technical details to IT. It is worth understanding the four stages, though, because an error can arise at any of them:

1. Preparing the documents. Files are split into shorter passages, for example paragraphs or sections of a contract.
2. Capturing meaning. Each passage is turned into a sequence of numbers describing its meaning (a so-called embedding) and goes into a vector database, which searches by meaning rather than by exact words. That is what semantic search is: the question “how much does delivery to the Czech Republic cost?” will find a passage about “international transport charges”, even though it contains neither the word “delivery” nor “Czech Republic”.
3. Retrieval. When someone asks a question, the system selects a few of the best-matching passages.
4. Answer. The model writes an answer based on these passages, and in a well-built system it also states which document the information comes from.

Who RAG is for in a company, and in what situation

RAG makes sense in a company where people search documents for answers every day, or keep asking colleagues at the next desk the same things. Here are typical situations:

• Sales. Sales reps search price lists and specifications, as well as terms of trade, and the assistant answers by pointing to the document.
• Customer service. A RAG chatbot, i.e. a chatbot built on company documents, answers questions about deliveries, returns and warranties. We write about when such a chatbot helps and when it drives customers away in our guide to AI in customer service.
• PR. The communications department searches its archive of statements, positions and media responses in order to speak with one voice. We show, with data, how to write statements that are easy to find and quote in our article the press release in the age of AI.
• HR and internal communications. Employees ask the assistant about policies and benefits or procedures instead of writing to HR.

RAG, fine-tuning or long context: which to choose

RAG is not the only way to “teach” AI your company’s knowledge. Model tuning ( fine-tuning) changes the model itself. Long context means pasting entire documents into the conversation, which works with a few files, but with hundreds of documents that change every month it quickly becomes expensive, slow and hard to repeat. Each method suits a different situation.

Does RAG eliminate hallucinations?

No. It reduces them, but does not eliminate them. In 2024 Stanford researchers tested legal tools whose makers promised “hallucination-free” answers thanks to RAG, and found that the tools studied were wrong or made things up in 17–33% of cases, i.e. less often than general chatbots, but still too often for uses where every word counts.

Where do these errors come from? The system may retrieve the wrong passage. A document may be out of date, and the model may combine information from two places incorrectly. We break down the causes and ways of reducing fabrications in a separate article on AI hallucinations.

How to implement RAG in your company step by step

This sequence works both with off-the-shelf tools and with a solution built from scratch. If RAG is to be part of an agent, also read our guide on how to build your own AI agent.

1. STEP 01Choose one area of knowledge

   For example, terms of trade and price lists. Not “the entire company drive”.

2. STEP 02Collect real questions

   Write down 30–50 questions people really ask, together with the correct answers. This will be your test.

3. STEP 03Tidy up the documents

   Remove old versions, mark the dates from which documents apply and split large files into logical parts.

4. STEP 04Set permissions

   Who can ask about what? A sales assistant should not see payroll data.

5. STEP 05Build and test

   Run the entire list of questions through the system and count how many answers are correct, how many have the right source and how many are simply wrong.

6. STEP 06Appoint a knowledge owner

   One person makes sure the new price list replaces the old one in the database on the day it takes effect.

How much AI on company knowledge costs

The cost depends on whether you use an off-the-shelf tool or build your own system. In both cases the most expensive element is people’s time. Organising documents and testing eat up most of it, especially in companies where the same price list circulates in five versions across sales reps’ inboxes and nobody remembers which one is current.

The quality of documents determines the quality of answers

RAG won’t fix messy documents. It will expose the mess faster. Three versions of the same policy, an undated price list, an offer with an expired discount: the system will find them all, and the model will answer on the basis of whichever version best matches the question, not the one currently in force.

That is why a RAG rollout is half a knowledge-organisation project. The whole company benefits, because a single source of truth on prices and procedures makes it easier internal communication including in departments where nobody uses AI.

When RAG pays off, and when it isn’t worth the effort

RAG pays off when knowledge is written down in documents, changes often and many people look for it every day. A good moment is the period before a price list change, a new offer launch or the arrival of a larger group of new staff, because then the knowledge base has users from day one.

It isn’t worth the effort when there are only a few documents and they rarely change. It is enough to paste them into an assistant, as we show in our article on how an AI assistant works in a company. And don’t start if the knowledge exists only in people’s heads. It has to be written down first. We discuss choosing a tool to fit the company’s ecosystem in our comparison ChatGPT, Gemini or Copilot for business.

Security, permissions and personal data

An AI knowledge base sees as much as you put into it. If contracts with personal data or salary information end up there, anyone with access to the assistant can ask about them, and without any ill intent, with an ordinary question such as “how much does a regional manager earn?”. So set permissions at document level, not just for the system as a whole.

Also check where the provider stores the documents, whether it uses them to train models and whether you have a data processing agreement with it. We write more about the risks in our piece on AI risks and ethics. Want to build an assistant on your company’s knowledge with control over sources and permissions? Talk to Commplace® AI agency.

Where RAG projects most often fall apart

Read next