When companies first tried large language models at work, they hit an obvious wall. The model knew a great deal about the world and nothing about the company. It could not tell you last month's price, this customer's terms or the name of the plant manager. Retrieval-augmented generation, or RAG, became the standard answer. Fetch the relevant passages from company documents, place them in the prompt, and let the model answer from them with citations.
RAG works, and it is now underneath most enterprise chat assistants, search tools and support bots. But teams that run it in production keep meeting the same failures: answers built from an outdated document, answers that miss a fact sitting in a different system, answers that are correct and still useless because nobody acted on them. This article explains how RAG for enterprise works, where it breaks, what the research says, and what has to sit on top of retrieval before AI can be trusted with company work.
Key takeaways
- RAG solves a knowledge problem: it lets a model answer from your documents with sources.
- Inside companies, failures come from stale versions, facts spread across systems, missing records of what was decided, and no link to goals or owners.
- Better chunking, hybrid search and reranking reduce retrieval errors a lot. They do not add judgement, ownership or proof.
- Treat RAG as one layer. Above it you need context about the company, a view of what matters, and a loop that checks work was done.
What is RAG, and why did enterprises adopt it?
Retrieval-augmented generation is a method where a language model answers using passages retrieved from an external store at the moment of the question, rather than relying only on what it learned in training. The term comes from a 2020 paper by Patrick Lewis and colleagues, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, which described combining a model's "parametric memory" with "a dense vector index of Wikipedia" that it could search.
Enterprises adopted the pattern for three practical reasons. It avoids retraining a model every time a document changes. It lets answers cite their sources, so a person can check them. And it keeps company data in the company's own store, retrieved only when needed, which makes access control possible. For questions like "what is our returns policy for distributors?" RAG is often the right tool, and it is far better than asking a model that has never seen the policy.
How does a RAG pipeline work?
Most enterprise RAG systems follow the same five steps.
- Ingest. Documents, emails, tickets and pages are collected from connected sources.
- Chunk. Each document is split into passages of a few hundred words.
- Embed and index. Each passage becomes a vector that captures its meaning, stored in an index, often alongside a keyword index.
- Retrieve. When a question arrives, the system finds the passages most similar to it, usually the top five to twenty.
- Generate. The model writes an answer using only those passages and cites them.
Every step is a place where meaning can be lost. Chunking cuts a sentence away from the heading that explains it. Similarity search ranks by wording, not by authority or date. The model sees five passages and nothing else. The answer is as good as those five passages, and no better.
Where does RAG for enterprise fail inside companies?
Public benchmarks for RAG tend to use clean collections like Wikipedia, where each fact appears once and stays true. Company data is the opposite. Here are the failure modes that show up again and again.
- Stale versions win. Five drafts of a price list exist. The oldest one has the clearest wording, so it ranks highest. The answer quotes a price that changed two weeks ago.
- Decisions live outside documents. The new terms were agreed in an email reply or on a call. No document was updated. Retrieval finds the old contract and answers confidently.
- Facts are split across systems. The order is in one tool, the capacity plan in a spreadsheet, the customer's complaint in chat. No single chunk holds the answer, so none is retrieved.
- Chunks lose their context. A passage that says "revenue grew 3%" means nothing without the company and quarter it refers to.
- Aggregation questions break. "Which customers are at risk this month?" needs a view across hundreds of threads. Top-k retrieval returns a handful.
- Absence is invisible. The follow-up that was never sent produces no text. RAG cannot retrieve what does not exist.
- No sense of priority. Retrieval ranks passages by similarity to a query. It has no idea which of the company's open situations matter to its goals this quarter.
The first six are retrieval quality problems, and engineering can reduce them. The last one is different in kind. It is a question about what the company is trying to do, and no amount of better search answers it.
What does the research show about retrieval limits?
Several well-known studies make these limits concrete.
Long inputs are read unevenly. In Lost in the Middle, published in Transactions of the Association for Computational Linguistics in 2023, Nelson Liu, Percy Liang and colleagues found that model performance "is often highest when relevant information occurs at the beginning or end of the input context" and "significantly degrades" when the relevant passage sits in the middle. Retrieving more passages does not guarantee the model uses them.
Chunking removes meaning. Anthropic's September 2024 post on contextual retrieval stated that "traditional RAG solutions remove context when encoding information." Its example was a chunk reading "The company's revenue grew by 3% over the previous quarter," which is useless without knowing which company and which quarter. Adding context to each chunk before indexing, combined with keyword search and reranking, "reduced the top-20-chunk retrieval failure rate by 67%" in Anthropic's tests. That is a large gain, and it also shows how often plain retrieval missed.
Connecting facts is hard. Microsoft Research, introducing its graph-based approach in February 2024, wrote that "Baseline RAG struggles to connect the dots", which happens "when answering a question requires traversing disparate pieces of information."
Retrieval reduces errors without removing them. A 2024 Stanford study of commercial legal research tools built on retrieval found that they hallucinated in 1 out of 6 or more benchmarking queries. The authors wrote that "even RAG systems are not hallucination-free," while noting they did better than general chatbots.
Answers without learning repeat mistakes. MIT NANDA's 2025 report on enterprise AI concluded that "Most GenAI systems do not retain feedback, adapt to context, or improve over time," and named this learning gap as the main reason pilots stall.
RAG vs long context vs memory vs company intelligence
Teams often treat these as competing options. They solve different problems, and a working system usually needs several of them.
| Approach | What it solves | What it leaves open |
|---|---|---|
| RAG | Gives the model the right passages from a large store, with citations | Which version is current, facts split across systems, priority |
| Long context | Lets the model read a whole set of documents at once | Cost, uneven attention to the middle, still no notion of goals |
| Memory layer | Keeps facts and past interactions across sessions | What those facts mean for today's work |
| Context graph | Links people, customers, promises and deadlines | Deciding what to do, and checking it was done |
| Company intelligence | Weighs all of the above against the company's goals, assigns one owner and a date, closes on proof | Depends on the layers below being sound |
Anthropic's guidance on context makes a related point: for knowledge bases smaller than 200,000 tokens, its contextual retrieval post suggested including the whole knowledge base in the prompt with no retrieval at all. Retrieval is a tool for scale. It is not a theory of what a company needs to know. The memory vs context vs intelligence article lays out the three layers in more depth, and company brain explained covers what a memory layer can and cannot do.
What has to sit on top of retrieval?
If retrieval is the first layer, four more are needed before AI can be trusted with company work.
- Authority and recency. The system must know that a signed order outranks a draft, that an email from the buyer outranks an internal guess, and that last Tuesday's message outranks last month's file.
- Relationships. Customers, orders, people, lines and deadlines connected, so a change in one surfaces its effect on the others. This is what an enterprise context graph provides.
- Goals. A view of what the company is trying to achieve this quarter, so the system can tell which of a thousand facts deserve attention.
- Ownership and proof. Each situation assigned to one person with a date, and closed only when evidence shows it happened.
Anthropic's engineering team describes the craft of choosing what goes into a model as context engineering, with the aim of finding "the smallest possible set of high-signal tokens" for the task. Retrieval is one way to fill that context. Deciding what is high-signal for a company requires knowing what the company is trying to do. See what is context engineering for the full picture.
Common mistakes with RAG for enterprise
- Indexing everything. Old drafts, duplicates and abandoned folders compete with current documents. Curate sources before indexing them.
- Ignoring dates and authority. Pure similarity ranking treats a 2022 draft and a signed 2026 order as equals.
- Testing on easy questions. Demos ask questions with one clean answer in one document. Production questions need facts from three systems.
- Treating citations as truth. A citation proves the answer came from a document. It does not prove the document is current.
- Skipping the conversations. Many decisions live in email and chat. A RAG system over documents alone misses them.
- Stopping at the answer. The answer appears, and nobody owns what happens next.
How to improve a RAG system you already have
- Collect fifty real questions.
Take them from the people who use the system, including the ones it got wrong. Write the correct answer and its source next to each.
- Classify each failure.
Wrong version, fact in another system, lost context, aggregation, or missing record. The mix tells you where to spend effort.
- Fix retrieval first.
Add context to chunks, combine keyword and semantic search, add reranking, and weight results by date and authority.
- Add the conversations.
Connect email, calendar and chat read-only, so decisions made outside documents are visible.
- Add a layer that knows the goals.
For questions nobody asks, such as "which promise is slipping?", retrieval is the wrong starting point. Add a system that checks open work against goals.
- Close the loop.
For any answer that leads to action, record an owner, a date and the proof that it happened.
How to measure whether retrieval is enough
| Measure | What it tells you |
|---|---|
| Answers citing the current version | Whether recency and authority are handled |
| Questions needing two or more systems answered correctly | Whether facts are being connected |
| Problems found before someone asked | Whether the system goes beyond retrieval at all |
| Situations closed on proof by their date | Whether answers turn into done work |
If the first two numbers are good and the last two are near zero, your retrieval works and your company still lacks the layer above it.
How GeniOS approaches retrieval and RAG for enterprise
GeniOS treats retrieval as a starting point rather than the product. It reads the places where a company's work happens, such as email, calendar, chat and the business tools you already use, read-only to start. It forms the company's goals from its own emails, decks and meetings, and checks every team's pending work against them.
What reaches a person is intelligence: one situation, what to do, why, one owner and a date. The owner answers Delegate, Done, Later or Not relevant. Delegate hands work to a person or an AI agent, and nothing is sent until a human approves.
An item closes only on proof, such as the email that went out, and GeniOS learns from each outcome. That learning is the part retrieval alone never gets: a record of which situations mattered and how they turned out.
Worked example: the right chunk, the wrong answer
The names, dates and numbers in this example are made up to explain the idea. Vaultex makes safety footwear in Greater Noida. Bata wants 12,000 pairs of S3 safety shoes at $12 a pair, about $144,000, on 60-day payment terms, with a vendor review on Friday. Reebok samples run on Line 2. Adidas has a vendor audit coming and has asked for IS 15298 test reports.
Harsh in sales asks the company's RAG assistant: "Can we hold $12 a pair for Bata?" The assistant retrieves the costing sheet from last month and the original quote email, and answers: yes, the margin at $12 is within policy. It cites both documents. Every citation is real.
The answer is still wrong, and here is why, failure by failure:
| What happened | Failure mode |
|---|---|
| The steel toe cap supplier raised prices 8% in an email on Monday. The costing sheet was never updated. | Decision outside documents, stale version wins |
| The 60-day terms add financing cost that the costing sheet does not include. | Facts split across systems |
| Anushka in production has Line 2 booked for Reebok samples, so the order has no confirmed slot. | Facts split across systems |
| Nobody asked about the Adidas test reports, which are still unsent. | Absence is invisible |
A layer that knows the goals reads the same sources differently. It sees that the Bata order and the Adidas audit both matter this quarter, connects the supplier email to the quote, and raises three owned situations: Harsh re-prices the quote by Thursday, Anushka confirms a slot on Line 2 or another line, and Anushka sends the IS 15298 reports before the audit. Each closes when the email or plan entry exists. The retrieval was accurate. What was missing was everything after it.
Conclusion
RAG for enterprise was the right first step. It taught models to answer from company facts and to show their sources. Inside a real company, though, the facts disagree, change by email and live in different systems, and the most important situations are the ones nobody asks about. Improve retrieval where it fails, then add what retrieval cannot do: authority, relationships, goals, ownership and proof. For the difference between search and the layer above it, read enterprise AI search is not company intelligence, and for why answers alone break in production, see why AI agents fail in production. To see that layer on your own data, book a call.
Sources
- arXiv, Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 22 May 2020
- arXiv / TACL, Liu et al., Lost in the Middle: How Language Models Use Long Contexts, July 2023
- Anthropic, Introducing Contextual Retrieval, 19 September 2024
- Microsoft Research, GraphRAG: Unlocking LLM discovery on narrative private data, 13 February 2024
- Stanford HAI, AI on Trial: Legal Models Hallucinate in 1 out of 6 (or More) Benchmarking Queries, 23 May 2024
- MIT NANDA, The GenAI Divide: State of AI in Business 2025, July 2025
- Anthropic, Effective context engineering for AI agents, 29 September 2025