What Is RAG? Why AI Needs an Open-Book Exam

Modern AI can feel ridiculously smart. It writes, summarizes, reasons across languages, and answers questions from almost any field.

Then you ask something simple and specific:

• How many vacation days does our company policy allow?
• What changed in yesterday's sales report?
• What happened in the news a few minutes ago?

Suddenly, that smart assistant either says it does not know or gives you a confident answer that is completely wrong. The second case is commonly called a hallucination.

The reason is straightforward. A language model does not automatically know your private documents, and the knowledge inside the model does not update itself whenever the world changes.

RAG is one way to bridge that gap.

TL;DR

RAG stands for Retrieval-Augmented Generation. Instead of asking a language model to answer only from what it already knows, a RAG system first retrieves relevant information, adds that information to the model's context, and then asks the model to generate an answer.

Think of a normal LLM as a brilliant student taking a closed-book exam. An LLM with RAG is the same student taking an open-book exam with a small set of relevant pages on the desk.

This does not guarantee a correct answer. It gives the model better material to work from and can make the answer easier to verify.

Why a smart model still does not know your information

Imagine an LLM such as ChatGPT, Gemini, or Claude as a gifted student who has read an enormous library.

The student may know mathematics, physics, Thai, English literature, and thousands of other topics. But two limitations remain:

1. Its knowledge has boundaries. New events or details not represented in the model's knowledge may be missing.
2. It has never read your private notebook. Internal policies, spreadsheets, chat history, product manuals, and company reports do not appear in its context automatically.

I ran into this problem while researching GPUs for a local AI setup.

Around early 2025 , I was looking at the RTX 5070 Ti, RTX 5080, and RTX 5090 for an AI Studio Worker on a limited budget. Reviews were still limited, and the AI assistant I used could not give me information I trusted about the RTX 50 Series.

I ended up arguing with the assistant about whether the products had already launched. The answers felt more like guesses than evidence I could use for a purchase decision. In the end, I went back to Search. It cost time and, honestly, a bit of patience.

The same issue becomes more serious with company knowledge. A model cannot analyze a document it has never received.

Fine-tuning works for certain problems, but if you're dealing with new or frequently changing info, it shouldn't be your first choice. You don't send someone through years of medical school just to teach them that Tylenol isn't the only brand of paracetamol.

A more direct option is to let the student look up the current information before answering.

RAG in three steps

The full name sounds technical, but the basic flow is simple.

1. Retrieval

The system searches for documents or passages that appear relevant to the question.

2. Augmentation

It places the retrieved material into the context sent to the language model, together with the original question and instructions.

3. Generation

The model reads the question and supplied context, then writes an answer.

That is the open-book exam analogy. A standalone model relies on what is already inside it. A RAG system attempts to bring the right pages to the desk first.

The word “attempts” matters. Retrieval can return the wrong passage. The source document can be outdated. A prompt can still allow unsupported interpretation. RAG is not a magic accuracy switch.

What happens after you ask a question?

From a user's point of view, the flow looks like this:

[Your question]
      |
      v
1. Retrieval
   Find relevant passages or documents
      |
      v
2. Augmentation
   Add the retrieved information to the question
      |
      v
3. Generation
   Ask the LLM to answer from that context

Some tools search private documents. Others search public web sources. More complex systems can use several sources together.

In every case, the useful questions go beyond “Which model are we using?” We also need to ask:

• Where did this context come from?
• Is it current and relevant?
• Is the user allowed to access it?
• Can a human inspect the source behind the answer?

This is also how I think about using RAG while developing NEXT4I. Model capability matters, but so does the path that brings information to the model.

What can RAG improve?

Working with changing information

A system can update its knowledge source and rebuild or refresh an index through its ingestion pipeline. How quickly the new information becomes available depends on the actual extraction, validation, indexing, and deployment process.

Reducing unsupported answers

Instructions can tell the model to answer only from the supplied context and to say when the information is missing. This can reduce the space for guessing, but it does not eliminate hallucinations.

Making answers easier to inspect

If the system preserves metadata and citations, the answer can point back to a source document or passage. A person can then check whether the summary is faithful to the evidence.

Applying access controls during retrieval

Putting documents into a vector database does not make access control automatic. Permission filters, metadata filters, tenant isolation, and audit logs have to be designed into the retrieval path.

RAG by itself is not a security guarantee. If retrieval ignores permissions, the model may receive context that the user should never have seen.

Where can RAG be useful?

Common use cases include:

• HR assistants that retrieve leave and benefit policies a specific employee can access
• Customer support tools that find the right manual for a product and issue before drafting a response
• Enterprise search and second-brain systems that locate contracts, meeting notes, and historical reports
• Medical or legal assistants that retrieve and summarize evidence for expert review, rather than replacing professional judgment

Connecting an LLM is only one part of each use case. The underlying knowledge still needs to be organized, searchable, current, and permission-aware.

Start with the knowledge, not only the prompt

For me, taking RAG seriously means paying attention to the details of knowledge organization and retrieval.

If your documents are duplicated, outdated, poorly extracted, or missing a clear source of truth, a stronger model will not automatically fix the pipeline. The same problems will keep coming back.

RAG does not directly rewrite what the model knows. It brings the model to a more appropriate bookshelf before asking for an answer.

So before asking, “How do we train our own model?” it may be more useful to ask:

“Is our knowledge organized well enough for a system to retrieve and verify it?”

Reliable RAG does not begin with the prompt alone. It begins with the quality of the information and the path used to find it.

---