Building an AI Question-Answering System: What Is Chunking, Why Does It Matter, and How Can You Keep AI from Getting the Facts Wrong?

Imagine asking an AI connected to your company's knowledge base, “Can I claim the eyewear benefit if I haven't passed probation yet?” The AI answers, “Yes, up to 3,000 baht per year.” But the company handbook says you must complete probation first.

In this example, the AI found the correct amount but missed the eligibility condition. This points to a problem with how the information was split before being given to the AI: the chunking was not good enough. We need to keep facts and their conditions together.

What is chunking, and why not just give AI the entire file?

RAG, or retrieval-augmented generation, finds information in documents for AI to read before answering. Chunking means dividing text into searchable pieces without changing the original content. The system then gathers chunks—pieces of text relevant to the question—and gives them to the AI to help it answer.

Giving AI the entire file can mix many topics together and make relevant information harder to find. Small but important details can get lost among the rest of the content, or be cut out altogether. Think of a mixed fruit juice: once everything is blended, you can barely distinguish the taste of each fruit.

How can cutting text at the wrong boundary change its meaning?

For example:

Section 8.1: Prescription Eyewear Benefit
The company reimburses up to 3,000 baht per year.
Only permanent employees who have completed probation are eligible.

If these lines are split apart, the system may find the amount without finding the condition. The AI then cannot determine eligibility: the missing condition is that the benefit applies only to permanent employees who have completed probation. If the prompt sent to the AI leaves that condition out, even a highly capable model cannot recover it from the provided text. The essential information is missing. Keep the heading and the rule together, and check the answer against the original document or include a reference to the source passage.

What are the different ways to do chunking?

You can combine several methods. For example, split by heading first, then break down only the sections that are too long.

Here are some common approaches:

1. Fixed-size: Split by length

Choose a fixed length and split the text at those intervals—for example, every 40 tokens. This is easy to do, but it can cut through a sentence or a table. Use the actual model's token counter: Thai words do not map to tokens at a fixed one-word-to-one-token ratio.

2. Fixed-size + overlap: Repeat text across chunk boundaries

Copy some text from the end of one chunk to the beginning of the next. This can help preserve information near a cut, but it does not connect related conditions that are several pages apart. It may also bring in unrelated information.

3. Recursive splitting: Work from larger boundaries to smaller ones

Split by heading or paragraph first, then divide further when the text is still too long. The limitation is that this method does not understand meaning. A single paragraph may contain several topics, while one topic may continue across multiple paragraphs. Depending on the source document, related information can still get separated.

4. Structure-aware: Use the document's structure

Use headings and tables to guide the split. But structure alone is not enough if values lose their labels. For example, a table entry containing just “3,000” has little meaning on its own. Keep the units and explanatory labels with it, including the relevant column heading.

5. Semantic chunking: Split when the topic changes

This approach uses a model to identify shifts in topic. It can help with long passages that have no headings, but using a model to decide where to split adds cost. If the original document already has clear headings, start with those. Asking an LLM to handle all the splitting can quickly inflate the budget.

6. Parent-child and contextual: Separate what you search from what AI reads

Parent-child retrieval searches for a small piece of text, the child, then expands it to the larger section, the parent, for the AI to read. Preserve the link to the original source, control the amount of text added, and check access permissions for the parent as well.

Contextual chunking adds the document title, heading, or an explanation drawn from the original context before building the search index, so it is clear what each chunk is about.

If you use an LLM to add an explanation, check that it has not invented any conditions. Keep the actual source text available for references; do not use generated text as a substitute for original evidence. Otherwise, the AI may make things up, or the rewrite may leave out a condition.

Choose what fits the document, not what sounds the most sophisticated

Document	Starting approach	What to check
Manuals or Markdown with clear headings	Split by heading, then divide sections that are too long	Parent headings, tables, and notes
Plain text with inconsistent structure	Split by paragraph, adding only as much overlap as needed	Conditions near chunk boundaries
Contracts or policies that require reading several paragraphs	Retrieve short passages, then add related sections	Clause numbers, document versions, and access permissions
Scanned or multi-column PDFs	Fix text extraction and reading order first	OCR errors, mixed-up columns, and broken tables

These are only starting points. You still need to check the actual documents.

A lesson from a Thai tourism PDF I worked with

While experimenting with a Thai tourism PDF, I found misplaced vowel and tone marks, along with interference from colors and watermarks. I compared the outputs and had a person check for spelling errors. This was one experience, not a formula for every PDF.

Splitting the text again cannot restore words that were read incorrectly. Markdown makes the structure easier to see, but tables still need headings. Read more about my PDF experience (in Thai).

How large should chunks and overlap be?

There is no fixed setting. 300–500 tokens with 10–20% overlap are starting values to experiment with, not a standard or measurements from my own system. Consider these questions:

1. Does each chunk preserve the main point and all relevant conditions?
2. Does the total text, including the question and answer, fit within the model's token limit?
3. Does overlap add useful information, or just repeat the same words?
4. A question asking for a name or a number may work with small chunks. But a question comparing several conditions may require retrieving multiple chunks or expanding the retrieval context.

Conclusion: Getting AI answers right depends on chunking, not just a capable model or a successful search

RAG is not intended to read the entire document every time you ask a question. The AI should read the information the system retrieves and sends to it. That means where you draw a chunk's boundaries affects the boundaries of the answer.

My lesson from working with PDFs in the pipeline is this: If the text was read incorrectly because of OCR errors, even excellent splitting cannot fix it. And if the text was read correctly but a condition was cut off or left out during chunking, the answer can still be wrong. Both the quality of the source text and the craft of preserving meaning in each chunk matter.