How to organize documents for an AI knowledge base
Organizing documents for an AI knowledge base requires selecting the right material, identifying versions and preserving context. Putting every file in a folder does not solve information retrieval by itself.
In this exercise, we will prepare a small collection of internal procedures. The result is an inventory useful for search or a RAG application, without assuming every file is ready to answer questions.
1. Start with the expected questions
List what users need to discover: who approves a request, which fields are required or what happens when a registration is incomplete. Then identify the documents that actually answer those questions.
A file may matter to the organization without belonging in the collection’s first scope. Separating necessary content reduces the chance of retrieving irrelevant subjects or versions.
2. Create a source inventory
Record title, owner, version, status and access level. An approved, current source should be distinguishable from a draft or historical file.
| Field | Fictional example |
|---|---|
| Identifier | PROC-014 |
| Title | Materials request |
| Version | 3 |
| Status | Approved |
| Owner | Operations team |
| Access | Authorized team members |
The W3C’s data best practices address metadata, provenance and versions. Here we apply those ideas so a retrieved passage can be understood and found again.
3. Correct text before splitting it
Check whether headings, lists and tables were extracted legibly. A visually correct PDF may produce text with mixed columns or broken words.
Inspect several pages and compare them with the original. If extraction loses the connection between a value and its condition, retrieval may return a passage that was ambiguous from the start.
Remove duplicates or explicitly identify the current version. Do not delete a required history: keep it separate and marked so it cannot be confused with the current rule.
4. Split without destroying context
Smaller passages can make retrieval easier, but they must retain the information that makes a rule understandable. Separating an exception from its condition is one example of a poor split.
Microsoft’s document chunking guidance presents segmentation strategies. There is no universal size that removes the need to test your sources and questions.
Review these proposed chunks for a knowledge base.
Identify missing headings, references without context and rules separated from exceptions.
Preserve each source identifier and version in its passages.
Do not add rules absent from the original.
Suggest changes and explain which question each passage could help answer.Use the result to support review. Decisions about approved content remain with the person responsible for the procedure.
5. Preserve permissions during retrieval
A knowledge base should not make a restricted document accessible to everybody. Record who may consult each source and how the system will apply that condition during retrieval.
Do not rely only on an instruction telling the model not to disclose something. The application must control which passages can enter each user’s context.
6. Test updates and traceability
Choose a question, retrieve its passage and check that you can open the correct source. Then simulate a new version and verify how the collection stops using the previous one for current queries.
This small test reveals whether the project has a maintenance path beyond the initial file import. Record the outcome so the next update can follow the same procedure.
To connect this preparation with answer generation, read RAG in practice and keyword versus semantic search.
Frequently asked questions
What should I record about each document?
Record its identifier, title, version, status, owner and access permissions, preserving a reference that lets you find the source again.
Is there a universal ideal chunk size?
No. Test segmentation with your documents and questions while preserving rules, conditions and exceptions that need to be interpreted together.
Is a prompt enough to protect a restricted document?
No. The application must control which documents and passages each user can retrieve and receive as context.