RAG architecture for internal knowledge assistants: the design decisions behind grounded answers

This guide is for architects and engineering leads designing the retrieval pipeline behind an employee-facing assistant. It explains the decisions and their defaults. If you want the engineering delivered, see our enterprise knowledge and RAG service; for the packaged product view, see the AI knowledge assistant. Knowledge assistants are one of the AI steps covered in our AI automation guide for business operations.

In this guide

• Why retrieval decides the quality
• The pipeline decisions at a glance
• Ingestion and chunking
• Retrieval: keyword, vector or hybrid
• Permissions: filter before the model sees anything
• Freshness and deletion
• Grounded answers and refusals
• Evaluating a RAG system
• RAG, long context, fine-tuning or search: alternatives and selection criteria
• AI in the RAG pipeline
• What delivery record exists, and what does not
• Limits of this guide
• Frequently asked questions
• Next step

Why retrieval decides the quality

Retrieval-augmented generation, introduced by Lewis and colleagues in 2020, pairs a language model with an external index: the system first finds the passages relevant to a question, then asks the model to answer from them. For company knowledge this matters twice. The model was never trained on your contracts or procedures, and those documents change every week.

The consequence is easy to miss: when an assistant gives a wrong answer, the cause is usually upstream of the model. The right passage was never indexed, was split in the wrong place, ranked below noise, or filtered out. Swapping the model rarely fixes that. The decisions below are where quality is won or lost.

The pipeline decisions at a glance

Ingestion and chunking

Start with a source inventory rather than a crawl. For each system record the owner, the access model, the update frequency and whether it holds the current version of the truth. Duplicates and outdated copies are the most common cause of confident wrong answers, so decide which source wins before indexing anything.

Chunking is the decision with the largest effect on retrieval. Fixed-size windows are simple but cut procedures and tables in half. Structure-aware chunking splits on headings, list items or table rows and stores the heading path with each chunk, so a passage titled "Refunds > Damaged goods > Outside the EU" can be found and cited on its own. For long procedures, index small chunks for matching but hand the model the parent section, so the answer sees the whole step list. Keep document metadata on every chunk: source, owner, access rights, version and date.

Retrieval: keyword, vector or hybrid

Vector search finds passages that mean the same thing in different words. Keyword search finds exact matches: product codes, contract numbers, names and internal jargon that embeddings blur. Company questions mix both, which is why hybrid retrieval is the sensible default. Microsoft's documentation for its search service describes the common pattern: run full-text and vector queries in parallel and merge the two ranked lists with reciprocal rank fusion.

A reranker then reads each candidate together with the question and reorders them. It is slower than first-stage retrieval, so it runs only on the candidates, and it is often the cheapest single improvement to answer quality. Keep the final set short and diverse. Liu and colleagues showed in "Lost in the Middle" that models use information at the start and end of a long context better than information in the middle, so more context is not automatically better.

Permissions: filter before the model sees anything

An assistant must never quote a document the asker could not open. Enforce that in retrieval, not in the prompt: store access rights with each chunk, pass the asker's identity and groups into the search query, and drop anything they may not see before generation. Filtering after generation, or asking the model to withhold restricted content, leaks information through summaries and hints.

OWASP lists vector and embedding weaknesses as LLM08 in its 2025 Top 10 for LLM applications, including unauthorised access through the index, cross-tenant leaks in shared vector stores and poisoned content. Its mitigations match this design: fine-grained, permission-aware access to the vector store, validated ingestion from trusted sources, content classified by access level and immutable logs of retrieval. When an assistant can also act, not only answer, the controls in our AI agent security guide apply as well.

Freshness and deletion

Company knowledge changes, and the index must follow. Re-index on change events where the source system provides them and on a schedule where it does not. Store the last-updated date with each chunk and show it next to the citation, so a user can see that a policy passage is two years old. Deletion matters as much as updates: when a document is withdrawn or a person's access is revoked, the index must reflect it within the time your data policy promises, and retention of questions and answers needs its own rule.

Grounded answers and refusals

The generation step is a contract: answer from the passages, cite each claim to a passage, and say "I don't know" when the passages do not support an answer, pointing to the document owner. NIST's Generative AI Profile names confabulation, the confident statement of false content, among the core risks of generative AI; citations and refusals are how users check for it. For regulated topics such as HR, legal or finance, route answers to a draft-for-review queue instead of replying directly.

RAG, long context, fine-tuning or search: alternatives and selection criteria

Four criteria decide it: how often the knowledge changes, whether different people may see different documents, whether answers must be traceable to a source, and how many documents are involved. In practice the approaches combine: RAG for facts, a light fine-tune or instructions for format, and search as the fallback. Choosing which knowledge problem to solve first is part of an enterprise AI readiness assessment.

What delivery record exists, and what does not

• What exists. Netbase has delivered retrieval-augmented knowledge assistants, document AI and MLOps pipelines for clients that are not named; the anonymised RAG record describes the kind of system and Netbase's role. Delivery follows Netbase's security practices, including role-based access control, MFA for admin dashboards, TLS in transit and AES at rest, secure code review and vulnerability scanning.
• What does not. The record publishes no client name, corpus size, accuracy, adoption or time-saved figure, so none is offered here. The defaults on this page are design guidance, not measured results from that project.

Limits of this guide

• The defaults suit internal assistants over documents and records in one organisation; public chatbots, multi-tenant products and regulated advice need further controls.
• Research papers and vendor documentation are cited for methods, not as endorsements; retrieval quality depends on your content and must be measured on your questions.
• Personal data in the index brings data-protection duties that this guide does not replace; take legal advice where the GDPR or other privacy law applies.

Next step

Share the sources your people search today, who may see what and ten real questions, and we will book a solution review to sketch the retrieval design and its evaluation set. You can also see AI automation and agents or more Netbase insights.

Related services and solutions