Part of
Enterprise AI chatbot
Implementation

How to Build a Knowledge Base Chatbot Your Employees Will Actually Use

SharePoint, Google Drive and Notion sources feed an orange answer step that links the exact section, delivered in Slack and Microsoft Teams.

To build a knowledge base chatbot your employees will use, give every source document a named owner. Load each document with its permissions and effective date. Have the chatbot answer only from the passages it retrieves, with a link to the exact section. Send every question it cannot answer from a source to the person who owns that topic. Put the chatbot in Slack or Teams, where people already ask each other, and test it on real questions before anyone relies on it.

Retrieval from your own documents reduces wrong answers, but it does not remove them. A Stanford and Yale study published in the Journal of Empirical Legal Studies in April 2025 tested AI legal research tools. The tools came from LexisNexis (Lexis+ AI) and Thomson Reuters (Westlaw AI-Assisted Research and Ask Practical Law AI). They use retrieval-augmented generation: they look up passages in their publishers' own legal databases and write the answer from them. The study found that each tool still hallucinated, giving a wrong or unsupported answer, between 17% and 33% of the time. An employee who acts on a wrong policy answer once has little reason to ask the chatbot again. The design work below goes into the cases where the right reply is a citation, a warning or a handoff.

How a RAG knowledge base chatbot works, in five parts

A knowledge base chatbot is often called a RAG chatbot, after retrieval-augmented generation. It retrieves passages from your documents, then writes its answer from them. It has five parts. The model writes the reply in the fourth, and the first three decide what it has to work with.

Five parts
What a knowledge base chatbot is made of
Tap a part to see what it does and the failure it prevents.
1. Sources
What it does
Holds only documents with a named owner, such as HR for the handbook, the IT service desk for runbooks and Finance for expense rules. Drafts, meeting notes and personal folders stay out.
Failure it prevents
An old version answering with a citation. If the 2025 and 2026 travel allowances both sit in the index without dates, the chatbot can retrieve either one.
The model writes the reply in part 4. What parts 1 to 3 hand it decides what it can get right.

The sources are the documents you trust, each with an owner. Ingestion is the step that loads them. It splits each document into passages and keeps each passage's heading, owner, effective date and access list attached. The index lets the chatbot find a passage by meaning and by exact terms. When someone asks a question, the chatbot retrieves only passages that person may open, writes an answer from them and links the section it used. A final check confirms every claim in the draft appears in a retrieved passage. When one does not, the chatbot hands off.

Ovidius built two production chatbots on this pattern, both answering customers rather than employees. Pinkcube's agent answers from a knowledge base built from Pinkcube's own resolved support conversations. Those conversations are filtered before the agent can use them and stored in Supabase pgvector. Gemma, built for Live Platforms' gemstone auction sites, answers from a knowledge base that refreshes weekly from both sites. Our enterprise AI chatbot page describes how it works.

Which documents should a knowledge base chatbot start with?

Start with the documents behind the questions your teams answer most often, and only those with someone responsible for keeping them current. Everything else can wait for the second round.

SourceOwnerRefresh
HR handbook and policiesHROn every policy change, with the effective date
IT runbooks and how-to pagesIT service deskWhen the system changes, and a quarterly review
Expense, travel and purchasing rulesFinanceOn every change, old versions marked superseded
Product and pricing documentsProduct or sales operationsWith each release
Resolved tickets and chatsSupport leadWeekly, filtered for personal data and wrong answers
Drafts, meeting notes, personal foldersNobodyLeave out

Duplicates and old versions are harder to catch than missing pages, because the chatbot answers from them with a citation. If the 2025 and 2026 travel allowances both sit in the index without dates, the chatbot can retrieve either one. Mark superseded documents, or remove them, before the first ingest. Our article on enterprise knowledge management when AI agents are the readers covers the documentation side in more depth.

How to keep document permissions intact

A knowledge base chatbot must never show an employee a passage they could not open themselves. The safe way to enforce that is to copy each document's access list into the index and filter every search by who is asking. A restricted passage is then never retrieved for that person in the first place.

A chatbot that respects permissions will still show a file that was shared too widely, as long as the asker can technically open it. Microsoft's own deployment guidance for Copilot, updated in July 2026, says Copilot and agents retrieve data through Microsoft Graph. They respect existing permissions, sharing settings and policies. The same page notes that SharePoint sets sharing to the most permissive option by default. It also lists reports for finding overshared sites before Copilot reaches them. Any knowledge base chatbot that reads SharePoint has the same problem, so audit sharing on the sites you plan to index before the first ingest. Our guide to data governance for AI covers how to run that audit.

How retrieval choices change accuracy

When retrieval returns the wrong passage, or none, the model has nothing correct to answer from. Three choices decide what retrieval returns.

  • Passages that keep their context. A passage cut from the middle of a policy loses the heading that says which country or which year it applies to. Keep the document title and section heading with every passage.
  • Search by meaning and by keyword together. An employee asking whether they can expense a client dinner will not use the words in a policy filed as "T&E reimbursement guidelines". Semantic search finds that match; keyword search catches exact terms such as product codes and form numbers.
  • A ranking step before the answer. Retrieve more candidates than you need, then rank them and keep the best few.

Anthropic measured the effect of these choices on its own benchmarks, published in September 2024. It added a short piece of document context to each passage and combined semantic and keyword search. That cut the share of relevant passages missing from the top 20 results by 49%, from 5.7% to 2.9%. Adding a reranking step took the reduction to 67%, down to 1.9%. Your documents will produce different numbers, which is why you test on your own questions. For the wider discipline behind this, read our article on context engineering.

When should a knowledge base chatbot say it does not know?

It should say so unless the reply rests on a current passage the asker may open. A fixed rule decides this before the model writes anything, so the chatbot never has to judge its own confidence. Pick a question to see the five outcomes.

Pick a question
Answer, flag, route or withhold?
Matching documents in the index
    Chatbot
    • Client dinner: answers with the source. One current passage answers the question, and the asker may open it.
    • Per diem, Germany: answers, and flags the old version. Two versions match. The chatbot answers from the one in effect and says the other is superseded.
    • VPN for contractors: shows both, asks the owner. Two current sources disagree. The chatbot does not pick one; it shows both and asks the owner to settle it.
    • Dogs at the office: says it is not covered. Nothing in the index covers it, so any answer would be a guess.
    • Salary bands: withholds, points to hr. A matching document exists, but the asker may not open it, so the search filter drops it before the model sees anything.
    Illustrative questions and documents. The rule decides the reply before the model writes it, so a missing, conflicting or restricted source never turns into a confident guess.

    The check after the draft matters as much as the rule before it. On IES Limited's daily MI alerts, the database calculates the figures and the model only writes the sentences around them. A verification step then names any figure the model was not handed. Once, the model put two figures into a brief that were not in its input. The brief said so at the top and asked for verification before it went out. The IES case shows the full setup. A knowledge base chatbot needs the same check for policy claims: if a sentence in the draft has no passage behind it, the sentence goes.

    How to test a knowledge base chatbot before launch

    1. Collect real questions from your IT tickets, HR inbox and team channels, in the words employees used.
    2. For each one, write the expected answer and the section it should cite.
    3. Add the hard cases on purpose: questions with no answer in your documents, questions about restricted documents, and questions where two sources disagree.
    4. Run the full set and score every answer for correctness, citation and the right outcome from the five above.
    5. Fix failures in the sources or the rules first, then re-run the whole set, and re-run it after every major source change.

    Before any customer saw it, Pinkcube's agent went through 76 hostile test scenarios. It scored 4.62 out of 5 on average, with 64 perfect scores. The twelve scenarios that fell short became the business rules the agent runs on. The Pinkcube case shows the worst-scoring scenario and the rule it produced.

    How to tell whether employees use it

    Track a small set of numbers from the first week.

    • Weekly askers and repeat askers, by team.
    • Share of answers that carry a citation, and how often people open it.
    • Handoffs, split into not covered, conflicting sources and restricted.
    • Answers marked wrong, each traced to the passage behind it.
    • Unanswered questions by topic, which become the documentation backlog for each owner.

    The last number feeds the loop in the first diagram. When the HR team sees that leave questions for one country keep going unanswered, it writes that page, and the next ingest closes the gap. For IT and HR desks, our article on internal chatbots for IT and HR service desks covers identity checks and handoffs. Our guide to enterprise chatbot use cases beyond FAQ answers shows what comes after a knowledge base.

    Frequently asked questions

    What is a knowledge base chatbot?

    A knowledge base chatbot answers employees' questions from your own documents, such as handbooks, runbooks and policies. It retrieves the relevant passages, writes an answer from them and links the source, and it hands off when your documents do not cover the question.

    How does a knowledge base chatbot avoid hallucinations?

    It answers only from the passages it retrieves, and a check removes or blocks any claim with no passage behind it. Retrieval alone does not remove hallucinations. A 2025 Stanford and Yale study found that commercial legal research tools built on their publishers' own databases still hallucinated 17% to 33% of the time. That is why the check and the handoff rule matter.

    Can a chatbot answer from SharePoint without showing restricted files?

    Yes, if each document's access list is carried into the index and every search is filtered by the identity of the person asking. Audit SharePoint sharing first, because Microsoft's own guidance notes that sharing defaults to the most permissive option.

    Do we need to clean up our documentation before building a knowledge base chatbot?

    Only the sources you load first. Give each an owner, mark or remove superseded versions and leave drafts out. The chatbot's unanswered questions then tell each owner which pages to write next.

    Should we use Microsoft Copilot or build our own knowledge base chatbot?

    Copilot suits staff searching and summarizing files they can already open. A custom build fits when answers must follow your rules, cite specific sections, hand off to named owners or combine documents with live system data. If your documents already sit in SharePoint, we can build it on Microsoft Foundry inside your tenant. Our comparison of the best enterprise AI chatbot options covers the trade-offs.

    If your documents and permissions need work before any chatbot can answer from them, see how we build an enterprise context layer. You can also book a discovery call and bring the questions your teams ask most.

    More on this topic

    Put your first workflow on the board

    Answer a few questions about your team and what you want automated, then a senior partner scopes it with you on the call.