AI Agent Knowledge Base & RAG Setup: How to Build the Foundation for Task Execution

We want to hand things over to the AI, yet in the end someone has to step in every single time and explain how we actually do it here.

Those are the words of Mr Saeki, who had been trialling a task-executing AI agent within the DX promotion division of a manufacturing firm. Six months earlier, the company’s sales, IT, legal and frontline teams each kept their information in separate folders and Slack channels. The AI could manage general answers well enough, but it could not take account of internal rules, client-specific exceptions or the history of past decisions. Whenever they tried to delegate quotation drafting or enquiry handling, the work came straight back: “That judgement is different for Company A,” or “It hasn’t looked at the latest version of the policy.”

Today the picture is rather different. Over the past three months they have organised roughly 1,200 business documents, tagging each with metadata such as document type, update date, managing department and access rights, and they are close to a state where the AI agent can retrieve the internal knowledge it needs from a RAG environment. With a platform like Kanata, which lets you handle AI chat and learning-data libraries on a per-project basis, you can keep references separate by department or use case and try things out in stages.

This article is for organisations that want to build a task-executing AI agent RAG setup. As a shared, foundational piece, it walks through how to build a knowledge base for AI, how to design vector search, how to manage freshness, and which evaluation metrics to watch. The aim is a state in which the AI assists with work while checking your organisation’s own assumptions, rather than trotting out generalities. That said, RAG design alone will not automate every decision. Read on with your own operation in mind, taking in document preparation, permission design and human review along the way.

Why a task-executing AI agent needs a RAG environment

A task-executing AI agent plays a rather different role from an AI chat tool that simply answers questions.

Responding to enquiries, organising quotation terms, checking the content of internal applications, suggesting the next action in a customer interaction. When you ask an AI to assist with work of this kind, it needs not only general knowledge but also the company’s rules, its past decisions, its workflows and its exception conditions.

Suppose, for instance, you were faced with the following questions.

• Which clauses of the standard contract should apply to this client?
• Can this enquiry be handled with a first-line response, or does it need a legal check?
• Does this application pass muster under internal policy, so that it can be approved?

These are questions that general-purpose generative AI alone struggles to answer accurately. The answers lie not in the general commentary found on the internet but in your own contract templates, internal policies, history of past decisions and department-by-department operating rules.

That is where a task-executing AI agent RAG setup comes in.

RAG stands for Retrieval-Augmented Generation, known in Japanese as “search-augmented generation” Before the AI generates an answer, it retrieves relevant information from an external knowledge base and refers to that content in its response. For a general explanation of RAG, seeGoogle Cloud’s overview of RAG, among others.

By combining internal knowledge search, vector search and metadata search, you make it easier for the AI to produce answers that reflect not merely “what is generally the case” but something closer to “how this company tends to decide.”

A RAG environment is not a way of making the AI memorise all your internal common sense

Building a RAG environment does not mean the AI will automatically understand everything about your organisation.

RAG is not a mechanism for having the AI swallow knowledge wholesale. Rather, it is a way of getting it to retrieve the internal knowledge it needs, at the moment it needs it, and use that as the basis for its answer. The quality of a RAG environment therefore turns largely on four things.

1. The quality of the registered documents. If the system is stuffed with outdated files, duplicates, PDFs with no headings and notes of unknown authorship, the AI too will struggle to find the grounds it needs.
2. Search design. Plain keyword search alone copes poorly with questions phrased a little differently. Lean solely on vector search, on the other hand, and you weaken the exact matching of document names, policy numbers and contract clauses.
3. Access rights. If the AI ends up referring to material it was never meant to see, you have an information-governance problem before you have any efficiency gain at all. The risks of inadequate authorisation and data leakage in LLM applications are set out as a key concern in theOWASP Top 10 for LLM Applications 2025.
4. Operation. A knowledge base for AI is never finished once built. You have to add new material, swap out the old, and review search results that went unused and answers that went wrong.

In short, RAG design is not simply the task of feeding knowledge to an AI. It is the business design of getting your internal knowledge into a state the AI can handle safely.

Metadata underpins the AI’s retrieval accuracy

In a knowledge base for AI, what matters is not only the body text but the metadata.

Metadata is the supplementary information attached to a document itself, things like department, business category, target client, document type, creation date, update date and permission class.

When a person hunts for material, they think “the proposal in the sales folder, for Company A, updated last year.” The AI needs the same sort of retrieval cues.

In a RAG environment, metadata makes the following kinds of control much easier.

• Refer only to policies updated on or after April 2025
• Search only the manufacturing-sector proposals among the sales team’s material
• Refer only to documents in projects this user can access
• Base answers on official rules only, not on reference material

For a task-executing AI agent in particular, what matters is not just the relevance of the search results but whether the material may be referred to at all. However relevant a document, the AI must not use material the user has no rights to.

For that reason, metadata design should consider at least the following items.

Knowing when to use vector search and when to use keyword search

In internal knowledge search, vector search comes up a great deal.

Vector search converts text into a vector, a sequence of numbers, and finds related documents on the basis of semantic proximity. For the thinking behind vector search and databases built for RAG, seeIBM’s explanation of vector databases for RAG. So when someone asks “I’d like to know the deadline for settling travel expenses,” even if the phrase “travel expenses” does not appear in the document, the system can more readily find related expressions such as “travel-allowance claim,” “reimbursed expenses” and “settlement due date.”

This is a genuinely useful technique in a RAG environment, since the wording of a user’s question and that of an internal document often fail to match.

That said, there are situations where vector search alone will not do.

Policy numbers, contract clauses, product codes, client names and matter numbers turn on an exact match rather than semantic proximity. You must not conflate “Article 12” and “Article 21” merely because they are semantically close.

In practice, then, you need to use the methods in concert as follows.

For a task-executing AI agent RAG setup, it is realistic to assume hybrid search from the outset.

If an employee asks “what is the maximum I can claim for a hotel on an overseas trip,” you might picture this flow: vector search to find the relevant travel policy, metadata to narrow it to the latest, company-wide version, and keyword search to confirm “accommodation cost,” “overseas trip” and “upper limit.”

Chunk design changes the quality of the answer

In a RAG environment, rather than searching whole documents wholesale, you split them into set units. This unit of division is called a chunk.

Make the chunks too large and extraneous information creeps into the results. The AI has a harder time finding the relevant passage, and the answer turns vague.

Make them too small, by contrast, and the surrounding context is lost. A policy’s conditional clauses and exceptions get severed from one another, which can lead to misreadings.

Suppose, for instance, you had the following document.

The accommodation limit for domestic travel is set at 12,000 yen per night. However, where there are unavoidable business reasons, such as accompanying an executive, disaster response, or staying at a client-specified facility, an exception may be granted with the approval of a line manager.

Cut this out as just “the accommodation limit for domestic travel is set at 12,000 yen per night” and the exception conditions vanish. Conversely, make the entire travel policy a single chunk and you may surround the needed information with far too much else.

For chunk design, the following ideas help.

• Put one point in one chunk
• Do not sever conditions from their exceptions
• Include heading information in the chunk
• Handle tables and bullet lists in units that keep their meaning intact
• For policies and contracts, work to the clause as your basic unit
• For FAQs, keep question and answer together as a set
• Split minutes by theme

Chunk design is an area that tends to come unstuck if decided by the technical team alone. The business side needs to be involved, confirming “how far does one unit of judgement extend.”

Access rights are at the heart of a RAG environment

When you have a task-executing AI agent refer to internal knowledge, access rights are unavoidable.

The more useful the AI becomes, the wider the range of information it can reach. But it does not follow that everyone may be shown company-wide information.

Client-specific proposals from the sales team, appraisal material from HR, contract-negotiation notes from legal, minutes of management-committee meetings, each of these has a limited set of people who may see it. When an AI agent searches across these, the scope of its search must be controlled according to the individual user’s own permissions.

A common failure is for the knowledge base to hold permissions while the RAG search itself takes no account of them.

If material a person cannot see on the file server ends up mixed into the AI’s answer, that amounts to a genuine information leak.

For that reason, RAG design should rest on the following principles.

• If a user cannot see a document, neither can the AI
• Separate knowledge bases by project or department
• Hold permissions at the document level, and where possible at the chunk level
• Manage confidentiality classes as metadata
• Show the source in the answer so it can be verified
• Reflect permission changes for leavers and people who move roles

Kanata’s best practice frames projects around the question “is it acceptable for the people involved to see the same information?” That same lens, deciding whether the sales team’s and the engineering team’s information may sit together or ought to be kept apart, applies neatly to a RAG environment.

A RAG setup with no freshness management reproduces stale decisions

In a RAG environment, there is a risk of referring to outdated information.

For a task-executing AI agent this is a serious matter. The more natural the AI’s answer sounds, the more readily users take it to be correct.

If, for instance, an old policy lingers after the work rules have been revised, the AI may relay outdated application conditions. If old-price proposals remain after the price list is updated, a sales AI agent may draft a proposal at the wrong figure.

For that reason, the knowledge base needs a mechanism for managing freshness.

In concrete terms, you decide on operations such as the following.

What deserves particular care is that the longer-lived a piece of material, the more prone it is to going stale.

Internal FAQs, sales material, case studies and policy collections may be correct when written, yet six months on the conditions have changed. In a RAG environment, “removing” old material is itself part of improving quality.

Build an AI that can say “I don’t know”

For a task-executing AI agent, what matters is not solely producing correct answers. Being able to say “I don’t know” when it does not know is every bit as important.

If the knowledge base offers no grounds and the AI papers over the gap with generalities, you have an accident waiting to happen in the course of business.

Suppose, for example, an employee asks “can this be claimed as an expense,” and the AI replies “generally speaking, it would likely be allowed” even though the policy says nothing explicit about the case. That is dangerous.

The desirable answer in this case is something like the following.

I could find no explicit mention of this case in the registered expense policy. Please check with the finance team. The most likely related policy is the travel-expense settlement in Chapter 3.

Getting the AI to answer this way requires both prompt design and evaluation design.

Kanata’s best practice likewise holds that, for policy and FAQ help-desk uses, prompts that make the AI “say when it doesn’t know” are important, and that forcing it to answer regardless leads to mistaken interpretations of policy.

In a RAG environment, you build rules like the following into the prompt.

• Where no supporting material is found, do not answer by guesswork
• In the answer, cite the source document name, chapter, page and clause
• Where old and new material conflict, present both and prompt the person to check
• Where information needed for a judgement is lacking, ask a follow-up question
• Where a final business judgement is needed, direct the user to check with the relevant department

An AI that “answers within the bounds of its grounds and stops outside them” is better suited to executing work than one that simply “can answer.”

Without evaluation metrics, you cannot improve

A RAG environment is not a build-it-once-and-be-done affair. You need to improve it while watching the search results, the answer content, the user’s edits and the business outcomes.

For that, you need evaluation metrics decided in advance.

Representative evaluation metrics include the following.

What matters here is not to evaluate the AI’s answer in isolation.

The purpose of a task-executing AI agent is not to turn out elegant prose but to carry work forward safely and reproducibly. Evaluation metrics therefore need to include business-side measures as well.

For enquiry handling, you look at first-response time, first-contact resolution rate, return rate and escalation rate to the relevant department.

For sales support, you look at proposal-drafting time, deal-preparation time, gaps in CRM entry and the knowledge-reuse rate.

For internal applications, you look at the application-defect rate, time to approval and the number of clarification requests.

Rather than chasing the AI’s accuracy alone, what matters is to see where in the work things have improved.

Steps for starting a RAG environment small

Start a RAG environment company-wide all at once and it becomes unwieldy. It is realistic to begin narrowly, one task, one department, one body of knowledge.

Here I will set out an approach assuming roughly a 30-day trial. The actual period will vary with the complexity of the target task, the volume of documents and the difficulty of permission design.

Common pitfalls

In building a RAG environment, there are several recurring failure patterns.

An example of knowledge-base operation, thought through with Kanata

You can build a RAG environment as a standalone search system, or you can run it in combination with an existing business-support platform or AI foundation.

For instance, if you useKanata, one design might be to create a project for the sales team and, within it, place a sales-support AI chat, an AI summariser for condensing proposals, and e-learning for sales training. Register proposal templates, competitor comparisons, past deal notes and sample sales talk in the project library, and you make it easier to keep references organised by use case.

Kanata lets you add apps such as AI chat, AI summarisation and e-learning within a project, and describes a structure in which AI settings, prompts and learning data are managed from the project library.

With this structure, it becomes easier to keep the knowledge referred to separate by task.

For an HR and general-affairs project, build an enquiry bot that refers to work rules, the expense policy and FAQs. For a sales project, build a deal-preparation AI that refers to proposals and customer case studies. For a marketing project, build a content-creation AI that refers to past articles, the brand tone and the list of prohibited expressions.

Splitting knowledge and apps by project in this way makes it easier to organise access rights, purpose of use and evaluation metrics. That said, this is by no means peculiar to Kanata; the same design can be realised with other AI foundations, internal search platforms or document-management systems. What matters is not the name of the tool but aligning knowledge, permissions and evaluation metrics around the unit of work.

Design a RAG environment from three perspectives

As a shared, foundational point to hold onto, a RAG environment has three perspectives.

For the same RAG environment, the points that matter shift depending on whose vantage point you take. That is precisely why it is important first to align on a common design philosophy, and then to design for your own organisational structure and the particulars of the work.

In summary

Getting a task-executing AI agent to assist with your own work takes more than the answers of a general-purpose AI.

What you need is a RAG environment that can search internal knowledge, respect permissions, refer to the latest version, and answer while showing its grounds.

That said, a RAG environment is not completed by technology alone. Only when document preparation, metadata design, access rights, freshness management, evaluation metrics and human review all come together do you have a foundation usable in the business.

There is no need to aim for a company-wide rollout from the outset. It is realistic to begin with one task, one department and one body of knowledge, and to nurture the internal knowledge the AI agent can refer to while verifying the search results and answers.

Turning tacit knowledge and scattered material into a knowledge base the AI can handle. It is the accumulation of that work that moves a task-executing AI agent from a teller of generalities towards a practical partner that can act on your organisation’s own assumptions.

Q&A