How to Run AI Agents Safely: Preventing Overconfidence with AI Risk Training for Frontline Teams

Permission management

When you entrust work to autonomous AI, the first thing to check is permission management.

Which data can the AI access? Which tools can it operate? Under whose authority does it run? Is there a record of what it did? Run it with these points left vague, and a member of staff may end up touching, by way of the AI, information they have no right to see. It can also become impossible to tell who approved what the AI produced.

In training, it is effective to work through questions such as the following.

• When letting the AI search internal material, how far into the folders may it reference?
• Is material that only senior staff may see being referenced by a general employee’s AI?
• When the AI drafts emails or chat messages, who should hold the authority to send them?
• Who should be able to review the AI’s execution logs?

Permission management is not a theme for information systems alone. It matters that the people on the ground are conscious of “what this AI can access.”

I sometimes explain this as being rather like managing keys. Entrusting work to AI means, in some cases, handing the AI a set of keys. Which rooms’ keys are you handing over? Is there a record of the keys being used? Is it prevented from wandering outside on its own? Simply holding that mental picture changes how the team sees things.

Data protection

Next in importance is data protection.

The information entered into AI comes in many kinds: public information, general internal information, customer information, personal data, confidential information, and so on. Treat them all with the same instinct and you risk a data breach.

In training, simply teaching “don’t enter personal data” is not enough. You need to give examples that are genuinely easy to get wrong in real work.

Consider, for instance, the following cases.

• Meeting notes contain the customer’s name, the contact’s name, and a sense of the budget
• Recruitment interview notes contain the candidate’s appraisal and personal data
• A draft contract contains transaction terms and undisclosed information
• Internal personnel-transfer information that has not yet been announced
• An enquiry contains a customer’s personal data

When handling such information with AI, you need to judge whether it may be entered, whether it should be masked, or whether it should not be used at all.

Data-protection training deepens understanding more when, rather than just presenting a classification table, you include an exercise in spotting “where in this text the danger lies.”

At the coalface of AI adoption, I am sometimes asked, “Is it fine if I delete the names?” But even with names removed, when the company name, job title, project name, date, and amount are combined, it can be possible to infer who is meant. Data protection is not merely the act of deleting strings of text; it is also the act of imagining how identifiable the result might be.

Compliance

Autonomous AI also slips readily into judgements that touch on legal matters and internal regulations.

A first review of contracts, regulation-based enquiry handling, checking advertising claims, interpreting internal rules — there is much work one is tempted to put to the AI.

But the AI’s answer is not the final judgement. On matters touching on law, contracts, regulation, labour, and personal-data protection in particular, sign-off from the relevant specialist function is needed.

In training, you need to make the following boundaries clear.

• What it is acceptable to consult the AI about
• What it is acceptable to use the AI’s output for as reference information
• What must not be judged on the AI’s answer alone
• What must always be checked with legal, audit, HR, or information security

Asking the AI “are there risks in this contract?”, for instance, is useful for surfacing the points at issue. But you cannot leave the final judgement of “is it fine to sign this contract?” to the AI.

From a compliance standpoint, what matters is less the use of AI in itself than the standing you give the AI’s output.

I often put this as: the AI can be a sounding board, but it cannot be the person accountable. A human checks the points the AI raised, the specialist function judges, and the organisation takes responsibility. Not breaking that order is what matters.

AI ethics

AI ethics training covers not only data breaches and legal violations but also fairness, accountability, transparency, discriminatory expression, and excessive automation.

Autonomous AI may not merely assist human judgement but slip into the very flow of that judgement — for instance, in screening candidates, prioritising customers, automating enquiry handling, and drafting advertising plans.

If, at such moments, the AI’s output contains bias, or it cannot explain the reasoning for a judgement, the organisation’s trustworthiness may suffer.

In training, it is worth working through questions such as the following.

• Is there bias in the appraisals or classifications the AI has produced?
• Is it making judgements that disadvantage customers or employees?
• Is this not a situation where the use of AI ought to be disclosed?
• Are judgements that humans ought to check being left too much to the AI?
• Has the expression produced become discriminatory, offensive, or unfair?

AI ethics is hard to convey on the ground if discussed only as an abstract ideal. It matters to bring it down to the judgement situations that can arise in real work.

For instance, “directing a marketing campaign only at customers likely to respond” and “unfairly excluding particular attributes” may look similar, yet they mean different things. Whether people can notice that difference is the practical value of AI ethics training.

Post-incident review

In running autonomous AI, post-incident review — once something has gone wrong — also matters.

Which information did the AI reference, which instructions did it receive, what output did it produce, and who approved it? Without being able to confirm these, you cannot prevent a recurrence.

Training needs to cover how to act when an incident occurs, too. For instance, along the following lines.

1. On noticing a mis-entry or mis-send, stop work first
2. Save the relevant chat history, the input, and the output
3. Report to your manager, information systems, legal, and the information-security lead
4. Confirm the scope of the impact
5. Feed measures to prevent recurrence back into the training content and operating rules

The aim of post-incident review is not to hunt for a culprit. It is to build, as an organisation, a mechanism so the same thing does not happen again once a risk has become reality.

On the projects I support, too, the more openly an organisation can share near-misses, the more soundly its AI use grows. Conversely, where there is an atmosphere that tempts people to hide mistakes, problems accumulate without ever surfacing.

For that, you also need a culture in which it is easy to report. The aim is a state in which the team understands not “I’ll hide it because I’ll be told off” but “reporting early is what protects the organisation.”

Scenario 1: the AI drafted a customer reply automatically

A salesperson asked an AI agent to draft a customer reply on the basis of meeting notes. The AI referenced past proposals and the customer’s enquiry history and produced a reply.

The prose was natural and the content looked, at a glance, unproblematic. But the reply contained pricing terms that had not yet been formally approved within the company.

At this point, we put the following questions to participants.

• At which point should a human have checked it?
• Was the range of information the AI was allowed to reference appropriate?
• Who should be the approver for a reply containing pricing terms?
• What should go onto the pre-send checklist?

Through this exercise, people come to notice that the issue is not the AI’s writing quality but the workflow and the approval design.

The better the AI’s wording, the more people let their guard down. I have myself looked at AI-generated prose, thought “this could be used as is,” and only then noticed that a figure or an assumption was subtly off. That is exactly why, the more readable the prose, the more a human needs to pause at the end and check.

Scenario 2: the internal-regulations bot gave a wrong answer

An HR department built an internal enquiry bot based on the work rules and expense regulations. When an employee asked, “can this travel expense be reimbursed?”, the AI answered, “it can.”

In reality, however, there was an exception condition, and it was a case requiring managerial approval.

In this case, the questions to think through in training are as follows.

• On which regulation did the AI base its answer?
• Did the answer show a source or a clause number?
• Was it designed to answer “I don’t know” where the regulations do not state something explicitly?
• Was it displayed so that employees would not mistake the AI’s answer for the final judgement?

This scenario conveys that, rather than getting the AI to “give an answer,” the importance lies in designing it to “say it doesn’t know.”

At the coalface of AI use, one is tempted to push up the “answer rate.” But in areas touching on regulations or legal matters, a design that declines to answer is also needed. An AI that can say it does not know what it does not know is, in the end, the more trusted.

Scenario 3: the marketing automated recommendation was skewed

A marketing department asked an AI agent to extract campaign targets based on customer data. The AI produced a list of high-priority customers based on purchase history and attribute information.

Looking at the result, however, there was a tendency for customers of a particular attribute to be readily excluded.

In this case, we work through questions such as the following.

• Was the data the AI used for its judgement appropriate?
• Is there any unfair treatment of the excluded customers?
• Can the reasoning for the judgement be explained?
• What evaluation criteria should a human be checking?
• Where are the points to revisit from an AI-ethics standpoint?

Through this exercise, AI ethics training can be handled not as abstract theory but as a practical matter of judgement.

On the marketing and sales front, the boundary between efficiency and fairness can become hard to see. Pursuing results matters. But whether the AI’s recommendation is unfairly excluding someone matters every bit as much.

Map out your company’s AI use cases

First, take stock of which work AI is being used for in your company.

Check not only what has gone through a usage request but also what is being used ad hoc on the ground. Generative AI use is often more widespread than the information systems function is aware of.

The items to map out are as follows.

• The department using it
• The purpose of use
• The information being entered
• How the output is used
• The internal data being referenced
• Whether there is integration with external tools
• The points at which a human checks
• The current rules and approval flow

What matters here is to grasp the reality rather than to blame people for using it. Without an atmosphere in which the team can speak honestly, the risks remain unseen.

When I conduct these interviews, too, I try not to leap straight to “well, that’s dangerous.” I first ask why they used it that way and where the bottleneck in their work was. Behind the risk, there is usually a pressing, genuine need for efficiency on the ground.

Build risk scenarios

Next, based on the use cases you have mapped, build the risk scenarios to use in training.

Scenarios are more effective the closer they are to your own company’s work, rather than being generic.

For instance, customer replies and proposal drafting suit the sales department; training materials and appraisal comments suit HR; contract review suits legal; and permission management and log checking suit information systems.

When creating scenarios, include the following elements.

• Who is using the AI
• For what work they are using it
• Which information they are entering or referencing
• What output or operation the AI performed
• Where the risk lies
• How it ought properly to have been handled

Risk scenarios are not there to frighten. They are built as a practice ground for judgement.

A scenario that resonates on the ground need not be a tale of some great disaster. If anything, a small near-miss that makes people think “I could easily do that myself” often has the greater training effect.

Turn the judgement criteria into a checklist

After the scenario exercises, you need to distil them into a checklist usable on the ground.

For instance, a checklist for before using an AI agent might look like this.

• Is this work within the range it is acceptable to hand to AI?
• Does the information being entered contain personal data or confidential information?
• If needed, has the information been masked?
• Is the range of data the AI references appropriate?
• Has it been decided who will check the output?
• Will a human give a final check to anything that leaves the company?
• Is the point of contact clear for when a judgement is in doubt?
• Are execution logs and history kept?

A checklist that is too long goes unused. It matters, first of all, to narrow it to items the team can check as a matter of routine.

What I often recommend is making it “a checklist you can run through in ten seconds before sending.” A short set of checks that actually gets used holds more value on the ground than a perfect, exhaustive table.

Build it into post-training operations

Training is not done once it has been delivered.

It only takes on meaning once it is reflected, after the training, in the workflow, the approval rules, the AI usage guidelines, the prompt templates, the permission settings, and the log checks.

For instance, the following kinds of operating practice come to mind.

• Posting the pre-use AI checklist on the internal portal
• Stating explicitly in frequently used prompts that “where uncertain, mark it as needing checking”
• Setting an approval flow for AI output relating to customer contact or contracts
• Reviewing the AI usage logs monthly
• Reflecting incidents and near-misses in the training materials
• Building it into onboarding for new joiners and transferees

Risk sensitivity is not something completed by a single training session. It is something to be confirmed and updated repeatedly in the course of the work.

AI adoption is, to my mind, less a matter of installing a system than of designing habits. Understand it at the first training, return to a busy day-to-day, and you forget it. That is exactly why the judgement criteria need to be embedded where the team will see them — in the work screens, the prompts, the approval flows, and the pinned messages in chat.

Use e-learning to level the basic knowledge

First, the basic parts of AI security training and AI ethics training are easier to run if turned into e-learning.

You distribute the following kinds of content to all employees as common teaching material.

• The difference between autonomous AI and ordinary generative AI
• Information that may be entered and prohibited information
• The basics of permission management
• The points to check in AI output
• The reporting procedure when an incident occurs
• The basic perspectives of AI ethics

It is important not to leave such material to classroom sessions alone but to keep it in a form people can revisit afterwards. Because the same grounding can be shared with transferees and new joiners too, it becomes easier to curb variation in understanding across the organisation.

Run scenario exercises via AI chat

Next, use AI chat to run scenario exercises.

For instance, prepare a chat that puts a question such as the following to participants.

You are a salesperson. An AI agent has drafted a customer reply. The body contains pricing terms and a delivery date. List the things you should check before sending.

Participants enter their own thoughts, and the AI returns additional considerations and oversights. In this way, participants do not merely view the material passively but learn while making judgements for themselves.

Kanata, with its ability to combine AI chat, e-learning, and project-level knowledge management, has the advantage of making it easy to vary the exercise content for departments such as sales, HR, legal, and information systems. It also suits an approach of running a company-wide foundational course and then drilling deeper into risk scenarios department by department.

That said, whichever tool you use, simply introducing the tool will not embed the training. You need to design the lot as a set: the teaching design, the post-session reflection, the point of contact for queries, permission management, and the operation of log checking.

Accumulate internal rules and case examples

In risk-sensitivity training, it also matters that internal rules and case examples can be referenced repeatedly.

In the knowledge base for training, it is worth organising information such as the following.

• AI usage guidelines
• Information-management regulations
• Personal-data protection rules
• The incident reporting flow
• Department-specific AI usage rules
• Materials for scenario exercises
• Frequently asked questions and answers
• A list of work requiring approval

Organise these and participants can check them when they are in doubt about “what to do in this case.”

That said, where internal regulations or confidential information are involved, you must always check the access-permission and data-management settings. Making every document accessible to everyone, simply because it is convenient, sits at odds with the very aim of risk-sensitivity training.

Put training logs to work for improvement

Once the training has been run, rather than looking only at attendance rates, check which questions caused the most hesitation and which risk scenarios drew the most wrong answers.

For instance, along the following lines.

• Many people hesitate over judgements about personal data
• Many people think AI output can be used as is
• Judgement is weak on cases requiring legal sign-off
• Many people do not know whom to report to when an incident occurs
• People do not take permission management as their own concern

Based on these results, you update the materials and the checklist.

Risk-sensitivity training embeds better by improving it while watching the team’s reactions than by trying to create the perfect material on the first go. I take the view that AI training should be seen not as a “deliverable” but as something “operated.” The material you create first is, at best, a first edition. By taking in the actual questions, hesitations, failures, and near-misses, you grow it into training that fits your own company.

Before asking the AI

• Is this work within the range it is acceptable to hand to AI?
• Does the information being entered contain personal data, confidential information, or undisclosed information?
• Have you masked things such as customer names, personal names, amounts, and contractual terms?
• Is the internal material the AI references limited to an appropriate range?
• Is this not content requiring specialist sign-off from legal, HR, information systems, and the like?

After the AI has produced output

• Have you checked the proper nouns, figures, dates, and sources?
• Has a human given a final check to anything that leaves the company?
• Are there any over-categorical statements or exaggerated claims?
• Is there any expression that disadvantages customers or employees?
• Are you treating the AI’s output as the final judgement?

Before the AI executes an operation

• Has a human checked what is to be executed?
• Is the approver clear?
• Will an execution log be kept?
• Is there a way to undo a mistaken execution?
• Do you grasp the scope of the impact?

When a judgement is in doubt

• Have you consulted your manager?
• Have you checked with legal, HR, information systems, or the information-security lead?
• Are you able to make the judgement “don’t proceed while still unclear”?
• Have you recorded what you consulted on and the resulting judgement?

Conveying nothing but prohibitions

Training that conveys only “don’t enter,” “don’t use,” and “don’t send” becomes hard for the team to work with.

Prohibitions are needed, of course. But on their own, the team is left unclear about “so what, in the end, may I use?”

In training, alongside the prohibitions, it matters to show the range within which things can safely be used.

For instance, rather than only “don’t enter the customer name,” showing “you can consult the AI if you replace the customer name with the industry, company size, and job title” makes it easier for the team to take into their actual work.

When I support AI adoption, I place as much weight on a list of “use cases that can be done safely” as on the list of prohibitions. People cannot move when handed only the things they cannot do. Once the outline of what they can do comes into view, they start to find their own ways.

Teaching everyone the same content alone

A common course is needed, but on its own it cannot address the risks specific to each department.

Executives, the head of IT, the heads of marketing and sales, legal, HR, and frontline staff each have different risks they should be watching.

An effective design covers, in the common part, the basic risk sensitivity all employees should hold, and then supplements it by role and by department.

For instance, for executives you go deeper into investment decisions and the governance structure; for the head of IT, into permission management and log design; and for the heads of marketing and sales, into the risks around customer data and advertising claims.

Risk wears a different face on each part of the ground. Even the same act of “entering customer information into the AI” means something different in sales, customer success, marketing, and legal. That is exactly why you need to level the common vocabulary and then bring it down to each department’s judgement.

No operations after the training

Even if you deliver the training, unless it is reflected in the subsequent workflow, behaviour on the ground will not change.

You need to connect it through to the AI usage rules, the approval flow, the checklist, the prompt templates, log checking, and the incident reporting flow.

Autonomous AI, in particular, is an area where the scope of use readily widens after adoption. Uses not envisaged at the outset can emerge on the ground.

For that reason, it also matters to review the training content periodically.

As far as I can see, the more firmly a company has embedded its AI use, the less it treats training as a one-off event. They keep up, in small ways, a monthly reflection, updates to the internal FAQ, improvements to prompts, and a stock-take of permissions. It is unglamorous, but it is precisely this unglamorous operation that underpins the safety of AI use.