The customer received the wrong answer. The employee says the system recommended it. The technology team says the model belongs to a supplier. The supplier points back to the organisation’s process. Everyone was involved; nobody seems to own the result.
Putting a person near an AI system is not the same as giving them the time, authority and confidence to challenge it.
That is more than a theoretical concern. Government AI adoption research found that 84% of businesses using AI reported at least some human input or checking. The figure sounds reassuring, but it does not tell us whether that review can change the outcome.
The ICO’s guidance on human oversight is much more specific: reviewers should remain engaged and critical, have authority to override an AI output and be confident they will not be penalised for doing so. If people routinely accept the system’s answer without genuinely assessing it, the decision may effectively be automated.
Start with the business decision
Define what the system contributes to and what it is not allowed to decide. Drafting a response for an employee to review is different from automatically refusing a service, changing a price or ranking a person.
The greater the possible effect on people, money, rights or essential operations, the stronger the oversight and evidence should be.
Name a business owner
The accountable owner should understand the process, have authority to change it and be able to explain why AI is being used. Ownership should not default to the technology team simply because the system is technical.
The owner is responsible for approving the intended use, agreeing performance measures, reviewing problems and deciding whether the system remains suitable.
Define human review clearly
“A human is involved” is not enough. State which outputs require review, what the reviewer must check, how much time they have and whether they can disagree without pressure.
If employees routinely accept recommendations without meaningful review, the human step may exist only on paper. Training, workload and authority all affect whether oversight is real.
Set limits on data and use
Users should know which information can be entered, what the output may be used for and which tasks are prohibited. These rules need to be practical enough for day-to-day work.
Controls should also cover supplier changes. A model or feature update can alter behaviour even when the organisation has made no visible change to its own process.
Keep a record of important decisions
For higher-impact uses, retain enough information to understand what happened: the system version, relevant input, output, human action and reason for the final decision. The record should be proportionate and respect privacy.
Without a usable record, the organisation may be unable to investigate a complaint, correct a repeated problem or show that its review process worked.
Plan for errors before they occur
Agree who can pause the system, how users report problems and when legal, security or senior leadership should be involved. Decide how affected people will be informed and how incorrect outcomes will be corrected.
A short scenario exercise can reveal gaps before a live issue creates pressure.
Hold suppliers to account without outsourcing responsibility
Contracts should set expectations for security, data use, performance information, incident notification, changes and exit support. Ask what evidence the supplier will provide when an output is challenged.
Strong supplier terms help, but they do not replace the organisation’s own decision about whether and how to use the system.
What the board should expect
Leadership should be able to see an inventory of live AI uses, a named owner for each important system, the human review required, current performance, incidents and unresolved risks.
Clear accountability makes useful AI easier to support because leaders know what they are approving and employees know where decisions sit. The work should be done before live use, not after the first serious error.