Assigning decision rights establishes who may act. Knowledge and experience support informed judgement. But further still, an organisation must be capable of recognising when human intervention is needed and seeing the necessary actions through.
When it comes to human intervention in AI, this goes beyond responding to a technical failure. A system may remain available and perform its configured tasks while producing results that are unsuitable for the business activity it supports. The concern may first become apparent to someone who understands the work, rather than someone monitoring the technology.
The governance question is whether the organisation’s operational arrangements enable those people to recognise the concern, take appropriate action and secure the support needed to resolve it.
ISO/IEC 42001 provides the basis of an AI management system for these arrangements. Its controls and implementation guidance address oversight responsibilities and prepare people to review AI outputs and override AI decisions where appropriate.
Build on the Arrangements Already in Place
Information Technology departments have become used to running with practices such as incident, problem, and change management. These mainstays of transition and support provide an established means of coordinating investigations, controlling technical changes, and restoring services. Business continuity arrangements will harness such practices to address how essential functions can be sustained when normal operations are disrupted.
It goes without saying that AI oversight should connect with these practices in some way. Creating a separate response structure without examining existing responsibilities could duplicate work and introduce uncertainty about who is directing the response.
However, those arrangements need to cover the circumstances in which AI is used. Who examines questionable outputs? Who understands their consequences? Who can restrict their use, arrange an alternative and establish whether earlier work has been affected?
These responsibilities may span business operations, technology teams, suppliers and specialist functions. They need to be agreed upon before an intervention becomes urgent.
Recognise the Need to Intervene
Consider an AI-enabled application within which an AI utility recommends the customer cases that should receive priority attention. One or more staff members notice that cases with particular characteristics are repeatedly assigned a low priority, despite being critical in nature.
The service is available. No component has failed. The issue requires examination of the recommendations in their business context.
A user feedback loop is essential (see Figure 1). Staff inherently know what warrants concern, but guidance is required on what information they need to record about their concerns and where to refer them. Designated reviewers need sufficient understanding of the business activities in question and the system’s role and expected behaviour to distinguish a one-off discord from a pattern that requires intervention.
A big part of the feedback loop is facilitation of the review. To assess the concern, reviewers need access to the reported cases, the information used by the AI, and the recommendations it produced against agreed prioritisation criteria. Their findings should determine whether protective action or further investigation is needed. Any resulting actions should be assigned to the appropriate resolver group, and updates should be communicated to the staff who raised the concern.
Make Human Intervention an Executable Responsibility
Human oversight will almost invariably be represented as a principle, goal and rule of an organisation's AI policy. As such, it will need to be translated into a clear operational arrangement.
Where a person is expected to review AI recommendations or actions, the question of what they are authorised and equipped to do must be answered explicitly. Moreover, there must be detailed guidance covering the range of scenarios and options the reviewer may encounter, including available temporary workarounds. For example, are they authorised to reject a report? Put a stop to the relevant automated action(s) and temporarily restrict the function(s) under investigation? To whom can they escalate a case, and what happens to the work while the issue is examined?
Answers to these kinds of questions inform the structure of permissions, procedures, interrelationships (including suppliers), and remediation pathways. In this way, the responsible party can exercise the appropriate authority and has practical means to intervene within the available time.
The procedures will need to distinguish actions taken by staff (users) from those requiring specialist support or further authorisation. This will involve lines of communication and escalation paths. The larger contribution of the procedures should detail the temporary workarounds that staff will need to enact while an issue is in progress. These arrangements should be put before and ratified by the relevant governance board or committee, which oversees their adequacy and ensures that delegated authority allows timely intervention.
The NIST AI Risk Management Framework reinforces this requirement through its provisions for defined human oversight responsibilities and mechanisms to override, disengage or deactivate AI systems.
Address the Affected Work
A human intervention must address the work already affected and prevent further impact on subsequent cases.
In the ‘case-prioritisation’ example outlined above, a suspension of the AI utility would stop recommendations from being generated. However, there would be some cases in mid-flow that are already prioritised but not yet acted upon. So, the first task would be to manually confirm the import of those cases and correct their priority and required handling where applicable.
A coordinator should be identified to lead the response activities and assign ownership of the necessary tasks. Completion should be checked against the issue in question, including any follow-on consequences.
The staff who raised the concern should receive an explanation of the outcome. Any changes to case handling or the use of existing recommendations should be communicated to all affected staff.
Establish the Basis for Resumption
Before the AI utility is returned for use, the organisation needs evidence that the issue that prompted its suspension has been adequately addressed.
In the case-prioritisation example, any correction should be evaluated using cases representative of the affected work, including those that previously received inappropriate priorities. The results should demonstrate that recommendations meet the agreed prioritisation criteria. Any changes to operating procedures or limitations on use should also be explained to the staff concerned.
A summary of events should be provided to the roleholder authorised to approve the resumption. As with all decision points, the evidence and whether it supports the proposed return to use must be established, and if any continued restrictions or monitoring are required, these should be specified as a condition of resumption.
Test the Arrangements
A practical review should follow one plausible concern from its initial detection through intervention, investigation and resumption.
Can the organisation identify who recognises the issue, who acts, how the action takes effect and who checks the result? Can the affected activity continue safely through an alternative process? Are the required people, permissions and supplier support actually available?
An independent review of the organisation’s current AI management system (AIMS), or equivalent governance and oversight mechanism, can help establish whether these provisions are adequate and effective in practice. It can also identify gaps between documented responsibilities and the ability to intervene.
The answer may involve strengthening existing teams, assigning additional responsibilities or establishing dedicated capacity. The structure should follow the work and the exposure.
Effective human intervention depends on people who can understand what is happening, act within clear limits and obtain a timely response. AI Governance must ensure that those operational conditions are in place.