An AI model might satisfy its evaluation criteria and still form part of a solution that is not ready for operational use. This does not necessarily mean that the evaluation was inadequate. Rather, it highlights the different lenses through which the AI model and the AI solution must be viewed.
Model evaluation findings should not be treated as sufficient evidence of operational readiness. AI incidents and failures repeatedly show what can happen when confidence in model performance is extended to the wider solution. The risk of harm may be realised not only from the model's behaviour but also from its integration with organisational functions.
AI assurance must examine the whole solution, incorporating both the model’s behaviour and the operational arrangements through which it is utilised.
Deployment Goes Beyond the Evaluated Model
Model evaluation can show that selected aspects of the model’s behaviour meet the defined criteria under the stated conditions. That evidence is important, but the enterprise does not deploy the model in isolation. It deploys a configured AI-enabled solution to support specific business use cases within defined operating contexts.
At the appropriate time, the organisation must decide whether the solution may be deployed or, if already in production, remain in operational use. Any approval must specify the intended uses and operating conditions to which it applies.
The assurance scope must therefore extend beyond the evaluated model to the wider solution into which it is integrated. Depending on the architecture, this may include the infrastructure, platform and runtime, data and knowledge, applications, service arrangements and operational uses. Integration, interfaces and tools may connect these elements. Orchestration and automation may coordinate processes, tool use and human hand-offs across them.
Model Performance is not Enough
Consider a model that has met its evaluation criteria for prioritising customer cases for urgent review. In operation, the enterprise does not rely solely on the model output. It relies on a service that turns that output into an operational priority and, ultimately, a human response. That transfer depends on conditions not established by the model evaluation.
- The production data must contain the information on which the model depends and remain sufficiently consistent with the conditions under which it was evaluated.
- The application must preserve the meaning and uncertainty of the output rather than present a recommendation as a definitive judgement.
- The workflow must use the output within the role and limits for which the model was evaluated.
- Reviewers must have the information, capacity and authority needed to challenge the result and intervene when necessary.
These dependencies are not exclusive to AI. Conventional software can also become unreliable when inputs, interfaces, workflows or human controls are inadequate. Deterministic behaviour is generally reproducible against explicit rules, which can make incorrect states and outputs easier to detect through validation and conventional monitoring.
For AI models, however, an additional, less visible issue may arise. Production conditions may shift away from those originally represented in the evaluation. Once deployed, the model continues to make predictions based on its training, using production data as inputs. If the distribution of values across those production inputs changes materially from that represented during training and evaluation, data drift has occurred.
Conversely, concept drift can occur even when the inputs remain relatively aligned with the original training data. In this case, it is the relationship between the inputs and the target outcome that has changed. Patterns that previously indicated a particular result may become weaker or stronger indicators of that outcome, or cease to indicate it at all.
In either case, the model may continue to produce plausible outputs while its reliability declines. In the example we use here, the operational consequence is that an urgent case may be delayed even though the service remains available and no obvious software fault is reported. The implication is that model performance should not be considered a sufficient gauge of confidence in the wider solution’s operational readiness for at least two reasons.
- It does not establish that the wider service will use the output appropriately.
- It cannot account for changes in operational conditions after deployment.
Thus, the model should be viewed as just one component of the wider service solution.
Connecting the Evidence
Whole-solution assurance does not require every component to be examined to the same extent. Assurance effort should concentrate on the parts of the solution that materially shape the relationship between model behaviour and operational outcomes.
The necessary work may be divided across separate assessments, each with its own scope and objectives. But, regardless of these individual efforts, all relevant findings should be shared and considered collectively. Otherwise, inconsistent assumptions, gaps in coverage and unclear responsibilities can remain hidden between assessment boundaries.
A Whole-Solution Assurance Outcome
The decision to deploy or continue operating an AI solution relies on whole-solution assurance, which in turn should draw its conclusions from a clear, firm, and reviewable basis. Five questions test whether that basis has been established.
- Does the assurance scope cover the intended uses, operating contexts and material dependencies?
- Do the planned assurance activities, criteria and evaluation conditions address the material risks?
- Was the planned assurance work completed, with results, deviations and limitations recorded?
- Does the combined evidence support deployment or continued operation for the defined intended uses?
- Are reassessment triggers defined, monitored and assigned to accountable owners?
If the organisation cannot answer all five questions in the affirmative, it may have useful evidence about individual elements, but not a justifiable basis for confidence in how they operate together.
That is why Enterprise AI Assurance must address the whole AI-enabled solution, not the model alone