How to Ensure AI Robustness and Accuracy for EU AI Act Compliance
Article 15 of the EU AI Act requires that high-risk AI systems achieve appropriate levels of accuracy, robustness, and cybersecurity throughout their lifecycle. These are not static properties. They must be established before deployment, maintained through monitoring, and tested periodically. This guide sets out the practical steps for meeting Article 15 requirements.
Article 15 requires that high-risk AI systems are designed and developed in a way that achieves an appropriate level of accuracy, robustness, and cybersecurity, and that they perform consistently with regard to those characteristics throughout their lifecycle. Article 15 specifically requires resilience against attempts by unauthorised third parties to alter system use or performance, and requires technical redundancy measures including fallback plans.
What Article 15 Requires in Practice
Accuracy, robustness, and cybersecurity are three distinct but related dimensions:
- Accuracy refers to the system performing correctly relative to its intended purpose: producing correct predictions, classifications, or recommendations at a level appropriate to the deployment context. The appropriate accuracy level is not fixed by the Regulation; it must be defined, documented, and justified for each system based on the consequences of errors.
- Robustness means the system continues to function correctly when faced with errors, faults, inconsistencies in inputs, or unexpected inputs outside its training distribution. It also means failing safely. If the system cannot produce a reliable output, it must do so gracefully rather than producing a silently incorrect output.
- Cybersecurity requires that the AI system and its supporting infrastructure are protected against attacks that target the model itself (adversarial inputs, model extraction, data poisoning) as well as conventional infrastructure attacks.
Step-by-Step: Implementing AI Robustness
Establish Accuracy Baselines Before Deployment
Before a high-risk AI system is deployed, define and document the accuracy level it is expected to achieve. This must be expressed in measurable terms: precision, recall, F1 score, AUC, or other domain-appropriate metrics. Specify these metrics separately for different subpopulations if performance disparities are anticipated (e.g., different demographic groups). Document the test dataset used to establish the baseline, including its composition and any known limitations. These baselines serve as the reference point for ongoing post-deployment monitoring.
Test for Edge Cases and Out-of-Distribution Inputs
Edge case testing examines how the system behaves at the boundaries of its intended operating envelope. Identify input distributions the model was not trained on: unusual combinations of input values, rare but valid scenarios, missing fields, corrupted inputs, and inputs in unexpected formats. For each edge case, document whether the system: produces a correct output, produces an incorrect output, produces a low-confidence output that it correctly flags as uncertain, or fails in a way that is detectable by the system or by downstream processes. Systematic edge case testing should be documented and repeated after any model update.
Adversarial Testing
Adversarial testing examines whether the system can be manipulated to produce incorrect or harmful outputs through crafted inputs. The relevant attack types depend on the system architecture. For large language models: prompt injection (embedding instructions in user input intended to override system behaviour), jailbreaking, and indirect prompt injection (malicious content in retrieved documents). For predictive models: input perturbation attacks that change model outputs with minimal changes to inputs. For all systems: model extraction attacks (repeatedly querying to reconstruct the model), and data poisoning (compromising training data). Document your adversarial testing methodology and results, and implement input validation controls to reject or flag suspicious inputs.
Implement Fallback Mechanisms
Article 15 specifically requires that high-risk AI systems have technical redundancy and fallback measures. When the system cannot produce a reliable output, due to low confidence, out-of-distribution inputs, system errors, or adversarial manipulation, it must not silently produce an unreliable output. Define and implement the fallback behaviour for each high-risk AI system: options include routing to a human reviewer, returning a conservative default decision, displaying a confidence warning to the operator, or suspending the AI decision pending manual review. The fallback mechanism should be tested regularly and should itself be robust. It must work even if the primary AI component is completely unavailable.
Monitor Accuracy Post-Deployment
Model performance in production is not static. As the real-world data distribution drifts away from the training distribution, accuracy degrades. This is a well-documented phenomenon, sometimes called "model drift" or "concept drift", that has caused real-world AI failures in credit scoring, fraud detection, and medical diagnosis systems. Implement continuous monitoring of production accuracy against the baselines established before deployment. Set alert thresholds that trigger a human review when accuracy falls below acceptable levels. The monitoring infrastructure requires the ability to capture AI inputs and outputs in production, which is most naturally implemented at the API layer where AI requests transit.
Document Testing Methodology for Regulators
Robustness testing for high-risk AI is not merely a development activity. It is a compliance demonstration. Market surveillance authorities may request your testing documentation as part of an audit or investigation. Maintain records of: the test methodology and test datasets used, the results obtained, any issues identified and mitigations applied, the individuals who conducted testing, and the date of each test cycle. When the model is updated, the testing cycle should be repeated and new documentation produced.
Types of Robustness Testing
Out-of-distribution testing
Testing with inputs outside the training distribution to assess generalisation and failure modes
Adversarial perturbation
Systematically modifying inputs to identify inputs that cause incorrect outputs with high confidence
Prompt injection (LLMs)
Testing whether instruction-following models can be overridden through crafted user or document inputs
Stress testing
Testing performance under high load, degraded infrastructure, or missing dependencies
Subgroup performance analysis
Measuring accuracy separately for different demographic and contextual subgroups
Fallback testing
Verifying that fallback mechanisms activate correctly when the primary AI component fails or produces low-confidence outputs
Cybersecurity Requirements for AI Systems
AI systems face both conventional and AI-specific security threats. Article 15 requires cybersecurity protections covering both.
Infrastructure security: Protect the serving infrastructure, model weights, and training data with standard security controls: access management, encryption at rest and in transit, network segmentation, and vulnerability management.
API security: AI capabilities accessed via APIs are exposed to all the standard API vulnerabilities: injection attacks, broken authentication, excessive data exposure, and rate limiting failures. Apply rigorous API security controls to all AI API endpoints, including input validation and schema enforcement.
Anomalous traffic detection: Monitoring AI API traffic for unusual patterns, such as unexpectedly large payloads, unusual request rates, systematic variations in inputs, or outputs that deviate significantly from normal distributions, can surface both adversarial probing attempts and system failures before they escalate. Automated alerting on anomalous AI traffic patterns supports both cybersecurity and robustness objectives simultaneously.
← Back to Assessment