sota.io
2026-04-15·13 min read·

The Risk Identification Gap That Regulators Will Find First

Your EU AI Act conformity assessment documents the intended purpose of your system. Your Art.9 risk register lists the risks that arise when the system is used as designed. Your Annex IV technical documentation describes the intended users and intended deployment context.

But Art.9(3) requires something more:

"The risk management system referred to in paragraph 1 shall take into account [...] risks as a result of the reasonably foreseeable misuse of the high-risk AI system, including, where applicable, the risks arising from the interaction of the system with other AI systems."

The definition of "reasonably foreseeable misuse" sits at Art.3(24):

"reasonably foreseeable misuse means the use of an AI system in a way that is not in accordance with its intended purpose, but which may result from reasonably foreseeable human behaviour or interaction with other systems, including other AI systems"

This is not the same as "intended use." It is not "all possible misuse." It is a specific middle category: use that falls outside the intended purpose but that a reasonable actor — provider, deployer, regulator, or plaintiffs' counsel in an AI Liability Directive claim — would expect to occur in practice.

If your Art.9(3) risk identification exercise only covers your system's intended use scenarios, your risk management system is incomplete. Market surveillance authorities checking your technical documentation will look specifically for misuse scenario analysis. The AI Liability Directive's Art.4 rebuttable presumption of fault applies if your system causes harm through a foreseeable misuse pathway that your risk register does not address.

This guide covers the Art.3(24) definition in detail, the two-dimension misuse test, a practical taxonomy of misuse categories for each Annex III sector, the downstream obligations that apply to identified misuse risks, and Python tooling to build a structured misuse scenario assessment into your Art.9 process.

What Art.3(24) Actually Says

The definition has two independent dimensions:

Dimension 1 — Reasonably foreseeable human behaviour:

The system is used in a way that is not intended, but a reasonable observer would expect some humans to use it that way. This captures the full range of human decision-making in real deployment environments: cognitive shortcuts, pressure to misapply a tool to save time, scope creep driven by organisational incentives, and automation bias that leads operators to override the system's design assumptions.

The "reasonably foreseeable" qualifier is load-bearing. It is not "all possible misuse" — a provider of a CV screening tool is not required to model a scenario where a deployer uses the output to plan corporate espionage. The standard is the foreseeable range of human behaviour in the actual deployment context. If HR managers at large enterprises routinely use CV scoring outputs beyond the approved use case, that is foreseeable whether or not the provider approved it.

Dimension 2 — Interaction with other systems, including other AI systems:

Misuse also arises from system-to-system interaction. An AI system that is compliant in isolation may produce harmful outputs when its outputs feed into a downstream decision pipeline, when it is chained with a GPAI model that amplifies its conclusions, or when it operates in an environment with different assumptions from the ones in which it was validated.

This dimension is increasingly important as AI systems are integrated into orchestration pipelines. A high-risk CV screening API whose output is passed directly to a salary benchmarking model without human review may be operating within a system configuration that neither provider independently designed for.

Why the Distinction From Intended Use Matters

Art.3(12) defines "intended purpose" as the use for which an AI system is designed, including the specific context and conditions of use, as specified in the instructions for use, promotional material, or technical documentation. Intended purpose is what the provider says the system is for.

Art.3(24) "reasonably foreseeable misuse" is use that is not the intended purpose but that the provider, applying reasonable foresight, should anticipate will occur in practice.

The practical consequence: intended purpose defines your product scope. Reasonably foreseeable misuse defines your compliance scope. You cannot limit your Art.9(3) risk identification to the first category and ignore the second. The Regulation uses "including" language — Art.9(3) says your risk management system must "take into account" misuse risks. This is not discretionary.

Three categories of risk that a provider must assess under Art.9(3):

  1. Risks arising from intended use — the system performing as designed in the intended context
  2. Risks arising from reasonably foreseeable misuse — the system used outside its intended purpose in ways a reasonable actor could foresee
  3. Risks arising from interaction with other systems — including chaining, orchestration, and downstream decision pipelines

The Five-Category Misuse Taxonomy

For structured risk identification under Art.9(3), the following taxonomy maps Art.3(24)'s two dimensions into five operational categories:

Category 1: Scope Creep

The most common misuse pattern. A system approved for a specific use case is applied to adjacent cases that fall outside the declared intended purpose but are instrumentally convenient for the deployer.

Example — CV Screening (Annex III point 4): System approved for screening English-language software engineering applications at mid-size technology companies. Deployer applies it to screen candidates for executive roles, roles in regulated industries (financial services, healthcare), or international candidate pools with different language and cultural norms than the training data.

Why it is foreseeable: Deployers face pressure to scale their compliance investment. If the system generates a score, expanding its application requires no technical change — only organisational discipline. Market surveillance investigations of employment AI have repeatedly found scope creep as the primary real-world misuse pattern.

Art.9(3) implication: Your risk register must include a "scope creep" risk ID for each Annex III use case. The mitigation must address both technical controls (input validation, deployment context gates) and contractual controls (ToS restrictions, deployer audit rights).

Category 2: Automation Bias and Override Failure

The system is used as intended but operators systematically defer to its outputs in ways that defeat the Art.14 human oversight mechanism. The system is not technically misused — it produces outputs within its documented parameters — but the human oversight process breaks down in practice.

Example — Credit Scoring (Annex III point 5): A creditworthiness assessment tool is designed with an Art.14-compliant human review stage. In practice, loan officers approve all applications above a score threshold without reviewing the underlying explanation, and reject all applications below a second threshold without exercising override authority. The human oversight mechanism exists but is inoperative.

Why it is foreseeable: Automation bias is well-documented in human factors literature. Any system that produces a clear output score will experience this pattern. The Art.14(4) requirement for operators to "understand the capacities and limitations" of the system exists precisely because the Regulation recognises this risk.

Art.9(3) implication: The misuse risk here is that human oversight becomes a compliance formality rather than an operational control. Mitigation requires designing the oversight interface to make automation bias harder: requiring active engagement with explanations before approval, flagging cases near decision thresholds for mandatory escalation, and logging whether override authority was actually exercised.

Category 3: Input Manipulation

Adversarial actors — applicants, borrowers, students, or others subject to the system's decisions — deliberately manipulate inputs to affect outputs. This is not the intended use of the system but is foreseeable wherever the system's outputs have material consequences for individuals.

Example — Educational Assessment (Annex III point 3): An AI system that assesses student work for academic integrity produces a score that affects academic standing. Students learn to manipulate their submissions to stay below detection thresholds or to use AI-assisted writing in ways that evade detection while still qualifying as prohibited conduct under institutional policy.

Why it is foreseeable: Any system with known or discoverable decision criteria will face manipulation attempts from actors with incentives to do so. This is a well-documented adversarial ML pattern. Art.15 (accuracy, robustness, cybersecurity) overlaps with this category, but Art.9(3) addresses it as a misuse risk that must be identified in the risk register regardless of Art.15 robustness testing outcomes.

Art.9(3) implication: Risk ID must capture who has incentives to manipulate inputs, what manipulation pathways exist, and what the consequences of successful manipulation are for the system's high-risk decision. Mitigation may include adversarial robustness testing (Art.15 intersection), output monitoring for anomalous score distributions, and periodic revalidation of decision boundaries.

Category 4: Secondary Use of Outputs

The system produces an output for its intended purpose, but the output is subsequently used — by the deployer, by third parties, or by the subject of the decision — for purposes beyond the intended use.

Example — Biometric Authentication (Annex III point 1): A facial recognition system designed for physical access control produces a confidence score for each authentication event. The deployer retains these confidence scores and subsequently uses them to build an attendance tracking record, an emotional state inference model (intersection with the Digital Omnibus Art.5(1)(j) prohibition), or a behavioural profile that feeds into employment decisions.

Why it is foreseeable: Where outputs are valuable and retained, secondary use is a standard organisational behaviour. GDPR Art.5(1)(b) purpose limitation applies to personal data processing, but the Art.3(24) misuse framework requires the AI provider to address the risk at the AI system level, independent of the GDPR controller-processor relationship.

Art.9(3) implication: The risk register must include secondary output use scenarios for any system where outputs are machine-readable and retained. Mitigation options include output minimisation (returning only the access decision, not the raw confidence score), output format controls (non-extractable display), contractual restrictions on output retention and secondary use, and audit rights to verify deployer compliance.

Category 5: Cross-System Interaction

The system operates as designed but is integrated into a pipeline where its outputs, combined with other AI system outputs, produce outcomes that neither system was individually designed or assessed for.

Example — Migration and Asylum (Annex III point 7): A document authenticity verification AI system assessed for use in border control contexts. A deployer integrates its output with a separate risk scoring model for migration applications and a GPAI-based text analysis system. The combined pipeline produces an asylum recommendation score that no single provider independently designed or assessed. The document verification system's false positive rate, which is acceptable in isolation, amplifies into a significantly higher error rate in the combined pipeline due to error cascading.

Why it is foreseeable: Multi-model deployment is the standard architecture for AI systems in regulated sectors. Any provider of a high-risk system component should anticipate integration. Art.9(3)'s explicit reference to "interaction with other AI systems" addresses this category directly.

Art.9(3) implication: The risk register must include integration scenarios for each realistic deployment context. The provider cannot rely on the deployer's integration assessment — Art.9 is a provider obligation. Mitigation includes API design that makes error rates and confidence intervals machine-readable for downstream systems, documentation of validated integration configurations, and contractual restrictions on integration patterns not covered by the risk assessment.

The Art.9(3) Risk Identification Pipeline for Misuse

Art.9(3) provides the framework within which Art.3(24) misuse scenarios must be assessed. The pipeline:

Step 1: Intended Use Risk Identification Document the system's intended purpose (Art.3(12)), the intended deployment context, the intended user population, and the foreseeable range of intended use cases. This is the baseline that Art.3(24) misuse is defined against.

Step 2: Art.3(24) Misuse Scenario Identification For each of the five misuse categories above, generate specific misuse scenarios relevant to the system's Annex III category and deployment context. A scenario is a foreseeable deviation from the intended use that a reasonable actor in the deployment environment could execute.

Step 3: Harm Pathway Analysis For each misuse scenario, identify the harm pathway: who is affected, what harm category applies (physical, psychological, financial, dignity-related), how the AI system's output feeds the harm, and whether affected individuals are in vulnerable groups (Art.9's explicit consideration for vulnerable populations).

Step 4: Art.9(4) Evaluation — Severity and Likelihood Apply the Art.9(4) evaluation framework to each misuse risk: severity of harm (including reversibility), likelihood of occurrence, and breadth of affected population. Misuse scenarios often have different severity/likelihood profiles from intended use risks — scope creep may be high likelihood but moderate severity; input manipulation may be low likelihood but concentrated harm.

Step 5: Mitigation Mapping — Art.9(5) Hierarchy Apply Art.9(5)'s three-tier mitigation hierarchy to misuse risks:

Step 6: Residual Misuse Risk Disclosure Art.9(6) requires providers to disclose residual risks to deployers. This obligation applies to misuse risks as well as intended use risks. Deployers who are not informed of foreseeable misuse pathways cannot implement the operational controls that Art.14 requires them to maintain.

Downstream Obligations: Art.13 and Art.14

Identified misuse risks under Art.9(3) feed two downstream compliance obligations:

Art.13 — Transparency and Provision of Information to Deployers:

Art.13 requires providers to ensure their systems are accompanied by instructions for use that enable deployers to comply with their obligations. The instructions must include information on the intended purpose and other foreseeable uses. Art.13(3)(b) specifically requires instructions to cover "the known or foreseeable circumstances and conditions of use, including information on the contexts in which the system has not been validated."

This means your Art.3(24) misuse analysis directly populates Art.13 compliance. For each identified misuse scenario:

Deployers who receive this documentation cannot subsequently claim they were unaware of the misuse risk — and their obligation to maintain human oversight under Art.14 extends to foreseeable misuse pathways, not only intended use scenarios.

Art.14 — Human Oversight:

Art.14(4) requires that deployers ensure that persons assigned to human oversight have the competence, authority, and means to:

Misuse scenarios inform the design of the human oversight mechanism. If scope creep is a foreseeable misuse pattern (Category 1), the human oversight mechanism must include a process for verifying that the specific input case falls within the intended deployment context. If automation bias is foreseeable (Category 2), the oversight interface must make passive approval of score-threshold-based decisions identifiable and auditable.

Art.14(5) requires providers of high-risk AI systems to ensure the system can be "overridden or interrupted by designated operators." This is easier to design when the misuse scenarios that would trigger override are documented in the Art.9 risk register and communicated to deployers via Art.13 instructions.

The AI Liability Directive Exposure

The AI Liability Directive (ALD), applicable alongside the EU AI Act for AI-related civil claims, creates specific exposure for providers whose risk registers do not cover Art.3(24) misuse pathways.

ALD Art.4 — Rebuttable Presumption of Fault:

Where a claimant establishes that the AI system caused harm and the provider failed to comply with an applicable obligation under the EU AI Act, there is a rebuttable presumption of fault. Art.9(3)'s obligation to identify misuse risks is such an obligation. If a harm occurs through a misuse pathway that your risk register does not address, the plaintiff establishes the Art.9(3) non-compliance and invokes the presumption. The burden shifts to you to rebut it.

ALD Art.3 — Disclosure of Evidence:

Courts may order providers to disclose the risk register in ALD proceedings. A risk register that contains only intended use scenarios — and is silent on foreseeable misuse — is facially incomplete against the Art.9(3) standard. The completeness of the misuse section of the risk register is therefore directly relevant to litigation exposure.

Python MisuseScenarioAssessor

from dataclasses import dataclass, field
from enum import Enum
from typing import Optional

class MisuseCategory(Enum):
    SCOPE_CREEP = "scope_creep"
    AUTOMATION_BIAS = "automation_bias"
    INPUT_MANIPULATION = "input_manipulation"
    SECONDARY_OUTPUT_USE = "secondary_output_use"
    CROSS_SYSTEM_INTERACTION = "cross_system_interaction"

class HarmCategory(Enum):
    PHYSICAL = "physical"
    PSYCHOLOGICAL = "psychological"
    FINANCIAL = "financial"
    DIGNITY = "dignity"
    FUNDAMENTAL_RIGHTS = "fundamental_rights"

class MitigationType(Enum):
    DESIGN_ELIMINATION = "art9_5a_design_elimination"
    CONTROL_MEASURE = "art9_5b_control_measure"
    INFORMATION_PROVISION = "art9_5c_information_provision"

@dataclass
class MisuseScenario:
    scenario_id: str
    category: MisuseCategory
    description: str
    harm_category: HarmCategory
    harm_description: str
    involves_vulnerable_groups: bool
    severity: int          # 1–5
    likelihood: int        # 1–5
    reversibility: int     # 1 (irreversible) to 5 (easily reversible)
    breadth: int           # 1 (narrow) to 5 (population-scale)
    mitigations: list["MisuseRiskMitigation"] = field(default_factory=list)
    disclosed_to_deployers: bool = False

    @property
    def risk_score(self) -> int:
        return self.severity * self.likelihood * (6 - self.reversibility) * self.breadth

    @property
    def risk_level(self) -> str:
        score = self.risk_score
        if score >= 45:
            return "UNACCEPTABLE"
        elif score >= 20:
            return "HIGH"
        elif score >= 10:
            return "MEDIUM"
        return "LOW"

    @property
    def is_deployment_blocking(self) -> bool:
        return self.risk_level in ("UNACCEPTABLE", "HIGH") and not self.disclosed_to_deployers

@dataclass
class MisuseRiskMitigation:
    mitigation_id: str
    mitigation_type: MitigationType
    description: str
    post_mitigation_severity: int
    post_mitigation_likelihood: int

class MisuseScenarioAssessor:
    """Art.3(24) + Art.9(3) Reasonably Foreseeable Misuse Assessor."""

    def __init__(self, system_name: str, annex_iii_category: str, intended_purpose: str):
        self.system_name = system_name
        self.annex_iii_category = annex_iii_category
        self.intended_purpose = intended_purpose
        self.scenarios: list[MisuseScenario] = []

    def add_scenario(self, scenario: MisuseScenario) -> None:
        self.scenarios.append(scenario)

    def add_mitigation(self, scenario_id: str, mitigation: MisuseRiskMitigation) -> None:
        for scenario in self.scenarios:
            if scenario.scenario_id == scenario_id:
                scenario.mitigations.append(mitigation)
                # Apply mitigation effect to post-mitigation score
                scenario.severity = mitigation.post_mitigation_severity
                scenario.likelihood = mitigation.post_mitigation_likelihood
                return
        raise ValueError(f"Scenario {scenario_id} not found")

    def disclose_to_deployers(self, scenario_id: str) -> None:
        for scenario in self.scenarios:
            if scenario.scenario_id == scenario_id:
                scenario.disclosed_to_deployers = True
                return
        raise ValueError(f"Scenario {scenario_id} not found")

    def assess_art9_3_completeness(self) -> dict:
        """Check all five misuse categories are covered."""
        covered = {cat: False for cat in MisuseCategory}
        for scenario in self.scenarios:
            covered[scenario.category] = True
        return covered

    def generate_risk_register_summary(self) -> dict:
        blocking = [s for s in self.scenarios if s.is_deployment_blocking]
        unacceptable = [s for s in self.scenarios if s.risk_level == "UNACCEPTABLE"]
        undisclosed_high = [
            s for s in self.scenarios
            if s.risk_level in ("UNACCEPTABLE", "HIGH") and not s.disclosed_to_deployers
        ]

        coverage = self.assess_art9_3_completeness()
        uncovered_categories = [
            cat.value for cat, covered in coverage.items() if not covered
        ]

        return {
            "system": self.system_name,
            "annex_iii_category": self.annex_iii_category,
            "total_misuse_scenarios": len(self.scenarios),
            "unacceptable_risk_count": len(unacceptable),
            "blocking_scenarios": [s.scenario_id for s in blocking],
            "undisclosed_high_risk_count": len(undisclosed_high),
            "art9_3_category_coverage": coverage,
            "uncovered_misuse_categories": uncovered_categories,
            "art13_disclosure_complete": len(undisclosed_high) == 0,
            "deployment_ready": len(blocking) == 0 and len(uncovered_categories) == 0,
        }


# Example: CV Screening Tool Art.9(3) Misuse Assessment
assessor = MisuseScenarioAssessor(
    system_name="TalentRank CV Screening",
    annex_iii_category="Annex III point 4 — Employment and workers management",
    intended_purpose="Automated screening of software engineering applications "
                     "for English-language roles at technology companies with 50–500 employees",
)

# MISUSE-001: Scope Creep — executive role screening
assessor.add_scenario(MisuseScenario(
    scenario_id="MISUSE-001",
    category=MisuseCategory.SCOPE_CREEP,
    description="Deployer applies CV scoring to executive-level or C-suite applications "
                "outside the validated role type and seniority level.",
    harm_category=HarmCategory.FINANCIAL,
    harm_description="Executives have materially different hiring criteria; model trained on "
                     "junior/mid-level data produces invalid scores, disadvantaging qualified candidates.",
    involves_vulnerable_groups=False,
    severity=3,
    likelihood=4,
    reversibility=3,
    breadth=2,
))

assessor.add_mitigation("MISUSE-001", MisuseRiskMitigation(
    mitigation_id="MIT-001",
    mitigation_type=MitigationType.CONTROL_MEASURE,
    description="Input validation gate: reject applications tagged with seniority level "
                "Director/VP/C-suite. Return structured error with out-of-scope message.",
    post_mitigation_severity=3,
    post_mitigation_likelihood=2,
))
assessor.disclose_to_deployers("MISUSE-001")

# MISUSE-002: Automation Bias — passive approval at score threshold
assessor.add_scenario(MisuseScenario(
    scenario_id="MISUSE-002",
    category=MisuseCategory.AUTOMATION_BIAS,
    description="HR managers approve all applications above score 75 and reject all below 40 "
                "without reviewing explanation features. Human oversight mechanism is inoperative.",
    harm_category=HarmCategory.FUNDAMENTAL_RIGHTS,
    harm_description="Candidates in score boundary zones never receive meaningful human review. "
                     "Proxy variables encoding protected characteristics are not detected.",
    involves_vulnerable_groups=True,
    severity=4,
    likelihood=4,
    reversibility=2,
    breadth=3,
))

assessor.add_mitigation("MISUSE-002", MisuseRiskMitigation(
    mitigation_id="MIT-002",
    mitigation_type=MitigationType.CONTROL_MEASURE,
    description="Oversight interface requires active engagement with top-3 explanation features "
                "before approval action is available for scores 40–75. Log override events.",
    post_mitigation_severity=3,
    post_mitigation_likelihood=2,
))
assessor.disclose_to_deployers("MISUSE-002")

# MISUSE-003: Input Manipulation — applicant gaming score
assessor.add_scenario(MisuseScenario(
    scenario_id="MISUSE-003",
    category=MisuseCategory.INPUT_MANIPULATION,
    description="Applicants identify scoring keywords from job description embedding patterns "
                "and keyword-stuff CVs to inflate relevance scores without matching actual skills.",
    harm_category=HarmCategory.FINANCIAL,
    harm_description="Manipulated applications advance past screening; genuine candidates "
                     "with relevant skills but non-standard terminology are disadvantaged.",
    involves_vulnerable_groups=False,
    severity=2,
    likelihood=3,
    reversibility=4,
    breadth=2,
))

assessor.add_mitigation("MISUSE-003", MisuseRiskMitigation(
    mitigation_id="MIT-003",
    mitigation_type=MitigationType.CONTROL_MEASURE,
    description="Semantic validation layer: detect keyword density anomalies exceeding "
                "2σ from training distribution. Flag for mandatory human review.",
    post_mitigation_severity=2,
    post_mitigation_likelihood=2,
))
assessor.disclose_to_deployers("MISUSE-003")

# Generate report
report = assessor.generate_risk_register_summary()
print(f"System: {report['system']}")
print(f"Total misuse scenarios: {report['total_misuse_scenarios']}")
print(f"Blocking scenarios: {report['blocking_scenarios']}")
print(f"Art.9(3) coverage gaps: {report['uncovered_misuse_categories']}")
print(f"Art.13 disclosure complete: {report['art13_disclosure_complete']}")
print(f"Deployment ready: {report['deployment_ready']}")

Output:

System: TalentRank CV Screening
Total misuse scenarios: 3
Blocking scenarios: []
Art.9(3) coverage gaps: ['secondary_output_use', 'cross_system_interaction']
Art.13 disclosure complete: True
Deployment ready: False  ← Two misuse categories uncovered

The output shows the system is not deployment-ready: two of the five required Art.3(24) misuse categories — secondary output use and cross-system interaction — have no identified scenarios. Art.9(3) coverage requires at least one scenario assessment for each reasonably applicable category.

When a Misuse Category Is Not Applicable

Not every system has relevant misuse scenarios in all five categories. If a category genuinely does not apply, the risk register must document the rationale — not simply omit the category. Market surveillance authorities reviewing an Art.9 risk register that is silent on cross-system interaction for an API-exposed high-risk system will treat the silence as an oversight, not a deliberate determination.

The documentation standard: for each category assessed as not applicable, record:

For example, a high-risk AI system deployed as a fully offline, air-gapped tool with no API and outputs viewable only within a controlled interface can document cross-system interaction as not applicable based on the physical isolation constraint. The reasoning must be in the Annex IV technical documentation.

The 20-Item Art.3(24) Implementation Checklist

Intended Use Baseline (1–3)

Scope Creep (4–6)

Automation Bias (7–9)

Input Manipulation (10–11)

Secondary Output Use (12–13)

Cross-System Interaction (14–15)

Risk Evaluation and Mitigation (16–17)

Downstream Obligations (18–20)

Infrastructure Note: Misuse Documentation Under Art.18

Art.18 requires providers of high-risk AI systems to retain technical documentation for ten years from the date the system is placed on the market. The Art.9 risk register — including the misuse scenario assessment — is part of that documentation.

For a system deployed in 2026, the misuse risk register must be retained and accessible until 2036. Misuse scenarios that are identified, assessed, and mitigated must be documented in a form that survives personnel changes, system updates, and corporate restructuring.

EU-jurisdiction storage is directly relevant to Art.18 retention. Conformity documentation stored on US-jurisdiction cloud infrastructure is subject to Cloud Act disclosure — accessible to US law enforcement without MLAT process and potentially without notifying the provider. For regulated AI systems where technical documentation contains sensitive information about system vulnerabilities or misuse pathways, the jurisdiction of the infrastructure hosting conformity documentation matters.

This guide covers the EU AI Act as published in the Official Journal (Regulation 2024/1689). The Reasonably Foreseeable Misuse definition at Art.3(24) applies to all high-risk AI systems under Annex III from August 2, 2026 for new systems and August 2, 2027 for systems already on the market. The AI Liability Directive timeline for civil claims follows national implementation deadlines.

See Also