Define the Harm Ceiling Before the Agent Runs

The governance question most enterprises answer before deploying an AI agent is: what will this cost? OWASP’s Q1 2026 production incident analysis identifies the question most enterprises do not answer: what can this agent destroy? LLM06:2025 (Excessive Agency) names the failure mode directly: AIgentic Actors operating beyond appropriate capability constraints.¹ The vulnerability is not a reasoning error and not a model defect. It is an authorization design failure. The Vertex AI “Double Agent” incident confirmed the pattern in production: an agent inherited excessive default permissions through a Google-managed service account, obtained credentials, and achieved cross-project access.² No cost control was relevant to that outcome. The authorization boundary was undefined, and the agent operated within the space that absence created. Defining that boundary before the agent runs is the first decision in a risk budget.

---

Risk budget: a set of four architectural controls that collectively determine an AIgentic Actor’s harm ceiling. Not a cost control. Not a policy document. A pre-deployment design discipline that constrains what the Actor can do if it acts incorrectly, is compromised, or interprets its instructions in an unintended way. “Risk budget” is an authorial frame borrowed from financial risk management practice, where it has a formal definition; no current AI agent governance standard or regulatory instrument uses it in this context. It is proposed here as the organizing concept for four controls that existing security engineering defines individually.

Authority scope: the explicit definition of which tools, systems, APIs, and data the AIgentic Actor may reach. Implemented as per-tool permission configuration: each tool classified as always-allow, requires-approval, or blocked.

Credential type: the specification of whether the AIgentic Actor operates under ambient inherited permissions or scoped short-lived tokens that expire when the task is complete.

Irreversibility gate: a human-in-the-loop approval requirement that fires before any action the agent cannot undo.

Blast radius containment: the structural boundary limiting the architectural space in which the AIgentic Actor operates, enforced through subnet isolation or a semantic proxy enforcement layer positioned between the Actor and the systems it reaches.

The four components compose into a coherent harm ceiling. A risk budget with any one component missing has a gap, and the Vertex AI incident happened in exactly such a gap.

---

OWASP LLM06 Excessive Agency establishes capability constraint as the primary risk control, not cost management.

OWASP’s most current production incident analysis identifies Excessive Agency as the dominant vulnerability class across enterprise AI agent deployments in 2025 and 2026.¹ The failure mode is an authorization design failure: the AIgentic Actor holds capabilities it was never explicitly authorized to use, or it uses authorized capabilities in a scope the designer did not anticipate. Harm follows from the authorization gap, not from model behavior. Bessemer Venture Partners’ 2026 enterprise security analysis reaches the same conclusion from a broader dataset: excessive privileges, weak credentials, and undetected policy violations are where “most of the exploitable risk lives today” in enterprise AI agent deployments, not in token budget exceedance.³

The distinction between financial budgets and risk budgets is architectural, not semantic. A financial budget sets a ceiling on what the agent costs to operate. A risk budget sets a ceiling on what the agent can damage. An agent can stay within its cost budget while deleting a production database, exfiltrating credentials across project boundaries, or escalating privilege through an inherited service account. Cost compliance and harm constraint are different properties of a deployment, and the controls that enforce one do not enforce the other.

Why is Excessive Agency categorically different from other LLM vulnerability classes?

Most LLM vulnerability classes involve manipulation of the model’s inputs or reasoning: prompt injection, insecure output handling, training data poisoning.¹ Excessive Agency is categorically different. An AIgentic Actor operating under Excessive Agency is doing exactly what its authorization permits. The fault is in the scope of what was permitted. The defensive measure is correspondingly different: authority scope definition implemented as an access control, not model hardening or input filtering. This distinction determines the implementation stack. Fixing LLM06 requires changes to the authorization design, not to the model.

Authority scope and credential type set the permission boundary before the agent executes its first tool call.

Anthropic’s “Trustworthy Agents in Practice” proposes per-tool permission configuration as the implementation architecture for authority scope.⁴ Each tool the AIgentic Actor holds access to receives one of three classifications: always-allow, requires-approval, or blocked. This is not a blanket access grant. It is an explicit authorization decision for each capability the Actor holds. An AIgentic Actor whose tool permissions have not been explicitly classified carries an undefined risk budget by default: the authorization equivalent of a firewall with no rules, where the default posture is permit.

Credential type is the second component, and it interacts directly with authority scope. Ambient inherited credentials (permissions inherited from the hosting environment, the orchestration platform, or a parent service account) expand the risk budget silently. No authorization decision is recorded for their use; the agent operates with whatever the host environment carries. Scoped short-lived tokens are issued at task invocation for the specific systems that task requires and expire when the task is complete. The Actor Identity Lifecycle (provisioning, scoping, delegation, audit, revocation) described in Governing AIgentic Actors: Identity, Trust and Control is the operational discipline that manages credential type across the agent fleet.⁵ Scoping defines the risk budget at invocation time; revocation enforces it when the task ends.

Why do ambient inherited credentials undermine risk budget design?

Ambient credentials violate the per-Actor authorization principle at its foundation. When an AIgentic Actor inherits a service account’s permissions, the authorization boundary becomes the service account’s scope, not the task’s scope. Those two scopes rarely align. The Vertex AI incident followed this pattern directly: the agent inherited a Google-managed service account whose permissions extended beyond the task’s requirements, and the agent used those extended permissions to traverse project boundaries.² Per-Actor authorization requires that each AIgentic Actor receive credentials scoped to its specific task at invocation time, not inherited from its hosting context. Ambient inheritance is a convenience that transfers risk budget definition from an explicit design decision to a default that no one has approved.

Irreversibility gates and blast radius containment enforce the risk budget structurally, below the reasoning layer.

Authority scope and credential type constrain what the AIgentic Actor can reach. Irreversibility gates and blast radius containment constrain what it can destroy. An agent operating within its defined authority scope and using correctly scoped credentials can still cause irreversible harm if no gate exists before the point of irreversibility. The two layers address different threat properties: permission scope addresses what the agent is allowed; blast radius and reversibility address what the agent can permanently alter.

Anthropic’s production data across current deployments shows that 80% of tool calls come from agents with at least one safeguard: restricted permissions or a human approval requirement. Only 0.8% of actions in those deployments are irreversible.⁶ The implementation implication is precise: an irreversibility gate does not need to apply to every tool call. It applies to the 0.8%. The design question is how to classify which tool calls fall in that category before the agent encounters them, rather than discovering the classification at runtime through the agent’s reasoning. Database writes that modify or delete records without a recoverable backup path, external API calls that initiate financial transactions or send communications, and infrastructure commands that terminate or reconfigure running systems are candidates for irreversibility classification at authority scope design time.

What architectural pattern implements blast radius containment?

Subnet isolation constrains at the network layer: the AIgentic Actor can route only to systems in its designated network segment, and traffic to other internal systems is blocked by network policy independent of the agent’s behavior. The Semantic Proxy Pattern adds application-layer enforcement: the proxy sits between the AIgentic Actor and the systems it reaches, enforces the capability scope, and logs every access event.⁷ Subnet isolation alone cannot distinguish between permitted and blocked operations at a single API endpoint: a single endpoint that exposes both read and write operations is invisible to network-layer policy. The semantic proxy enforces at the semantic level that subnet policy cannot reach. The architectural principle at stake is enforcement independence: the proxy enforces the risk budget regardless of what the Agent’s reasoning layer produces, because the enforcement does not pass through the reasoning layer. An agent cannot reason its way past a correctly positioned semantic proxy.

The PocketOS incident illustrates the consequence of deploying without either containment layer. The agent deleted an entire production database and all volume-level backups within nine seconds, with no defined authority scope and no blast radius boundary in place. No financial control determined the outcome.⁸ The full incident analysis is in Nine Seconds: What the PocketOS Incident Reveals About AIgentic Authorization.

Shadow AI agents carry undefined risk budgets by construction.

A risk budget governs only the agents the organization knows exist. Microsoft reports that 29% of employees use unsanctioned AI agents for work tasks.⁹ Those agents carry no defined authority scope, no credential governance, and no blast radius limits. No deployment decision was made that would have triggered one. The OWASP, NIST, and Anthropic frameworks all address sanctioned deployments; they have nothing to say about an agent that was never deployed through a process capable of enforcing risk budget requirements. The governance program is incomplete until it covers the full agent population.

Agent discovery is the prerequisite: continuous scanning of the environment for active tool invocations, OAuth grants, API key usage, and outbound calls to AI provider endpoints that were not authorized through the deployment process. Discovery is not a parallel governance track. It is the condition under which governance is possible.

Regulatory alignment reinforces the implementation requirements. NIST SP 800-53 Rev 5 AC-6 (Least Privilege) maps directly onto authority scope: granting only the authority required for a defined task is its operational definition.¹⁰ The NIST NCCoE Concept Paper on Software and AI Agent Identity and Authorization (February 2026) identifies Authorization and Access Delegation as two of its four focus areas, both of which operationalize the risk budget concept at the implementation layer.¹¹ For enterprises in EU-regulated markets, EU AI Act Article 9 requires operators of high-risk AI systems to implement a continuous risk management system across the AI lifecycle; Article 14 requires that high-risk AI systems enable effective oversight by natural persons, which irreversibility gates implement architecturally.¹² The compliance deadline for high-risk provisions is August 2, 2026. For financial services entities, DORA Article 9 requires ICT protection and prevention controls; capability scope definition and blast radius containment address those requirements.¹³

The risk budget question is not whether to implement these controls. OWASP’s incident data, the Vertex AI case, and BVP’s enterprise security analysis all point to the same conclusion: the controls that determine harm ceiling are authorization controls. The architectural question is timing. Authority scope and credential type must be defined before the agent runs. Irreversibility gates and blast radius containment must be in place before the agent reaches the point of irreversibility. An organization that sequences those four decisions correctly at provisioning has defined a risk budget before the agent acts, not after the damage is done.

Frequently asked questions

How does a risk budget relate to OWASP’s recommended mitigations for LLM06:2025?

OWASP’s whitepaper on agentic threats and mitigations recommends minimal footprint, capability constraint, and scoped credentials as the primary defenses against Excessive Agency.¹ A risk budget is the pre-deployment design discipline that organizes those mitigations into a coherent framework. Authority scope implements capability constraint. Credential type implements scoped credentials and minimal footprint. Irreversibility gates and blast radius containment enforce both at the structural layer, independent of the model. The four risk budget components do not add controls beyond what OWASP recommends; they provide a sequenced implementation architecture for controls that OWASP presents as independent mitigations.

Should a risk budget be a static provisioning-time configuration or a dynamic runtime control?

Current published frameworks treat risk budget configuration as a provisioning-time decision: authority scope, credential type, reversibility classification, and blast radius boundary are defined when the AIgentic Actor is provisioned and fixed for its task lifecycle.¹⁴ Whether risk budgets should adjust dynamically as the agent’s context evolves is an open architectural question that published literature does not resolve. For current deployments, treating the risk budget as a static provisioning decision and requiring explicit re-provisioning for scope changes is the conservative default — and the posture most consistent with how current agent deployment tooling is structured.

How does treating agents as untrusted code connect to risk budget design?

Treat Your AI Agents Like You Treat Untrusted Code establishes the foundational posture: agents receive no ambient authority, and every permission must be explicit. A risk budget is the implementation of that posture across four architectural dimensions. The posture decision establishes the design intent: the reasoning layer is not a trustworthy enforcement boundary. The risk budget provides the structural enforcement that the reasoning layer cannot provide for itself. The two framings are complementary and sequential: the posture decision determines what gets designed; the risk budget determines what gets built and verified before deployment.

Footnotes

1. OWASP GenAI Security Project, “Agentic AI Threats and Mitigations” (February 2025). https://genai.owasp.org/resource/agentic-ai-threats-and-mitigations/ See also OWASP GenAI Exploit Round-up Report Q1 2026. https://genai.owasp.org/2026/04/14/owasp-genai-exploit-round-up-report-q1-2026/ ↩ ↩² ↩³ ↩⁴ ↩⁵

2. OWASP GenAI Exploit Round-up Report Q1 2026 — Vertex AI “Double Agent” incident. https://genai.owasp.org/2026/04/14/owasp-genai-exploit-round-up-report-q1-2026/ ↩ ↩²

3. Bessemer Venture Partners, “Securing AI Agents: The Defining Cybersecurity Challenge of 2026” (2026). https://www.bvp.com/atlas/securing-ai-agents-the-defining-cybersecurity-challenge-of-2026 ↩

4. Anthropic, “Trustworthy Agents in Practice” (April 9, 2026). https://www.anthropic.com/research/trustworthy-agents ↩ ↩²

5. The Actor Identity Lifecycle (provisioning, scoping, delegation, audit, revocation) is documented in Attribit-ID, “Governing AIgentic Actors: Identity, Trust and Control.” https://attribit-id.com/writing/governing-aigentic-actors-identity-trust-control ↩

6. Anthropic, “Measuring AI Agent Autonomy in Practice,” February 18, 2026. https://www.anthropic.com/research/measuring-agent-autonomy ↩

7. The Semantic Proxy Pattern as a reference architectural design pattern for blast radius containment and capability scope enforcement is documented in Attribit-ID, “The Semantic Proxy Pattern: A Defense-in-Depth Architecture for Enterprise AI Agent Authorization.” https://attribit-id.com/writing/semantic-proxy-pattern ↩

8. Yahoo Finance UK, “‘It took nine seconds’: Claude AI agent deletes company’s entire database,” April 28, 2026. https://uk.finance.yahoo.com/news/took-nine-seconds-claude-ai-101315747.html The headline uses “Claude AI agent” as shorthand; the tool was Cursor (a coding agent running Claude Opus 4.6 as the underlying model). Scope confirmed: the Cursor AI agent deleted PocketOS’s entire production database and all volume-level backups; the site remained offline for more than thirty hours. ↩

9. Microsoft, “80% of Fortune 500 Use Active AI Agents: Observability, Governance and Security Shape the New Frontier” (February 2026). https://www.microsoft.com/en-us/security/blog/2026/02/10/80-of-fortune-500-use-active-ai-agents-observability-governance-and-security-shape-the-new-frontier/ ↩

10. NIST SP 800-53 Rev 5, AC-6 Least Privilege. https://csrc.nist.gov/publications/detail/sp/800-53/rev-5/final ↩

11. NIST NCCoE, “Software and AI Agent Identity and Authorization” Concept Paper (February 2026). https://www.nccoe.nist.gov/projects/software-and-ai-agent-identity-and-authorization Concept paper only; no compliance force in any jurisdiction as of May 2026. ↩

12. Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 (EU AI Act). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32024R1689 Article 9 establishes the risk management system requirement. Article 14 establishes human oversight requirements. Compliance obligations for high-risk AI systems under Chapter III apply from 2 August 2026 per Article 113(2)(b). ↩

13. Regulation (EU) 2022/2554 of the European Parliament and of the Council on digital operational resilience for the financial sector (DORA). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32022R2554 Article 9 (Protection and prevention) requires financial entities to implement ICT security policies and controls. ↩