Most enterprises that deploy AI agents approve a budget for the deployment. That budget answers one question: how much will this cost? The figure appears in every board paper, every procurement approval, and every vendor contract. The question those papers do not answer is the governance gap that OWASP, Anthropic, and the enterprise security community consistently identify as the primary risk in production AI deployments today: how much damage can this agent cause if it acts outside its intended scope? An agent that operates within its cost budget while deleting a production database is not a hypothetical. It is the documented pattern — the PocketOS incident is its clearest illustration.1
Financial budget: the operational cost control. Sets dollar limits on API usage, token consumption, and inference spend. Answers: how much does this agent cost to run?
Risk budget: the security control. Defines authority scope, credential type, reversibility gates, and blast radius boundaries. Answers: how much damage can this agent cause if it operates incorrectly, is compromised, or interprets its instructions in an unintended way?
“Risk budget” is a term borrowed from financial risk management practice, where it carries a formal definition. No current AI agent governance standard or regulatory instrument uses it in this context; it is the editorial frame proposed here for four architectural controls that existing security engineering defines individually.
The two questions govern different things. Both require a board decision. Most enterprises have answered only one.
The standard AI procurement question leaves the consequential question unanswered.
Two budget conversations happen in most AI deployments. One is visible: the cost model, the API pricing, the license fee, the infrastructure spend. That conversation reaches boards. The other is invisible: what this agent is permitted to do if it acts outside its intended scope. That conversation happens, if at all, in engineering teams. Its conclusions rarely reach the people who approved the deployment. OWASP identifies this as the central vulnerability class across production AI deployments. LLM06:2025 (Excessive Agency) names the failure mode directly: agents operating beyond appropriate capability constraints.2 The vulnerability is not a cost problem. It is an authority problem.
What is the board actually approving when it approves an AI agent?
The approval of an AI agent deployment is, functionally, the approval of a set of capabilities and a set of permissions. A financial budget approval answers: can we afford this? A risk budget approval answers: what is the maximum harm we are prepared to accept if this agent operates incorrectly? The first question is operational. The second is a governance decision: one that, in most current deployments, no one has explicitly made. Approving a deployment without answering the second question is not a gap in the engineering team’s work. It is a gap in governance.
A risk budget is the four architectural decisions that determine an agent’s harm ceiling.
A risk budget translates into four concrete decisions, each of which an architecture team can implement and a board can require. Authority scope defines what systems and data the AIgentic Actor can reach. Credential type specifies whether the Actor uses inherited ambient permissions or scoped short-lived tokens that expire when the task is complete. Reversibility gates require human approval before any action the agent cannot undo. Blast radius containment limits the architectural space in which the agent can operate, typically through subnet isolation or a semantic proxy enforcement layer. An AIgentic Actor whose permissions are unbounded has an undefined risk budget. That is the default posture in current AIgentic deployments when governance has not explicitly specified otherwise.
In The Agent Problem: Why Your AI Workforce Needs a Different Kind of Oversight, the four governance properties boards should verify map directly to these four risk budget parameters. The vocabulary is different; the governance requirement is identical.
Where does the term come from?
The term is not invented for AI governance. In financial risk management, a risk budget is the established framework that determines which decisions a trader can make autonomously and which require escalation for upper management approval, a practice that financial firms have used for decades to constrain the maximum harm a trading desk can cause if its judgment is wrong.3 The question is not whether losses will occur but how much the business can absorb before the product ceases to be viable. Applied to AIgentic governance, the logic is identical: not whether an agent will ever act unexpectedly, but how much damage that action is authorized to cause before it crosses a governance threshold. The financial services industry has made this distinction for decades. AI governance has not yet made it.
Boards that have not defined a risk budget have approved deployment without authorizing governance.
The PocketOS incident demonstrates the distinction precisely. An agent completed nine seconds of autonomous action, deleting PocketOS’s entire production database and all volume-level backups.1 No financial control determined the outcome. The agent had no defined authority scope, no irreversibility gate, and no blast radius boundary. It had a financial budget and no risk budget. That combination is not an engineering failure. It is a governance failure: the organization approved what the agent would cost and did not approve what the agent was permitted to do. The full incident analysis is in Nine Seconds: What the PocketOS Incident Reveals About AIgentic Authorization.
Anthropic’s production data provides the rebuttal to the objection that risk budgets make agents operationally useless. Across current deployments, 80% of agent tool calls include at least one safeguard: restricted permissions or approval requirements. Only 0.8% of actions are irreversible.4 Risk budget controls are implementable at production scale without breaking operational utility.
What does a risk budget approval process look like in practice?
A board does not configure tool permissions. The board’s role is to require that the configuration exists and has been explicitly approved at the appropriate organizational level. The relevant governance question for each agent deployment is: has a risk owner, someone with explicit accountability for the outcome rather than the development team, who has defined and signed off on this agent’s authority scope, credential posture, reversibility gates, and blast radius limits? If the answer is no, the organization has a governance gap regardless of what the cost budget says.
EU AI Act and DORA compliance deadlines make risk budgets a regulatory obligation, not a best practice.
EU AI Act Article 9 requires operators of high-risk AI systems to establish a continuous risk management system across the AI lifecycle.5 Article 14 requires that high-risk AI systems be designed to enable effective oversight by natural persons. The compliance deadline for high-risk provisions is August 2, 2026. Irreversibility gates and human approval requirements on high-impact agent actions reflect the oversight intent of Article 14. For financial services entities, DORA Article 9 requires financial entities to implement protection and prevention controls as part of their ICT risk management obligations; defined capability scopes and blast radius containment address those requirements.6 Whether a specific deployment qualifies as high-risk under Annex III is a legal determination. The direction of travel is not: regulatory instruments are moving toward capability constraint as the governance standard, not cost constraint.
Shadow AI deployment renders every risk budget you have set incomplete.
A risk budget governs the agents the organization knows about. Microsoft reports that 29% of employees use unsanctioned AI agents for work tasks.7 By definition, those agents carry no defined authority scope, no credential governance, and no blast radius limits — no deployment decision was made that would have required one. The organization did not decide to accept that risk; no one approved the deployment. Sanctioned agents in a well-governed program can carry excellent risk budgets. The unsanctioned agents operating alongside them have undefined risk budgets by definition. A governance program that covers only sanctioned deployments is incomplete before it is tested.
What is the board’s accountability for shadow AI?
The organization cannot set a risk budget for agents it does not know exist. The board accountability is not to govern every unsanctioned tool but to require that the organization has visibility programs in place (agent discovery, usage monitoring, and policy enforcement) that reduce the shadow AI population. The risk budget question, applied to shadow AI, becomes: what is the organization’s tolerance for the unknown agents currently operating on its infrastructure?
Frequently asked questions
Does a risk budget require agents to pause before every action?
No. A risk budget defines the envelope of permitted action, not a per-action approval queue. Most agent actions fall within the defined authority scope and execute without human review. Irreversibility gates apply only to the subset of actions that are difficult or impossible to undo. Anthropic’s production data shows only 0.8% of agent actions in current deployments are irreversible.4 Risk budgets make agents governable without making them unusable.
Is a risk budget a governance control or a technical control?
It is both, and the distinction matters. A policy that states agents should not access production systems without approval is governance intent. A risk budget is the architectural enforcement of that intent: the tool permission configuration, the credential scope, and the network boundary that prevent access unless approval has been explicitly granted. Policy compliance and architectural enforcement are different outcomes. Boards should require both.
Where should the board start?
The most immediate question is not which risk budget to set but whether any explicit risk budget decision has been made for the agents currently deployed. A board that can affirm four things for every sanctioned agent is in a governed posture: defined authority scope, defined credential posture, defined reversibility gates, and defined blast radius limits. A board that cannot answer those questions has a governance gap. The next question is how many agents are running without one.
Footnotes
-
Yahoo Finance UK, “‘It took nine seconds’: Claude AI agent deletes company’s entire database,” April 28, 2026. https://uk.finance.yahoo.com/news/took-nine-seconds-claude-ai-101315747.html Scope confirmed: the Cursor AI agent deleted PocketOS’s entire production database and all volume-level backups; the site remained offline for more than thirty hours. Full incident analysis: Nine Seconds: What the PocketOS Incident Reveals About AIgentic Authorization. ↩ ↩2
-
OWASP GenAI Security Project, “Agentic AI Threats and Mitigations” (2025). https://genai.owasp.org/resource/agentic-ai-threats-and-mitigations/ See also OWASP GenAI Exploit Round-up Report Q1 2026. https://genai.owasp.org/2026/04/14/owasp-genai-exploit-round-up-report-q1-2026/ ↩
-
Diego Vega San Martín, “Risk Budgeting in Financial Firms and The Value of an Expert Opinion,” SSRN Working Paper 3070654, Southern Methodist University (2018). https://ssrn.com/abstract=3070654 The paper formalizes risk budgeting as “arguably the framework most financial firms use to manage risk”: a policy determining which decisions are fully delegated to the trader versus which require upper management approval. ↩
-
Anthropic, “Measuring AI Agent Autonomy in Practice” (2026). https://www.anthropic.com/research/measuring-agent-autonomy ↩ ↩2
-
Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 (EU AI Act). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32024R1689 Article 9 establishes the risk management system requirement. Article 14 establishes human oversight requirements. Compliance obligations for high-risk AI systems under Chapter III apply from 2 August 2026 per Article 113(2)(b). ↩
-
Regulation (EU) 2022/2554 of the European Parliament and of the Council on digital operational resilience for the financial sector (DORA). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32022R2554 Article 9 (Protection and prevention) requires financial entities to implement ICT security policies and controls to protect ICT assets from cyber threats and unauthorized access. ↩
-
Microsoft, “80% of Fortune 500 Use Active AI Agents: Observability, Governance and Security Shape the New Frontier” (February 2026). https://www.microsoft.com/en-us/security/blog/2026/02/10/80-of-fortune-500-use-active-ai-agents-observability-governance-and-security-shape-the-new-frontier/ ↩