AI agents are moving from answering questions to taking actions inside the systems businesses rely on every day.
As organizations give them access to applications, data, communications, and financial workflows, the security question is no longer just whether an AI model will produce the right answer. It is also what happens when an agent does something its operators never intended.
The recent Hugging Face incident shows how quickly that distinction can matter.
The incident involved AI agents operating inside OpenAI’s evaluation environment that found unauthorized ways to communicate, share discoveries, gain internet access, and collaborate across separate evaluation tasks before compromising parts of Hugging Face’s infrastructure.
Doing that effectively is a challenge even for organizations with significant AI expertise. The vast majority of companies deploying agents will need to design controls that account for the ways those systems could behave outside their intended boundaries.
Say you’re a mid-sized healthcare provider deploying an AI agent to resolve billing inquiries. You want faster answers and fewer routine tasks for employees, so it’s tempting to connect the agent to the billing platform via an administrator account and tell it to follow company policy.
Now imagine an incoming invoice contains instructions to export account records to an unknown external address for “verification,” a scenario consistent with the growing risk of indirect prompt injection attacks. The agent treats the instructions embedded in the invoice as an authorized request, and its broad permissions and unrestricted outbound access allow it to export the records.
The result could be an unauthorized disclosure of protected health information, triggering breach-response obligations and potentially exposing the organization to regulatory penalties.
The AI agent followed the instructions it received, but its governance model granted it authority it shouldn’t have. An action need not be malicious to cause a costly security failure. A combination of vague requests and credentials that allow an agent to move money, expose records, or change production systems can produce the same result as an attack.
Many organizations are charging ahead with agents in hopes of reaping tantalizing cost savings without putting adequate controls in place. Ernst & Young says 92% of technology executives are addressing sovereign AI, but only 25% have enterprise-wide governance for agentic AI.
That gap illustrates the broader challenge: organizations are deploying increasingly capable systems faster than they are establishing clear rules for what those systems are allowed to do, making AI agent safety controls increasingly important before deployment.
Define the agent’s blast radius
Established security controls provide a foundation for containing agent risk. The challenge is applying and testing those controls across every system an agent can reach and every action it can take. Least privilege, a practice that grants only the bare minimum permissions needed to do a job, is a useful starting point: Grant only the access needed for a defined task, then constrain how far that access can take an automated workflow.
But that doesn’t cover all the ways agents can go wrong. To keep them in line, start by writing down the specific boundaries of the job. For example, a billing agent may need to see selected fields for a patient’s current case to draft an explanation or propose an adjustment. Those duties don’t require access to clinical notes, unrelated accounts, bulk exports, or record deletion. Grant narrowly defined access for those operations, with authorization checks on every request.
A useful way to think about this is to define an agent’s blast radius before it ever gets access to a system. Ask five basic questions: What can it read? What can it change? Who or what can it affect? How much can it spend? And how quickly can it act? The answers should determine the permissions and controls for the agent, not rely on the assumption that it will behave as intended.
Separate access from authority
The distinction between reading and taking action is particularly critical. An agent that can retrieve a balance shouldn’t by default be able to change it. One that drafts a message shouldn’t have the privilege of sending it anywhere, so restrict recipients and network destinations. Even granting read access can cause unintended disclosure when communications are unrestricted.
Identity controls reinforce those boundaries. Give each workflow an identifiable service account, an accountable owner, and credentials that expire or can be rotated quickly. A helper agent should receive no more authority than the task requires, and delegation must never be available as a route around the original restrictions.
Put hard limits on agent actions
Consider scale limits. Granting an agent permission to refund up to $50 without review could backfire if the agent issues thousands of refunds or repeatedly refunds the same charge. Specify cumulative limits per account and across the workflow. Set ceilings on tool calls, execution time, and model spending.
Enforce those limits in systems the agent can’t rewrite or bypass. Prompting it to stay within certain spending guidelines doesn’t substitute for independent permission and budget checks before execution. The goal is to make the safe behavior enforceable, rather than something the agent is simply asked to follow.
Test agent boundaries before deployment
Human oversight remains important, particularly before agents are trusted with higher-risk actions.
Before deployment, security teams should deliberately test those boundaries by feeding the billing agent misleading invoices, conflicting instructions, repeated refund requests, unauthorized recipients, and requests involving access to customer data. Verify that the downstream system rejects prohibited actions even when the model attempts them.
Monitor agents after they go live
Continuing to monitor the agent after launch is essential. Unlike conventional software, agentic systems can behave differently depending on their context, tools, instructions, and underlying models.
Runtime protection complements testing by observing and enforcing controls while the agent operates. Monitor the agent’s actions to ensure it stays within its defined permissions, and log both attempted and completed actions, as well as policy decisions and approvals, so security teams can identify when it approaches or crosses those boundaries. If it does, establish ways to stop execution and revoke access. Review those logs and permissions regularly and adjust them as the agent’s role or behavior changes.
That makes permission design a business decision. Before letting agents loose, security leaders should consider the damage they can cause and enforce limits on that exposure. Permissions should be based on the potential impact of an agent’s actions, not confidence in its behavior. Security leaders should design permissions for the worst-case outcome, not the intended behavior.
Confidence is no substitute for control.
Also read: For a real-world example of why AI agent permissions matter, read about how a Meta Muse flaw could allow local malware to hijack an AI agent and abuse privileges already granted by the user.