AI Agent Payment Authorization After 4,371 Attacks

APort Vault replayed 4,371 human-written attacks against 14 language models from 8 labs. And the takeaway for AI agent payment authorization is blunt: the decision to move money should never live inside the model.

The Open Agent Passport (OAP), the open specification APort built and stress-tested in that exercise, intercepts every tool call synchronously before execution, checks it against a signed declarative policy.

And returns a deterministic allow or deny. The arXiv paper behind it reports a measured median of 53 ms for enforcement across 1,000 calls. A denied transfer does not happen, full stop. That is the entire thesis. And it is more radical than it sounds because almost nobody builds agents this way.

The Pre-Action Gap Is Where Your Money Leaks

The paper names the core problem the pre-action authorization problem: the gap between an AI agent deciding to act and the actual execution of the tool call.

A compromised model can still execute harmful actions even when its output is being filtered.

In payment terms, your guardrail prompt and your output filter watch what the model says. Nothing in that setup watches what the tool actually does.

I build agent automations for small clients, and this is the part that keeps me honest. It is trivially easy to ship an agent that can call a payment or purchasing tool, add a stern system prompt about being careful with money, and call it safe. The prompt is a request. The tool call is an action. Those are different layers, and only one of them moves real funds. APort's framing in its CTF writeup is sharper than most vendor positioning: the question is not "can you jailbreak the model," it is which agentic risks show up when strangers try to social-engineer an AI banking agent into unauthorized transfers. That is a question about plumbing, not personality.

What a Zero-Capability Passport Looks Like in Code

The clearest demonstration is Vault's Level 5.

Per APort's benchmark description, the level guards a $1,000,000 vault value with a 30-turn limit.

And the agent holds a "zero-capability passport." The conversation model can be sweet-talked, bargained with, or worn down.

The policy engine cannot, as it evaluates deterministically from the signed passport rather than from anything the model believes.

The published passport JSON for that level looked like this:

Read those limits again.

Zero transfers per day.

Zero dollars maximum.

No allowed recipients. The passport is active and the agent is conversational and charming. And it simply has no legal authority to move money. When the model gets convinced and decides to transfer anyway, the runtime calls a pre-action authorization hook, loads the passport, checks whether `transfer` is in the capability set, finds that it is not. And returns deny before any tool call leaves the process. The model's decision gets overruled mechanically. If you have ever lost an argument with a payment gateway's error response, you understand the vibe: the agent does not get to negotiate with the policy engine.

This is the design insight worth stealing even if you never touch APort. Put the ceiling in a signed artifact that the model cannot edit. And enforce it in the runtime, not in the prompt.

4,371 Attacks, 14 Models, 225,964 Evaluations

Here is why I take this seriously instead of filing it under vendor marketing.

APort released vault-benchmark-v1 on Hugging Face: 4,371 attacks written by humans in 1,128 sessions of a public competition between March and August 2026, replayed against 14 language models from 8 labs acting as simulated bank tellers with no real money moving. Each attack runs in two conditions: model alone.

And behind a pre-action authorization layer where every transfer the model requests is checked against the passport before it executes.

The scale is what makes it citable. The frozen snapshot from 2026-09-10 covers 225,964 completed evaluations of a 244,776-evaluation grid, across five policy configurations and two replay tracks. And 5,395 error rows are disclosed per cell in the results files rather than quietly dropped. A vendor that publishes its own error ledger alongside its own benchmark is a vendor behaving like a research group. And that buys a lot of credibility with me.

The cynic's note, since I have one: APort sells the authorization layer it benchmarked. The honest move is to download the dataset and check the model-alone versus behind-the-layer comparison yourself. That is exactly why releasing it openly matters more than any blog post. The public CTF itself ran 5 levels with $6,500 in prizes, Levels 1 and 2 free to attempt and Levels 3 through 5 paying up to $5,000. So the attack corpus was bought with real prize money, not generated in a spreadsheet.

What a Solo Operator Should Actually Do This Week

You do not need an enterprise security team to apply this.

The OAP spec is Apache 2.0 licensed, free to use today, and the reference implementation runs via `npx @aporthq/apport-agent-guardrails`. APort positions it as a "living specification for AI agent identity, capabilities. And policy enforcement" that is verifiable across platforms with one /verify call. And it supports both local and hosted evaluation, so you can start inside your own stack.

The architecture APort recommends is a clean split that any small builder can copy:

- OAuth/OIDC for authentication - prove who is calling. - OAP for per-action authorization - prove this specific action is permitted, checked before execution, with denied calls returning structured errors you can handle in code. - Evaluation tooling for systemic testing - replay adversarial transcripts against your agent before strangers do it for free.

For a small business, this reframes the buying decision.

You are not choosing between "safe" and "unsafe" agent platforms.

You are checking one thing: when your agent calls a tool that can spend money, is there a signed policy check in the loop before execution.

And is there an audit record of every allow and deny after it?

If the answer is "the model was instructed to be careful," you do not have a control, you have a wish. The pattern works with existing stacks, the decision objects give you an audit trail your clients' lawyers can actually read. And the whole thing costs nothing but integration time.

Before you hand an agent a payment key, write down what it may do, sign it. And enforce it at the hook. If you want help mapping which of your agent's tool calls need that treatment, that is the work my agency does every week - tell me what your agents can touch and I will tell you where the pre-action gap is.