AutomationBench-AA: The New Benchmark That Shows AI Agents Still Break the Rules

There's a new agentic benchmark worth paying attention to, and its most interesting finding isn't about which model wins — it's that every single model tested breaks business rules while doing the work.

AutomationBench-AA launched on July 6, built from Zapier's AutomationBench and run by Artificial Analysis on Zapier's private benchmark subset. What makes it different from most agent evaluations is the premise: it doesn't just ask whether a model can complete a SaaS workflow task, it asks whether the model can complete the task without violating any of the guardrails that represent real business rules. And on that measure, the results are humbling across the board.

What the Benchmark Actually Tests

This is a serious piece of evaluation infrastructure. Models have to complete 657 tasks spanning Finance, HR, Marketing, Operations, Sales, and Support — the actual functional areas where automation gets deployed in real companies. The tasks run across 40 simulated SaaS environments including Gmail, Google Sheets, Slack, Salesforce, Zendesk, Jira, and HubSpot.

A few design choices make this harder than it sounds:

Models interact with each app through REST APIs and have to discover the endpoints they need through structured tool calls — while navigating environments seeded with irrelevant and sometimes deliberately misleading records. That's much closer to real-world messiness than a clean sandbox.

Grading is programmatic and deterministic. Artificial Analysis notes that tasks are scored purely on whether the correct data ended up in the right systems — nearly 12,000 assertions built by Zapier check the final state of the environment. Each assertion is either an objective the agent must achieve, or a guardrail that starts in a passing state and must not be broken. Each task runs once with a 50-turn cap.

The headline score for AutomationBench-AA is the share of objectives a model completes without breaking any guardrails — which is a meaningfully stricter bar than raw task completion.

Who Leads

According to Artificial Analysis, Anthropic's models take the top two spots: Claude Fable 5 leads at 48.6%, with Opus 4.8 essentially tied at 48.5%. Google DeepMind's Gemini 3.5 Flash follows at 42.6%, and OpenAI's GPT-5.5 (xhigh) at 42.1%.

There's a wrinkle with the leader, though. Fable 5 fell back to Opus 4.8 on roughly 18% of tasks due to Anthropic's new classifier. It completes 73% of task objectives outright, but that fallback behavior likely explains why it barely edges out Opus despite being the nominally more capable model. In practice, Fable 5 and Opus 4.8 are performing at the same level here.

On the open weights side, Z.ai's GLM-5.2 (max) leads at 27.8%. That places the open weights frontier about 10 points behind Gemini 3.1 Pro Preview — and notably, it gets there with substantially higher guardrail violations per task. The open weights models can do the work; they're just messier about respecting the rules while they do it.

The Real Story: Everyone Breaks the Rules

This is the finding that actually matters. Every model evaluated at launch triggered guardrail violations. Not some. All of them.

The spread is wide. Guardrail violations range from 0.46 per task for Gemini 3.5 Flash to 1.26 per task for Qwen3.7 Plus. That's a nearly 3x difference in how often models break business rules while completing work.

Artificial Analysis introduces a useful way to think about this: violation-adjusted efficiency, measured as objectives completed per guardrail violation. On that metric, Gemini 3.5 Flash comes out on top at 15.0 objectives per violation — ahead of Claude Opus 4.8 (max) at 13.5. So the model that gets the most work done per rule broken isn't the top scorer overall; it's Gemini 3.5 Flash.

For anyone actually thinking about deploying agents into production SaaS workflows, this is the number to internalize. Raw task completion tells you what a model can do. Guardrail violations tell you what it might break while doing it — and in a real business, breaking a rule can cost more than completing a task is worth.

Cost Tells Its Own Story

Cost per task spans more than an order of magnitude. At the cheap end, DeepSeek V4, Gemini 3.1 Flash-Lite, and Qwen3.7 Plus come in under 5 cents per task. At the expensive end, Claude Opus 4.8 (max) runs close to $1.50 per task.

The standout on price-performance is Gemini 3.5 Flash. It delivers its third-place 42.6% score at $0.49 per task — effectively matching GPT-5.5 (xhigh, 42.1%) at roughly 37% of GPT-5.5's $1.32 per-task cost. If you're weighing capability against budget for workflow automation, that's a hard combination to argue against.

The leaders aren't automatically the most expensive, which is a useful reminder that top-of-leaderboard and best-value are different questions.

How Models Actually Work Differently

One of the more interesting details in the Artificial Analysis breakdown is how differently these models approach the same tasks.

GPT-5.5 (xhigh) is action-intensive — it averages 49 tool calls across 25 turns per task. Claude Opus 4.8 (max) is more deliberate: 35 tool calls packed into just 14 turns, and it breaks fewer guardrails while doing it (0.55 vs 0.66 violations per task). Grok 4.3 (high) takes the fewest turns at 13, but underperforms models that persist longer — consistent with declaring tasks complete prematurely rather than actually finishing them efficiently.

That last point is worth sitting with. Fewer turns isn't the same as more efficient. Some models are quitting early and calling it done.

Where Agents Still Struggle

Task difficulty varies sharply by business domain, and Finance is the clear weak spot. Across all models evaluated, agents complete roughly one-third of Finance objectives — about half the rate they manage on Support and Operations tasks (~60%).

That's an important signal for anyone planning automation rollouts. The functions where agents are most reliable right now are support and operations. Finance workflows — where accuracy matters most and mistakes are most expensive — are exactly where current agents are weakest. That's an uncomfortable alignment of risk and capability.

Why This Benchmark Matters

Most agent benchmarks measure whether a model can complete a task. AutomationBench-AA measures whether it can complete a task without breaking things it wasn't supposed to touch — which is the actual bar for production deployment. A model that finishes 73% of objectives but trips a guardrail every task isn't production-ready; it's a liability with good intentions.

The takeaways for anyone evaluating agents for real SaaS automation:

Anthropic's Fable 5 and Opus 4.8 lead on raw guardrail-clean completion, but Gemini 3.5 Flash offers the best balance of capability, rule-adherence, and cost. Open weights options like GLM-5.2 are closing the capability gap but still lag on discipline. And regardless of which model you pick, Finance workflows need human oversight for now — the automation isn't there yet.

Full results are published by Artificial Analysis, with the underlying benchmark, paper, and code available through Zapier's release.