AI Agent Benchmarks 2026 Explained
By Loïc Jané · Founder, Fleece AI
What Is an AI Agent Benchmark?
An AI agent benchmark is a standardized test of whether an AI model can complete real work: calling APIs, writing code, operating applications or finishing multi-step professional tasks. The benchmarks that matter for agents in September 2026 are APEX-Agents, MCP-Atlas, τ³-bench, Toolathlon-Verified, Terminal-Bench 4.0 and OSWorld 2.0.
At a Glance — Updated September 24, 2026. Leaders on each independent leaderboard:
- APEX-Agents — professional multi-app tasks: Claude Opus 5.5, 73.5%
- MCP-Atlas — tool use across real MCP servers: Muse Spark 1.1, 88.1%
- τ³-bench (banking) — multi-turn tool calls under policy: Qwen 3.8 Max, 55.2%
- Toolathlon-Verified — 600+ tool catalogs: GLM-5.3-Flash, 78.4%
- Terminal-Bench 4.0 — agentic work in a terminal: GPT-6 Astra, 58.2%
- OSWorld 2.0 — long GUI tasks: Claude Opus 5, 77.67% partial reward (v2.1)
Retired or frozen since February 2026: SWE-bench Verified, BFCL v4, OSWorld-Verified and Terminal-Bench 2.0. See below.
Benchmarks changed as much as models did in 2026. Several tests from the start of the year became saturated, flawed or unmaintained, and were replaced by harder versions. A score is only meaningful alongside its benchmark version, its source and its date, so every figure below includes all three.
For per-model tool-calling rankings, see Best AI Model for Tool Calling 2026. For a model-by-model comparison, see Best AI Models for Automation 2026.
What Changed Since February 2026
| Benchmark | Status in September 2026 |
|---|---|
| SWE-bench Verified | Retired by OpenAI (February 2026): 59.4% of audited hard tasks had flawed tests or descriptions, with signs of training contamination. OpenAI recommends SWE-bench Pro. |
| BFCL v4 | Frozen: last updated April 12, 2026. The newest models listed are Claude Opus 4.5, GPT-5.2 and Gemini 3 Pro. |
| τ²-bench | Replaced by τ³-bench. Telecom is saturated (97.8–98.2%), and 50+ airline and retail tasks were fixed. New models are run on the banking domain only. |
| Terminal-Bench 2.0 | Superseded by 2.1 (May), 3.0 (July) and 4.0 (August 28). 4.0 scores (roughly 20–66%) are not comparable with 2.0 scores (roughly 60–80%). |
| OSWorld-Verified | Replaced by OSWorld 2.0 (June 26), which has 108 long-horizon tasks. |
| MCP-Atlas | Re-scored in April 2026 with a new judge and a 100-tool-call budget. |
| Toolathlon | Now Toolathlon-Verified, a new score series. |
APEX-Agents (Professional Tasks)
What it measures: whether an agent can finish long, cross-application tasks from investment banking, management consulting and corporate law. The work spans files, spreadsheets, chat logs and calendars across 31 simulated workplaces and 240 tasks.
Why it matters: it is the closest public benchmark to real business automation.
| Model (effort) | Pass@1 |
|---|---|
| Claude Opus 5.5 (max) | 73.5% |
| Claude Fable 5.1 (max) | 68.6% |
| Gemini 3.7 Flash (high) | 67.8% |
| Claude Opus 5 (max) | 65.8% |
| Grok 4.6 (xhigh) | 65.3% |
Key insight: effort settings change results a lot. Opus 5.5 drops from 73.5% at max effort to 52.5% at medium, its API default. By job, GPT-6 Astra leads corporate law (73.4%) and Opus 5.5 leads management consulting (80.0%).
Source: Mercor APEX-Agents leaderboard (dataset v1.1), retrieved September 24, 2026. Only the top five are published.
MCP-Atlas (Tool Use Across MCP Servers)
What it measures: 1,000 tasks across 36 real MCP (Model Context Protocol) servers and 220 tools. A task passes when the answer covers at least 75% of the expected claims.
Why it matters: MCP is now the standard way to connect agents to external apps. This benchmark tests it directly.
| Model | Pass rate |
|---|---|
| Muse Spark 1.1 (Meta) | 88.1% |
| Claude Opus 5 (xhigh) | 85.8% |
| Qwen3.8-2.4T-A95B | 84.5% |
| GLM-5.3 | 84.2% |
| Gemini 3.5 Flash (high) | 83.6% |
| Kimi K3 (max) | 82.3% |
| GPT-5.6 Sol | 81.8% |
Key insight: scores carry a margin of about ±2 points, so the top ten are nearly tied. Three open-weights models (Qwen3.8, GLM-5.3, Kimi K3) sit in that group.
Source: Scale AI MCP-Atlas leaderboard, retrieved September 24, 2026.
τ³-bench (Multi-Turn Tool Calls Under Policy)
What it measures: a simulated customer conversation where the agent must call tools, search a 700-document knowledge base and follow business rules. τ³ extends τ²-bench with a banking domain and a voice mode.
Why it matters: support and operations agents fail less often on the tool call itself than on applying the right policy. τ³ banking tests both.
| Model (effort) | Banking pass^1 |
|---|---|
| Qwen 3.8 Max (xhigh) | 55.2% |
| Claude Opus 5 (max) | 48.7% |
| Grok 4.5 (high) | 47.9% |
| GPT-5.6 Sol (xhigh) | 46.9% |
| GPT-5.5 (xhigh) | 44.6% |
Key insight: telecom scores reached 98%, which is why Sierra moved to banking. Scores around 50% show how much room is left.
Source: Sierra τ-bench public submissions, retrieved September 24, 2026.
Toolathlon-Verified (Large Tool Catalogs)
What it measures: 108 tasks across 32 MCP servers exposing 604 tools, plus local toolkits.
Why it matters: agents connected to hundreds of apps must choose the right tool among many. Accuracy usually drops as the catalog grows.
| Model (effort) | Pass@1 |
|---|---|
| GLM-5.3-Flash (max) | 78.4% |
| Kimi K3 (max) | 76.5% |
| Claude Opus 4.8 (max) | 76.2% |
| Muse Spark 1.2 (xhigh) | 75.9% |
| DeepSeek V4 Pro 0813 (max) | 74.4% |
Key insight: the top two are open-weights models. Claude Opus 5 and 5.5, GPT-6 and Gemini 3.8 Flash are not listed yet.
Source: Toolathlon-Verified leaderboard, retrieved September 24, 2026.
Terminal-Bench 4.0 (Agentic Work in a Terminal)
What it measures: whether an agent can complete hard software and systems tasks from a terminal: navigating code, running commands, debugging and configuring.
Why it matters: it is the reference benchmark for coding agents such as Claude Code, Codex and Cursor.
| Model (effort) | Agent | Score |
|---|---|---|
| GPT-6 Astra (max) | Codex | 58.2% |
| Claude Fable 5.1 (max) | Claude Code | 57.9% |
| Claude Opus 5 (xhigh) | Claude Code | 53.9% |
| Claude Fable 5 (max) | Claude Code | 44.5% |
| GLM-5.3 (max) | Claude Code | 41.8% |
| Grok 4.7 (xhigh) | Grok Build | 37.6% |
| GPT-5.6 Sol (max) | Codex | 37.3% |
Key insight: Claude Opus 5.5 is not on the leaderboard yet; Anthropic reports 66.4% on its own setup. Vendors also report slightly different numbers for the same model (Opus 5 appears as 51.8%, 52.3% and 52.6% in Google, Anthropic and OpenAI tables). Prefer the public leaderboard when comparing models.
Source: tbench.ai leaderboard, retrieved September 24, 2026. Version 4.0 (August 28, 2026) revises 3.0 and is not comparable with 2.0 scores.
OSWorld 2.0 (Long GUI Tasks)
What it measures: 108 long tasks in a real desktop environment with 31 self-hosted websites. Most tasks take a human more than an hour. It reports a strict pass/fail score and a partial-credit score.
Why it matters: many business apps have no API. Agents that operate the interface directly can still automate them.
| Model | Version | Binary | Partial |
|---|---|---|---|
| Claude Opus 5 (max) | v2.1 | 44.33% | 77.67% |
| Claude Opus 5 (max) | v2026.08.08 | 31.43% | 68.31% |
| GPT-5.6 Sol (max) | v2026.08.08 | 27.34% | 62.72% |
| Claude Opus 4.8 (max) | v2026.06.24 | 20.6% | 54.8% |
Key insight: full completion collapses on long tasks. Every model plotted scores 0% on tasks longer than about 163 minutes. Newer models (Opus 5.5, Fable 5.1, GPT-6) have only vendor-reported OSWorld 2.0 numbers so far.
Source: OSWorld 2.0 leaderboard, last updated September 17, 2026.
SWE-bench Pro (Coding)
What it measures: real GitHub issues that the agent must fix, verified by the repository's test suite.
Why it matters: it replaced SWE-bench Verified after OpenAI's February 2026 audit found flawed tasks and training contamination in Verified. None of the major labs reports SWE-bench Verified in its September 2026 launch posts.
We do not list scores here: labs now report a mix of SWE-bench Pro, DeepSWE, FrontierCode and CursorBench, often on their own harnesses. Check swebench.com for current results.
ARC-AGI (Abstract Reasoning)
What it measures: reasoning on visual puzzles the model has never seen. It tests generalization rather than memorized knowledge.
Why it matters: agents meet unfamiliar situations. ARC measures how well a model handles something new.
Scores change often. Check the live ARC Prize leaderboard for current results.
BFCL v4 (Function Calling) — Frozen
The Berkeley Function Calling Leaderboard tested function name accuracy, argument correctness, parallel calls and multi-turn use. It has not been updated since April 12, 2026, when Claude Opus 4.5 led at 77.47% overall. It is still a useful test design, but it cannot rank current models.
Source: BFCL, retrieved September 24, 2026.
Which AI Agent Benchmark Should You Use?
| If your agent... | Look at |
|---|---|
| Connects to many apps through MCP | MCP-Atlas |
| Has hundreds of tools available at once | Toolathlon-Verified |
| Handles customer or operations conversations under policy | τ³-bench (banking) |
| Runs long professional workflows | APEX-Agents |
| Writes or fixes code | Terminal-Bench 4.0, SWE-bench Pro |
| Operates GUIs and web apps without APIs | OSWorld 2.0 |
No single benchmark ranks every model. The September 2026 releases (Claude Opus 5.5, GPT-6 Astra/Sol/Luna, Gemini 3.8 Flash) are still missing from most independent leaderboards.
The Complete Leaderboard (September 2026)
| Benchmark | Leader | Score | What it tests |
|---|---|---|---|
| APEX-Agents | Claude Opus 5.5 (max) | 73.5% | Professional multi-app tasks |
| MCP-Atlas | Muse Spark 1.1 | 88.1% | Tool use across MCP servers |
| τ³-bench banking | Qwen 3.8 Max | 55.2% | Multi-turn tool calls under policy |
| Toolathlon-Verified | GLM-5.3-Flash | 78.4% | Large tool catalogs |
| Terminal-Bench 4.0 | GPT-6 Astra | 58.2% | Terminal-based agentic work |
| OSWorld 2.0 (v2.1, partial) | Claude Opus 5 | 77.67% | Long GUI tasks |
No single model leads everything. Anthropic leads the professional-task and GUI benchmarks, OpenAI leads Terminal-Bench, and Meta, Alibaba (Qwen) and Z.ai (GLM) lead the three tool-use benchmarks.
What These Benchmarks Mean for AI Agents
For platforms like Fleece AI that connect agents to thousands of apps, the tool-use benchmarks (MCP-Atlas, Toolathlon-Verified, τ³-bench) and APEX-Agents are the most relevant. Fleece AI lets you choose the model for each agent. Its default is GLM 5.3 Flash; Kimi K3, also on every plan, scores 82.3% on MCP-Atlas and 76.5% on Toolathlon-Verified.
Frequently Asked Questions
What is the AI agent benchmark?
There is no single one. The term covers tests that check whether AI models can complete real work. In September 2026, the most relevant are APEX-Agents, MCP-Atlas, τ³-bench, Toolathlon-Verified, Terminal-Bench 4.0 and OSWorld 2.0.
Is SWE-bench Verified still used?
Not by the major labs. OpenAI stopped reporting it in February 2026 after an audit found that 59.4% of audited hard tasks were flawed, with signs of training contamination. It recommends SWE-bench Pro instead.
Why do labs report different scores for the same model?
They run benchmarks on different harnesses, effort settings and versions. Claude Opus 5 scores 51.8%, 52.3% or 52.6% on Terminal-Bench 4.0 depending on which lab's table you read. Independent leaderboards are the fairest basis for comparison.
Which single benchmark best predicts AI agent performance?
For business automation, APEX-Agents covers the widest range of real work. For tool reliability specifically, use MCP-Atlas and Toolathlon-Verified.
Where can I find official benchmark results?
APEX-Agents at mercor.com, MCP-Atlas at labs.scale.com, τ³-bench at taubench.com, Toolathlon-Verified at toolathlon.xyz, Terminal-Bench at tbench.ai, OSWorld 2.0 at osworld-v2.xlang.ai, and ARC at arcprize.org.
Keep Reading
- Best AI Model for Tool Calling 2026 — which model calls APIs most reliably
- Best AI Models for Workflow Automation 2026 — full model comparison
- Best AI Agents for Business Automation 2026 — platforms compared
Start automating with AI agents — deploy your first AI agent with Fleece AI.