τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains

The next pane walks the POMDP setup — agent / tools / policy / user-simulator — as four components. The second pane replays the tau-airline example where the user wants to change a basic-economy flight; the agent follows the policy and offers cancel-and-rebook. The third pane prints the pass-at-1 versus pass-at-k metric and the GPT-4o FC numbers. The fourth pane prints the main-results table with twelve models and two domains. The fifth pane breaks down the 36 failed GPT-4o function-calling trajectories on tau-retail.

primer — what you need before you start

the task. tau-bench benchmarks LLM agents on real customer-service domains — retail and airline. The agent must follow domain rules, query a real (hidden) database via APIs, and resolve the user's underlying request. Strong models score only 46-53% pass@1 — real customer service is hard.

how it was done before. Prior agent benchmarks used Q&A-style tasks (HotpotQA, FEVER) or static goal descriptions (ALFWorld). tau-bench is the first to combine: (1) realistic domain rules, (2) a real database backend, (3) a user simulator that adapts based on agent behavior. The agent must read the rule book and the database schema, then act accordingly.

glossary — terms you will see.
POMDP — Partially Observable Markov Decision Process; standard RL framing for sequential decision-making under uncertainty.
state S — the hidden database (agent never sees rows directly).
action A — API call to the domain toolset, or a message to the user.
observation O — what the agent sees back: API response, user message.
transition T — deterministic in tau-bench (no randomness in tool calls).
instruction R — the ground-truth goal: final DB state must match.
user simulator — an LLM that plays the customer, with persona and hidden goal.

symbol table — what the letters mean.
$S_t$ — hidden state at step t.
$A_t$ — action at step t: $A_{db}$ (database call) or $A_{user}$ (message).
$O_t$ — observation at step t: $O_{db}$ or $O_{user}$.
$\pi(a \vert s, h)$ — policy: action given state and conversation history $h$.
$\text{pass@k}$ — probability that all k independent trials succeed.
$\text{pass@1}$ — single-trial success rate.

tone legend — what the [brackets] mean.
[obs] observation   [tool] tool / domain call
[pol] policy / POMDP   [metric] pass@k metric
[eq] equation   [take] key takeaway
[sys] system / meta   [limit] paper-reported limit

Six panes cover, in pedagogical order: POMDP setup with the tau-airline example; pass@1 vs pass@k metric; main results across 12 models; failure mode analysis of GPT-4o FC; the paper's own limits. Each pane ends with key takeaways so you can stop after any section and still know what you learned.

point: tau-bench formalizes tool-agent-user interaction as a POMDP. The agent calls deterministic Python API tools (7 for tau-retail, 6 for tau-airline) and only sees the returned observations, never the database rows directly. A domain policy is injected into the agent's system prompt. The user is an LM simulator (GPT-4-0613) seeded with a task instruction that requires multi-turn clarification — if the airline customer cannot change her basic-economy reservation, she will ask the agent to cancel and rebook.

Four-component ASCII diagram printed first, then a one-line interpretation per component.

API tools (deterministic) domain policy (system prompt) user simulator (LM) trajectory step observation (tool return)

point: replay the verbatim trajectory from the published example: the user asks to change a basic-economy flight; the agent retrieves reservation JK9019, sees cabin=basic_economy and creation time 2024-05-14 18:00 EST (current time 2024-05-15 15:00 EST, about 21 hours elapsed); the policy forbids modification; the agent offers cancel-and-rebook as the only legal path; the user accepts and cancels; the user types ###STOP### to end the episode.

Trajectory text and policy excerpt are verbatim from the published trajectory — same reservation id, same timestamps, same agent reasoning.

user trajectory step API tool call observation policy excerpt

point: pass-at-1 = E[c/n] is the average success rate over n independent trials. pass-at-k = E_task[(c choose k) / (n choose k)] is the probability that all k trials succeed. A high pass-at-1 with a low pass-at-k means the agent succeeds sometimes but is unreliable across runs.

Formula and the GPT-4o FC reliability gap printed as two ASCII blocks — one for the metric, one for the numbers.

metric (formula / number) reliability gap

point: GPT-4o function calling leads at 61.2 / 35.2 / 48.2 average, but no model breaks 50% on tau-airline. GPT-4 turbo / 32k trail closely. GPT-3.5-turbo sits at 20.0 / 10.8 / 15.4 — the older generation is roughly a third of the new one. Claude-3-opus at 44.2 / 34.7 / 39.5 is the strongest non-OpenAI model. Llama-3-70B uses ReAct instead of function calling (table caption), which is why it sits at 14.8 / 14.4 / 14.6 — well below the function-calling models on the same domains.

Main-results sheet — 12 models × 4 columns, paper-report numbers, no rounding.

paper-report number

point: the failure analysis section breaks down 36 failed GPT-4o function-calling tau-retail trajectories into three failure types — wrong argument or info provided (the challenge of complex database reasoning), incorrect decision-making (the challenge of domain understanding and rule following), and partial resolution of compound requests. Removing the policy from the system prompt drops GPT-4o by 22.4% on tau-airline but only 4.4% on tau-retail — the policy matters most where the rules are not commonsense.

Breakdown pie plus three failure-type descriptions, each printed as a block with the paper's own phrasing.

failure / limit paper-report %