τ-bench (tau2-bench)
τ-bench (tau2-bench) is an evaluation tool maintained by Sierra138, version 1.0.1, released Jul 22, 2026. Its code is public13 under the MIT License3.
τ-bench is Sierra's open-source simulation framework for evaluating customer-service AI agents. In each domain an agent must follow a written policy and use tools while a simulated user takes part in the conversation, in turn-based text mode or full-duplex voice mode. The current repository, which carries the τ³-bench update, covers airline, retail, telecom, and banking-knowledge domains plus a mock domain.28
- Repository: GitHub repository (sierra-research/tau2-bench) (external site: github.com)
- Release notes: Release 1.0.1 (external site: github.com)
- Documentation: Task schema and evaluation guide (external site: github.com)
- Paper: τ²-bench paper (arXiv 2506.07982) (external site: arxiv.org)
- License: License (MIT) (external site: raw.githubusercontent.com)
Availability and license
Overall availability
Source code and domain task files on GitHub under the MIT License. Running an evaluation requires API access to the language models used for the agent and the simulated user.132
Availability is separate from permission: read the license before using or redistributing.
Component reuse rights
- weights
- Unknown — no complete fact-level rights review
- code
- Unknown — no complete fact-level rights review
- data
- Unknown — no complete fact-level rights review
- documentation
- Unknown — no complete fact-level rights review
No complete system-rights review is recorded for this release.
Public materials checklist
| Item | Status | Notes and evidence |
|---|---|---|
| CodeIs the evaluation code published? | Public | Published on GitHub under the MIT License.13 |
| Tasks / dataAre the tasks or test data available? | Public | The domains README documents per-domain data files stored in the repository under data/tau2/domains: policy, task definitions, voice tasks, task splits, and databases.4 |
| MethodologyIs the method for scoring described? | Public | The evaluation guide explains how rewards combine database-state, communication, and assertion checks, and the README cites the τ-bench, τ²-bench, τ-Voice, and τ-Knowledge papers.527 |
| ReproducibilityAre instructions for reproducing results published? | Public | The README documents installation with uv, API-key setup through LiteLLM, and example run commands; versioned releases and a changelog are published, and a pre-v1.0.1 tag is kept for reproducing earlier grading.26 |
| LimitationsAre known limitations documented? | Partial | The README states that, after grading fixes to the banking_knowledge domain, results from versions before 1.0.1 are not comparable with later results (other domains are unaffected), and the evaluation guide notes that the reference action sequence is only one valid solution path. No general discussion of the benchmark's validity limits was found in the pages read.25 |
What it is useful for
Run and use notes
- The README requires Python 3.12 or 3.13 (>=3.12, <3.14) and installs with uv; voice features need extra system packages such as portaudio and ffmpeg.2
Organization context
In the news
Sierra releases τ-bench 2.0.0 with six language packs and frozen voice and text benchmarks
Sierra's τ-bench repository published τ-bench 2.0.0, which its release notes date October 19, 2026 and title "Multilingual and Voice". The notes say a
--languageoption now selects one of six language packs (English, Spanish, Portuguese, Hindi, Chinese and Korean) with localized databases, tasks and caller personas for the airline, retail and telecom domains, and that frozen, pinned benchmark manifests such asvoice-v2andtext-v1replace ad-hoc task selection. The release also adds anintakedomain for entity capture, a post-hoctau2 judgessuite that the notes say is diagnostic only, and 2.0 leaderboard submission packages. The notes state that 2.0 results are not comparable with 1.x results and point to thev1.0.1tag for comparable runs. USASI has not tested this release.Sierra open-sources Hyper-τ-bench, an evaluation of models that build customer-service agents
On September 8, 2026 Sierra published Hyper-τ-bench, a long-horizon evaluation that measures whether a model can build a customer-service agent rather than only act as one. A developer agent works in a sandboxed workspace with a simulated business's records and a simulated client it can message, must recover the requirements and turn the business's actions into tools, and the finished agent is then tested on held-out tasks the developer never saw. Sierra says it is open-sourcing the benchmark, with code in the sierra-research/hyper-tau-bench repository, a public leaderboard, and a paper on arXiv; the post does not state a license. USASI has not run the benchmark.
U.S. eligibility
Eligible · basis: U.S.-governed project
The project is maintained in the sierra-research GitHub organization, its MIT license names Sierra Research as copyright holder, and Sierra's own blog says Sierra introduced τ-bench, presents τ²-bench as building on it, and links to this repository. Sierra Technologies, Inc. is a Delaware corporation headquartered in San Francisco, California, per its modern slavery statement.1389
Sources
This listing is not an endorsement, a safety assessment, or a federal approval.
Share or cite this page
- X (opens X)
- Bluesky (opens Bluesky)
- LinkedIn (opens LinkedIn)
- Reddit (opens Reddit)
- Hacker News (opens Hacker News)
- Email (opens your email app)
- RSS updates
Cite this page
APA style
USASI. (2026, September 29). τ-bench (tau2-bench). United States of America Superintelligence. https://unitedstatesofamericasuperintelligence.com/open/tau-bench/
BibTeX
@misc{usasi_tau_bench,
author = {{USASI}},
title = {τ-bench (tau2-bench)},
year = {2026},
month = sep,
howpublished = {\url{https://unitedstatesofamericasuperintelligence.com/open/tau-bench/}},
note = {United States of America Superintelligence. Last updated 2026-09-29}
}The date is when this page was last updated. Add the date you read it if your style needs one.
For maintainers: a “Listed in USASI” badge
Links to this page. It says the entry is listed, not endorsed or rated. About the badge
Markdown (for a README)
[](https://unitedstatesofamericasuperintelligence.com/open/tau-bench/)
HTML
<a href="https://unitedstatesofamericasuperintelligence.com/open/tau-bench/"><img src="https://unitedstatesofamericasuperintelligence.com/badges/listed.svg" alt="Listed in the USASI catalog" height="20"></a>