https://unitedstatesofamericasuperintelligence.com/hubs/evaluation/ Dated, sourced news items about the organizations and records featured in the USASI Benchmarks and evaluation hub. Independent project. Not a United States government website. en-us Tue, 13 Oct 2026 12:00:00 GMT https://unitedstatesofamericasuperintelligence.com/news/mlperf-training-adds-llm-post-training-benchmark/ https://unitedstatesofamericasuperintelligence.com/news/mlperf-training-adds-llm-post-training-benchmark/ Tue, 13 Oct 2026 12:00:00 GMT release MLCommons announced on September 24, 2026 that MLPerf Training is adding an LLM post-training benchmark, beginning with the v6.1 submission round in October 2026. The benchmark reports the wall-clock time a system needs to reach a quality target when post-training Qwen3.5-397B-A17B with reinforcement learning with verifiable rewards, using Group Relative Policy Optimization, on Python software engineering problems from the R2E-Gym dataset. Agents work in sandboxed Linux containers, and the specification and reference implementation are in the mlcommons/training repository. https://unitedstatesofamericasuperintelligence.com/news/cais-and-scale-ai-release-hle-diamond/ https://unitedstatesofamericasuperintelligence.com/news/cais-and-scale-ai-release-hle-diamond/ Tue, 13 Oct 2026 12:00:00 GMT release On September 22, 2026, the Center for AI Safety and Scale AI released HLE-Diamond, which the release notes describe as a refined subset of the Humanity's Last Exam (HLE) question collection, produced through a year-long process of cleaning and refinement with input from research communities. HLE-Diamond has 1,000 questions, split evenly into 500 reasoning and 500 knowledge questions, and is designed to be answered closed-book, with separate recommended settings for evaluations that use tools. The dataset is on Hugging Face as cais/hle-diamond, where the card lists an MIT license, requires agreeing to share contact information before access, and asks that the dataset not be publicly shared or re-uploaded. https://unitedstatesofamericasuperintelligence.com/news/mlcommons-publishes-mlperf-inference-v6-1/ https://unitedstatesofamericasuperintelligence.com/news/mlcommons-publishes-mlperf-inference-v6-1/ Thu, 01 Oct 2026 12:00:00 GMT release MLCommons published results for version 6.1 of the MLPerf Inference benchmark suite on September 16, 2026. The round added two tests: an end-to-end retrieval-augmented generation test that runs a question-answering pipeline of several separate models, and an Edge Agentic Inference test built around multi-turn workloads such as agentic coding. Speculative decoding is now allowed in the interactive scenario for two of the inference benchmarks. Results are published on MLCommons' datacenter and edge results pages. https://unitedstatesofamericasuperintelligence.com/news/sierra-open-sources-hyper-tau-bench/ https://unitedstatesofamericasuperintelligence.com/news/sierra-open-sources-hyper-tau-bench/ Tue, 13 Oct 2026 12:00:00 GMT research On September 8, 2026 Sierra published Hyper-τ-bench, a long-horizon evaluation that measures whether a model can build a customer-service agent rather than only act as one. A developer agent works in a sandboxed workspace with a simulated business's records and a simulated client it can message, must recover the requirements and turn the business's actions into tools, and the finished agent is then tested on held-out tasks the developer never saw. Sierra says it is open-sourcing the benchmark, with code in the sierra-research/hyper-tau-bench repository, a public leaderboard, and a paper on arXiv; the post does not state a license. USASI has not run the benchmark. https://unitedstatesofamericasuperintelligence.com/news/mlcommons-publishes-mlperf-storage-v3-0-results/ https://unitedstatesofamericasuperintelligence.com/news/mlcommons-publishes-mlperf-storage-v3-0-results/ Tue, 13 Oct 2026 12:00:00 GMT release MLCommons published results for version 3.0 of its MLPerf Storage benchmark suite on September 1, 2026. The suite measures the performance of storage systems for machine learning workloads, and v3.0 adds a KV Cache test of storage performance for LLM inference cache read and write operations and a Vector Database test covering vector indexing and querying. Version 3.0 also adds support for an S3 object storage access layer alongside the existing POSIX layer, supported for training, checkpointing, and some vector database tests. Results are on the MLCommons Storage benchmark results page. https://unitedstatesofamericasuperintelligence.com/news/terminal-bench-4-0-released/ https://unitedstatesofamericasuperintelligence.com/news/terminal-bench-4-0-released/ Tue, 29 Sep 2026 12:00:00 GMT release The Terminal-Bench team published version 4.0 of its command-line agent benchmark on August 28, 2026, a month after announcing that it would be maintained as a continuous benchmark with semantic versioning. The release removed eight tasks, fixed nineteen, and set a flat eight-hour agent timeout for every task; it is a major version because these changes require re-running trials. It runs through the Harbor framework. https://unitedstatesofamericasuperintelligence.com/news/mlcommons-releases-mlperf-client-v2-0/ https://unitedstatesofamericasuperintelligence.com/news/mlcommons-releases-mlperf-client-v2-0/ Tue, 13 Oct 2026 12:00:00 GMT release MLCommons released version 2.0 of MLPerf Client, its benchmark for running AI workloads locally on PCs, on August 18, 2026. The release adds an image generation category, with Flux.2 klein 4B as an experimental test, and an agentic AI category with Software Engineering Agent and Data Analyst Agent scenarios that break end-to-end results into LLM inference and tool execution time. Phi 4 Mini Instruct replaces Phi 3.5 mini instruct in the required LLM workloads, and Qwen 3 8B is added as an experimental test. MLCommons says v2.0 is free to download for Windows, macOS, and Linux from GitHub. USASI has not tested MLPerf Client. https://unitedstatesofamericasuperintelligence.com/news/mlcommons-releases-mlperf-endpoints-v0-7/ https://unitedstatesofamericasuperintelligence.com/news/mlcommons-releases-mlperf-endpoints-v0-7/ Tue, 13 Oct 2026 12:00:00 GMT release On July 28, 2026, MLCommons announced MLPerf Endpoints v0.7, a foundation release of a new MLPerf benchmark aimed at enterprise buyers procuring inference compute from neoclouds, cloud providers, and managed services. The release covers three benchmarks with initial results from CoreWeave, Google, Intel, KRAI, and NVIDIA, and supports automated submission pipelines, continuous review tooling, and dynamic result visualization. Results and tooling are at endpoints.mlcommons.org. MLCommons says it will deliver MLPerf Endpoints v1.0 later in 2026, with more buyer-centric rules and agentic workloads. https://unitedstatesofamericasuperintelligence.com/news/arc-prize-names-first-arc-agi-3-milestone-prize-winners/ https://unitedstatesofamericasuperintelligence.com/news/arc-prize-names-first-arc-agi-3-milestone-prize-winners/ Tue, 13 Oct 2026 12:00:00 GMT research On July 6, 2026, the ARC Prize Foundation announced the winners of the first ARC Prize 2026 milestone prize, which rewards mid-competition open-source solutions to the ARC-AGI-3 interactive reasoning benchmark; the first milestone period ran through June 30. The post names three winning submissions, including Tufa Labs' "The Duck," which plays the games by having a small open-source LLM write and run Python in a live REPL, and links each winner's Kaggle notebook. The second and final milestone prize was scheduled to end on September 30.