The MLPerf Reasoning Task Force came together to deliver an MLPerf Training benchmark that can complete a real-world LLM post-training workload in under 1024 GB300 hours. Real-world post-training combines techniques such as distillation, reinforcement learning, and supervised fine-tuning across domains including math, reasoning, and agentic tasks. After surveying the post-training landscape, we selected the Reinforcement Learning with Verifiable Rewards (RLVR) method because it is well suited to software tasks. An LLM-powered coding agent solves agentic software engineering (SWE) tasks in a sandbox environment; each attempt to complete a task is called a rollout. For each problem, the system samples multiple rollouts and assigns a binary pass/fail reward. The resulting traces are then used in Group Relative Policy Optimization (GRPO), which compares rewards within each group and updates the model to make higher-reward solutions more likely. The updated model weights are then used in the next iteration of rollouts.

Developing this benchmark required the most complex system-level integration in MLPerf Training to date. It requires deploying three different components: training, inference, and agent environments. At runtime, hundreds of concurrent rollouts contend for CPU time, GPU time, memory, and storage, and resources must be balanced between them. State-of-the-art training techniques are also particularly sensitive to numerics.

Model Selection

The model selected for this benchmark is Qwen 3.5 397B, the largest model in the Qwen 3.5 family released in February 2026. It is available as an open-weight model under the Apache 2.0 license. Qwen 3.5 is a mixture-of-experts (MoE) model with 397 billion total parameters and 17 billion active parameters per token. Qwen 3.5 was the first model family to productize a hybrid architecture consisting of Gated DeltaNet (GDN) and sparse MoE layers. Each MoE layer activates 1 shared expert and only 10 of 512 routed experts (2%). GDN is a time- and space-efficient alternative to standard attention that shows strong in-context learning abilities. In particular, GDN replaces the linear-growth KV cache with a fixed-size, compressed state. It conceptually learns past associations rather than explicitly storing them, and the fixed size is particularly valuable for long context. Both MoE and GDN reflect current trends in model architectures.

The MLPerf Training benchmarks measure time-to-train to a pre-determined accuracy metric as the score. This metric ensures the benchmark isn’t just a raw throughput measure; it measures an objective that mimics real-world training. More casually, the benchmarks require an answer that is both correct and fast. Benchmarks that measure only throughput can be misleading by encouraging performance optimizations that hurt final model quality and therefore don’t help real-world training. For example, reduced-precision numerical formats can improve throughput but may not provide enough numerical stability to achieve target accuracy. A more subtle example is increasing policy lag (in the asynchronous scenario), which may increase throughput for one system but degrade model quality. For this reason, AI benchmarks need to account for trained model quality in addition to throughput. The MLPerf benchmarks enable submitters to tune their systems for optimal performance while also keeping model quality in check.

Applying this philosophy to post-training is challenging. ll large parameter count open-weight models are already post-trained on most publicly available post-training datasets. As a result, developing a benchmark that can improve accuracy on a public dataset in a limited training time is quite difficult. The published weights for Qwen3.5-397B-A17B feature post-training, which makes the model capable of most agentic tasks. So this benchmark aims to slightly improve the model’s performance specifically on SWE tasks.

We thank the Qwen Team for providing this model to the community. Furthermore, we welcome and encourage other members of the community to publish open-weight models at different stages of training to support public benchmarking efforts.

Dataset Selection

The benchmark uses the Apache 2.0-licensed R2E-Gym dataset of SWE problems. This is one of the larger datasets of procedurally generated executable environments, created from GitHub commits of software projects in Python: Aiohttp, Datalad, NumPy, Pandas, Pyramid, Pillow, Sympy, and Tornado. The agent is given a Linux container environment with a Git repository, all dependencies pre-installed, and an issue description in the prompt. The task is to come up with a patch that would solve the problem.

While searching for a stable operating regime for the benchmark, we imposed restrictions on max number of agent turns and available max context length. To fit those constraints, we selected a subset of the R2E-Gym dataset based on proxies for difficulty such as the number of modified lines and files in the “golden” solution and the ability of the post-trained model to solve the tasks. The result is a dataset consisting of 700 training problems and a non-overlapping set of 251 validation problems. 

For post-training, the dataset includes prompts and container images, published on MLCommons storage in Apptainer SIF format. We faced interesting challenges when preparing the dataset: the original R2E-Gym container images were built for x86_64 in early 2025. Rebuilding them in 2026 for Aarch64 resulted in images that worked slightly differently, mostly due to dependency drift, updated packages, and architecture-dependent packages. Consequently, some of the problems could not be solved. We audited the dataset and fixed any such issues when possible or removed problems from the dataset to ensure that all of our environments work the same way across architectures and are solvable.

We would like to thank the R2E-Gym authors for providing this dataset to the community.

Environment

The model interacts with the sandbox environment through a harness. For a given context length and turn limit, the choice of harness determines how efficiently the model uses those budgets. We found that the choice of harness strongly influenced solve rate, which translates directly to training performance because RLVR uses sparse binary rewards. We selected the OpenHands harness for both training and validation based on its performance on the dataset. During RL training, models sometimes find ways to obtain a reward without solving the underlying task, for example by removing failing tests rather than fixing the bugs causing the test failure. This is called “reward hacking” and teaches the model solutions that score well in training but fail to generalize. To prevent reward hacking, the agent evaluation uses test files the agent can’t see or modify during execution.

To simulate extended multi-turn agentic work, the task force set a maximum context length of 65536, a maximum of 30 agent turns, and 16 generations per prompt. We chose these constraints to balance training performance with duration. A larger context requires more resources and enables stronger model performance, making it harder to achieve quality improvements. Conversely, a smaller context can hinder the model in solving tasks, which affects rewards and prevents GRPO from finding a training signal. The training objective is best characterized as optimizing the model’s ability to operate efficiently under the limited context/turn budget.

The model is configured in instruction-following mode with reasoning mode disabled, but the model has access to a “think” tool through the OpenHands harness. Multi-turn agentic rollouts with long context result in a workload that heavily exercises the KV cache management system of the generation endpoints. A balance must be struck between keeping the KV cache in GPU memory between turns and re-running prefill for previously processed context. Submitters will need to find that balance.

Implementation Details

The reference implementation of the benchmark is built on NVIDIA NeMo-RL and separates the pool of worker nodes into policy training and rollout generation workers. Agent sandboxes are executed on the same GPU nodes. Ray coordinates the distributed components:

• Megatron Bridge and Megatron Core train the policy.
• vLLM generates trajectories and receives refreshed policy weights after every training step.
• NeMo Gym runs the multi-turn software-engineering OpenHands agent in per-task Apptainer containers.

The reference utilizes asynchronous training, which is important in making RL performant. Under async RL, training and generation can overlap, resulting in stale (off-policy) model weights for generation. The maximum staleness, or age, of trajectories used for training is 1 weight version; submissions are free to use different values.

The disaggregated design of the reference highlights a central challenge in asynchronous reinforcement learning: to achieve optimal performance, training and inference must be rate-matched and optimized together. Optimizing only one part of this pipeline simply moves the bottleneck elsewhere.

Evaluation Approach

When evaluating model performance, we found high variability in the solve rate even with deterministic sampling (temperature=0.0). The model could solve a task on one attempt and fail on the next. Multi-turn agentic trajectories are never deterministically reproducible due to tool calling, timing, and batching effects. One approach to achieving a more stable evaluation score is pass@k.

Under pass@k, each problem is evaluated k times, and we measure how many problems are solved at least once out of k times. For this benchmark, we settled on the pass@4 metric as the validation metric, which makes validation consist of 1004 rollouts. The benchmark objective identified in the course of the reference convergence point study is to reach a pass@4 score of 0.69 – to solve 69% of validation problems at least once in 4 tries. Token sampling for validation uses temperature=0.1 and top_p=0.95. The chart below shows the dispersion of the metrics we evaluated: pass@4, pass@1-average-of-4, and pass^4 (all 4 attempts must pass). Pass@4 shows the lowest coefficient of variance.

Managing Variance and Evaluation Cost

Agentic reinforcement learning is significantly less deterministic than pre-training. The training data is not static, and a built-in feedback loop updates weights used for future rollouts, which in turn affect future training steps. At the trajectory level, small numerical differences change sampled tokens; those differences alter tool calls and intermediate outputs. These effects compound over many turns. Other sources of non-determinism include request routing, different training/generation backends, numerical sensitivity in MoE routing, non-determinism in executed kernels, and off-policy training. This variability is expected in the learning process built around exploration, but it complicates a benchmark that needs repeatable quality targets.

Evaluation is also expensive. Unlike a forward-only validation pass, each evaluation launches complete software-engineering agents that generate tokens, use tools, and execute tests. In the reference setup, one validation requires 1004 trajectories and can cost several training steps.

The benchmark addresses these constraints in three ways. First, grouped pass@4 makes the quality metric more stable. We also fixed the traversal order of the training dataset according to a difficulty-aware ordering to reduce variability. Finally, the benchmark rules delay the first full evaluation until the model is expected to be near the target. The reference qualified global batch sizes of 256, 512, and 1024 begin validation at 4608, 5120, and 7168 training samples, respectively, then evaluate after each later step. These evaluation starting points follow an empirical scaling law determined during the study of the reference’s convergence behavior: GBS * ceil(2.5 + 3840 / GBS). This schedule is meant to ensure that all runs converge to their target at the first evaluation.

Benchmark Parameters

This MLPerf Training benchmark reports the wall-clock time required to reach the fixed pass@4 quality target. The reference convergence point study includes global batch sizes of 256, 512, and 1024, with batch-dependent learning rates, gradient norm clipping, and first-validation steps. The study was performed on GB300 NVL72 racks, using 64 compute nodes with 4 Blackwell Ultra GPUs each (256 GPUs total) and BF16 for both training and generation, and required between 140 and 160 minutes per run, a baseline submitters are expected to improve on. These settings define the qualified reference points; submitters may select other global batch sizes and remain responsible for following the MLPerf Training rules that govern legal configurations, optimizations, and alternative implementations.

What This Benchmark Adds

The LLM post-training benchmark brings several previously separate systems concerns into one measured workload: very large mixture-of-experts policy training, high-throughput inference, frequent policy-weight transfers, asynchronous trajectory collection, and long multi-turn trajectories with heavy KV cache load.

That combination creates new optimization opportunities and challenges. Submitters can improve computation, communication, scheduling, rollout throughput, environment execution, and the balance between training and generation resources, but must maintain numerical stability and avoid training collapse. Because the benchmark measures the complete time-to-quality path, improvements must accelerate the system without sacrificing the agent’s ability to solve unseen tasks.

Conclusion

MLPerf Training’s LLM post-training benchmark reflects a major shift in how leading large language models gain useful capabilities. The workload does not stop at predicting the next token or adapting on a static dataset, but requires the model to learn how to take the right actions on real software-engineering tasks.

By establishing a common model, task, dataset, quality target, and evaluation procedure, MLCommons gives the industry a practical way to measure progress in agentic reinforcement-learning systems. The benchmark will help expose where time and resources are spent—and encourage advances across the full post-training pipeline.

The benchmark specification and reference implementation are available on GitHub.

We would like to thank the entire MLPerf Training Reasoning Task Force for coming together and shaping this benchmark.

About MLCommons

MLCommons is the world’s leader in AI benchmarking. An open engineering consortium supported by over 125 members and affiliates, MLCommons has a proven record of bringing together academia, industry, and civil society to measure and improve AI. MLCommons began with the MLPerf benchmarks in 2018, which quickly grew into a set of industry metrics for measuring machine learning performance and promoting transparency in machine learning techniques. Since then, MLCommons has continued to use collective engineering to build the benchmarks and metrics required for better AI – ultimately helping to evaluate and improve the accuracy, safety, speed, and efficiency of AI technologies.

For additional information on MLCommons and details on becoming a member, please visit MLCommons.org or email participation@mlcommons.org.

The post MLPerf Training Introduces Its First LLM Post-Training Benchmark appeared first on MLCommons.

AIRIS unites 21 partners from Europe, Canada and the US to develop generative AI models that integrate biological knowledge and clinical data. AIRIS’ main goal is to deliver an AI collaborator that helps researchers better understand disease progression and advance personalized medicine. Over the next four years, the consortium will receive €16.9 million in funding from the Horizon Europe Program to develop a generative AI platform that builds and reasons with mechanistic models of disease, rather than relying solely on statistical patterns. The goal is to design a platform that helps researchers identify previously unknown disease pathways and develop novel scientific hypotheses.

As a global leader in AI benchmarking, MLCommons will play a central role in ensuring that AIRIS is rigorously and independently evaluated. We will lead the development of a comprehensive evaluation framework and integrated benchmark suite that measures the platform across accuracy, robustness, fairness, interpretability, and usability, with dedicated benchmarks for each of the five disease areas and for detecting bias across patient subgroups such as sex, ethnicity, and age. 

“Rigorous, independent evaluation is what turns a promising AI system into one that researchers can actually trust,” says Alexandros Karargyris, Lead for the Medical working group at MLCommons. “By building open benchmarks that probe not just accuracy but fairness, robustness, and interpretability across real disease settings, we want to give the scientific community a transparent yardstick for mechanism-informed biomedical AI — and set a standard that reaches the whole community.” 

Building on this framework, MLCommons will coordinate AIRIS’ iterative evaluation rounds, track progress across successive versions of the platform, and open these benchmarks to external researchers so the wider scientific community can test AIRIS on their own data, establishing AIRIS as a reference benchmark for multimodal, mechanism-informed biomedical generative AI.

In the years ahead, AIRIS seeks to transform how scientists investigate complex diseases by providing a trustworthy AI collaborator that supports every stage of the research process – moving past black-box predictions and grounding its reasoning in biological knowledge.

To learn more about the AIRIS project and explore how MLCommons is advancing open benchmarks for biomedical AI, visit the AIRIS website and join the MLCommons Medical Working Group to help shape in the future of trustworthy, mechanism-informed healthcare AI.

The post MLCommons Joins EU-Funded AIRIS Project to Build and Benchmark Next-Generation Biomedical AI appeared first on MLCommons.

The highlights of this round include a record number of submitters, new accelerators that significantly improve per-device performance, and a new multi-turn benchmark. Read on for our results analysis.

Submitters and Submissions

This round brought a record 30 submitters spanning silicon vendors, system builders (OEMs and ODMs), cloud and neocloud providers, and specialized inference-software companies.
Several results this round were joint submissions, where two organizations collaborated: Dell_AMD, Dell_MangoBoost, RedHat_Intel, and RedHat_Supermicro.

The table below provides a breakdown of the systems and scores submitted by each organization:

Altogether, 120 systems across the Datacenter and Edge suites, in both the Closed and Open divisions, were submitted.

Benchmarks

MLPerf Inference benchmarks are divided into Datacenter and Edge categories. Each benchmark supports several scenarios, which may be one or more of: Offline, Server, Interactive, SingleStream, or MultiStream. Submissions can also be classified as either Closed or Open. Closed submissions require using a model that is mathematically equivalent to the reference implementation, essentially holding the model fixed to enable direct comparisons. For more details on the MLPerf Inference benchmark setup, visit https://mlcommons.org/benchmarks/inference-datacenter/.

MLPerf Inference v6.1 features 10 Datacenter and 6 Edge benchmarks, two of which are new: End-to-End RAG and Agentic Edge Inference. Two benchmarks have been updated: For VLM (Visual Language Model based on Qwen3), an Interactive scenario has been newly defined, and in GPT-OSS-120B, the interactive scenario has been updated to allow speculative decoding.

MLPerf Inference submitters can choose to submit to any of the benchmarks. In the Datacenter category, the most popular benchmark is gpt-oss-120b, followed by Llama2-70b. This is the first time gpt-oss-120b has topped the list, after Llama2-70b was the most popular for the last few rounds. This demonstrates that the MLPerf benchmarking community (which highly values stability and comparability) is now fully embracing MoE models.

Performance improvement this round

To gauge performance improvements since the last round, we compared the best per-accelerator Offline and Server scores against those from v6.0.

The biggest gains are in VLM and DeepSeek R1 for both Offline and Server scenarios, significantly surpassing gains in other benchmarks. This is because those 2 benchmarks were submitted on a new Preview category system powered by Nvidia Vera Rubin. Other benchmarks’ top scores were achieved on the same hardware as in 6.0, so results reflect more gradual performance improvements from software stack and algorithmic optimizations.

Llama2-70b is the longest-running LLM in the MLPerf Inference suite, being introduced in early 2024, round v4.0. As such, it is a strong vehicle for showing LLM performance improvements over time. This is shown below in median per-accelerator performance for Server scenario submissions, which has improved 5.58x over 6 runs. Three main reasons contribute to the performance gains: 

1. Reduced numerical precision: Some v6.1 submissions use FP4 precision, whereas earlier rounds used FP8. Since MLPerf Inference has strong accuracy requirements, the performance gains did not come at the cost of accuracy loss.
2. New generation of accelerators. Accelerators that were used in the initial rounds of Llama2-70b benchmarks have been replaced with newer, more performant ones in the latest rounds.
3. Software performance improvements. Every submission round sees performance improvements even on the same hardware, a testament to the improvements our submitters bring and a reflection of continued improvement in the field.

A similar trend appears in the DeepSeek R1 benchmark, despite being around for less time. The figure below shows that best per-accelerator performance has increased by 2.7x in Offline and 5.7x in Server scenarios, respectively, within a year. Similarly, the Interactive scenario, which had only 2 submission rounds, increased by 2.7x over that period.

Results Highlights

Beyond performance improvements, results also contain additional important achievements. One continued trend is multi-node inference. In the last few rounds, the number of multi-node submissions has increased considerably, as shown in the figure below:

The upward trend started in round 4.1, 2 years ago, and this round reached an all-time record of 16 multi-node submissions. We also see the size of submitted systems increasing. Just last round, a record of 288 accelerators was set, only to be broken in this round by Cruose submissions 6.1-0026 and 6.1-0027, which used 512 accelerators. The latter of the submissions set a new record for the number of tokens/second generated in the MLPerf Inference benchmark by generating almost 5.8M tokens/second in GPT-OSS-120b offline test.

Another trend in MLPerf Inference is hybrid submissions that use different types of accelerators working together. These systems pose unique challenges because submitters must account for the different computational capabilities of the devices. This round contains two such submissions:

New Accelerators and System

v6.1 introduced a diverse set of newly submitted accelerators and systems — from on-device parts to a next-generation rack-scale platform — spanning multiple accelerator vendors and different scales of deployment.

Submitters in their own words

Many participating organizations submitted supplemental statements describing what they consider most significant about their v6.1 submission. Full statements are available in the results repository; here are the highlights.

Summary

As always, these results are most valuable when read in context: the Closed division remains the foundation for apples-to-apples comparison, while the Open division surfaces the optimization techniques that often preview the mainstream. We encourage readers to consult both the results tables and the submitter statements.

A round like this reflects an enormous collective effort. Our thanks go to every submitting organization — established vendors and first-time participants alike — and to the many Inference Working Group members who build, review, and audit these benchmarks and keep MLPerf rigorous, relevant, and fair.

The post Where the Industry Is Investing: A Look at MLPerf Inference v6.1 appeared first on MLCommons.

The open-source MLPerf Inference benchmark suite measures system performance in an architecture-neutral, representative, and reproducible manner. It creates a level playing field for competition, driving innovation, performance, and energy efficiency across the entire industry. The published results provide critical technical information, allowing customers who are procuring and deploying AI systems to make informed decisions based on trusted empirical data.

Evolution of the benchmark: new tests, new support for popular optimizations 

MLPerf Inference v6.1 introduces two new tests to the suite that reflect the industry’s evolution towards more complex, multi-step, and agentic deployments of AI inference technology, both in the datacenter and at the edge.

The End-to-End Retrieval-Augmented Generation (RAG) benchmark tests a complex pipeline of multiple distinct AI models for question answering. This workload decomposes the task into a sequence of steps: an embedding model turns a query into a vector, a retriever pulls candidate passages from a vector database, a re-ranker refines the list, and one or more LLMs reason over the data to produce an answer. The test generates performance data for two separate but related tasks: ingesting a document corpus to create a vector database, and query-answering against a pre-built vector database. More information on the End-to-End RAG benchmark is available here.

The Edge Agentic Inference benchmark reflects the shift from single-shot interactions toward complex multi-turn workloads such as agentic coding. Unlike single-shot inference, agentic workloads feature growing conversational history in which each query depends on those that came before, and the model merges evidence gathering with subsequent reasoning over that evidence, iterating as necessary to arrive at a final result. This is a much more powerful usage model and unlocks tremendous new capabilities that single-shot inference lacks, while radically increasing demands on AI compute. Agentic AI deployment is increasingly popular at the edge, where serving one user at a time with fixed memory, processing, power, and context creates unique performance challenges. This test adopts the framework and methodology from the soon-to-be-released MLPerf Agentic datacenter performance benchmark and specializes it with an edge model and quantization, a single-stream coding workload, latency metrics, and a statistically robust accuracy gate. It incorporates two tasks: measuring workload accuracy under time constraints and measuring the performance of a deterministic workload in which accuracy checks are intrinsic to the workload. More information on the Edge Agentic Inference benchmark is available here.

MLPerf Inference 6.1 also adds support for a performance optimization now in common use in production inference systems. Speculative Decoding, a popular optimization technique that predicts and verifies multiple tokens in a single forward pass, is now supported in the interactive scenario for two of the inference benchmarks, with added support in the GPT-OSS task. 

“We are working hard to ensure that the MLPerf Inference benchmark continues to reflect the scenarios that the AI community values most,” said Miro Hodak, MLPerf Inference working group co-chair. “We added the End-to-end RAG test because it’s clear that query-answering has evolved beyond simply an LLM trained on a corpus; stakeholders need to understand the real-world performance of the types of multi-step, multi-component pipelines that are being built today. Likewise, we added the Edge Agentic Inference test because complex inference systems with agentic properties are increasingly hosted on edge computing devices, creating a new set of performance challenges our customers face today. We are committed to providing timely and relevant performance data that reflects real-world production systems and the performance optimizations that are being deployed today.”

Submissions and Results Demonstrate Speed of AI Innovation

This round of the benchmark highlights the rapid evolution of AI technology and deployments, with submitted systems representing a range of hardware options across accelerators, boards, workstations, and servers. The results include five new processors or accelerators: AMD Ryzen AI Max+ 395, AMD Instinct MI350P, and Intel Arc Pro B70 are all available, and NVIDIA Rubin and NVIDIA Vera Rubin NVL72 is in preview.  They also include the largest system ever submitted to the MLPerf Inference benchmark, with 512 accelerators and two novel heterogeneous systems. The first heterogeneous system combines high-performance networking with accelerators from two different vendors, and the second system is geographically distributed – spanning the Pacific Ocean. 

Performance gains continue to compound and shatter records for AI accelerator capabilities. For the Visual Language Model (VLM) test, the best per-accelerator server scenario result for this benchmark round improved by 2.99x from v6.0 just six months ago. For the Deepseek R1 test, the best per-accelerator server scenario result was 5.7x better compared to v5.1 one year ago. This performance gain translates into more capabilities for end customers, potentially serving more users or enabling more sophisticated inference use cases.

“With the critical data from the MLPerf Inference benchmark, the AI community is once again proving that what can be measured can be improved,” said Frank Han, MLPerf Inference working group co-chair. “As workloads and the underlying systems increase in capability, scale, and complexity across a broader variety of form factors and physical configurations, we are clearly seeing a renewed interest in ensuring that inference can take full advantage of hardware, software, and architectural advances – both in the datacenter and at the edge. With performance data from the Inference v6.1 benchmark, customers can better understand the cost-benefit tradeoffs and make informed decisions on how to procure and deploy their AI systems.”

The AI community continues to embrace and invest in MLPerf Inference

The MLPerf Inference 6.1 benchmark received submissions from a record-high 30 participating organizations: AMD, ASUSTeK, Atlas Inference, Cisco, CoreWeave, Crusoe, Dell, Fujitsu Limited, GigaComputing, Google, Hewlett Packard Enterprise, Intel, Inventec Corporation, KRAI, Lambda, MangoBoost, Microsoft Azure, MiTAC, Nebius, NVIDIA, Oracle, Orrick, Quanta Cloud Technology, RedHat, ScitiX, Supermicro, Telecommunications Technology Association, VibeHPC, Wiwynn and individual contributor Naeem Khoshnevis. 

“I would like to welcome our six first-time submitters: Atlas Inference, Crusoe, Orrick Industries LLC, ScitiX, VibeHPC, and individual contributor Naeem Khoshnevis, ” said David Kanter, Head of MLPerf. “The groundswell of participation from the AI community tells us that our work is important and is making a difference to stakeholders.”

In addition, over 50% of submitters used MLPerf’s new API-centric harness, the foundation of the next-generation MLPerf Endpoints benchmark suite, in their submissions. The new harness uses a true client/server architecture over industry-standard APIs to communicate inference queries and results to the system under test, more accurately reflecting real-world datacenter deployment. “We’re excited to see submitters embrace our customer-centric tooling and workloads and partner with us to improve the state of industry-wide benchmarking for AI,” said Kanter. “Moving forward, MLPerf Endpoints will replace Inference in our family of benchmarks for the datacenter, and the quick uptake of our API-centric harness will contribute to making that transition seamless.”

View the results

To view the results for MLPerf Inference v6.1, please visit the Datacenter and Edge benchmark results pages and read details on individual submissions in our supplemental. For an interactive view of datacenter results, access the MLPerf benchmark results dashboard: https://mlcommons.org/visualizer. Additionally, submitters have provided supplemental statements to help contextualize and understand their submissions and results.

About MLCommons

MLCommons is the world’s leader in AI benchmarking. An open engineering consortium supported by over 130 members and affiliates, MLCommons has a proven record of bringing together academia, industry, and civil society to measure and improve AI. The foundation for MLCommons began with the MLPerf benchmarks in 2018, which rapidly grew into a set of industry metrics for measuring machine learning performance and promoting transparency in machine learning techniques. Since then, MLCommons has continued to use collective enMLCommons Sets Participation Record with New MLPerf Inference v6.1 Benchmark Results

For additional information on MLCommons and details on becoming a member, please visit MLCommons.org or email participation@mlcommons.org.

Press Inquiries: contact press@mlcommons.org

The post MLCommons Sets Participation Record with New MLPerf Inference v6.1 Benchmark Results appeared first on MLCommons.

Why this matters

AI agents don’t just answer questions — they take actions. That makes them powerful, but it also means institutions and supervisors need a credible way to know what an agent is actually permitted to do, and whether it has stayed within that scope. Today’s agent-identity standards can prove who an agent is, but not what it should be authorized to do once it connects to financial digital infrastructure — a gap we call the authorization and supervision gap.  

The Agent Reliability Profile, the working group’s flagship initiative, aims to close that gap by creating a standardized, regulator-compatible framework for describing, validating, and benchmarking the reliability of agentic deployments in financial services.

The C:>DIR hackathon

C:>DIR is run by the University of Cambridge’s Digital Innovation and Regulation Initiative, backed by the BIS Innovation Hub, the Global Financial Innovation Network (GFIN), and the Digital Regulation Cooperation Forum, among more than 35 supporting organizations. It’s a global competition for regulators and industry to build practical, deployable prototypes that strengthen trust and accountability in agentic AI.

The scale of the competition underscores the interest in solving this problem: they received 336 submissions from over 65 countries, with 36 teams — six per problem space — advancing to the final round.

Our submission

Entered by working group co-chairs Mike Hsu and Medha Bankhwal, our submission competes in the Know Your Agent (KY-A), Digital Verification & Digital Public Infrastructure problem space. It demonstrates two tools built on the Agent Reliability Profile:

• Profile Builder reads an institution’s own evidence—design docs, configuration exports, policies—and compiles it into a Level 1 Asserted Profile, surfacing where stated intent and as-built configuration diverge.
• Profile Validator, which takes that Profile as a test specification and tries to falsify it against real system behavior, producing a Level 2 Validated Profile.

The demo puts both tools to work on an open-banking data-sharing and consent scenario, testing whether an agent stays within the data-sharing scope a customer actually consented to — a concrete, high-stakes test of the authorization and supervision gap in practice.

What’s next

Final-round builds run from September 1–8. Live demos and judging by a panel of regulatory fellows from the Cambridge Regulator Visiting Fellowship Program will run September 15–16. Winners will be announced September 18, 2026, at the C:>DIR Summit.

Join us

The Agent Reliability Profile is a working group effort, and we’re looking for financial institutions, regulators, and technical contributors to help shape it. The Financial Services Working Group meets Wednesdays, 12:00–1:00 PM ET — reach out to Participation@mlcommons.org to get involved.

The post MLCommons Agent Reliability Profile Named a Finalist in Global Agentic Regulator Hackathon appeared first on MLCommons.

Two new benchmark tests

Version 3.0 adds a new KV Cache test, which measures storage performance for LLM inference cache read/write operations. KV caching is a widely adopted technique to increase performance in transformer-based AI inference applications, particularly autoregressive ones such as LLMs that iteratively access the same key-value vectors.

It also adds a new Vector Database (VDB) test that measures storage performance for vector indexing and querying workloads. Vector databases are used to store high-dimensional data that exceed the limits of traditional database architectures, and they are frequently used by AI applications to store unstructured data such as text and media.

“These new additions to the benchmark suite round out the test collection, covering a larger range of AI inference workloads that drive storage needs,” said Brian Belgodere, MLPerf Storage working group co-chair. “On the training side, we have training and checkpointing tests, and on the inference side we now have KV cache and vector database tests. Including tests that decompose monolithic AI systems and focus on specific storage uses and patterns, such as checkpointing, KV caching and vector databases, gives stakeholders a much clearer idea of how to engineer and provision AI systems to minimize storage performance bottlenecks.”

New S3 object storage access layer

In addition, Version 3.0 adds new support for an S3 object storage access layer, alongside and as an alternative to the existing POSIX-compliant layer. Adding this capability in v3.0 enables performance comparisons across a wider range of hosted storage options for the same workload. The S3 layer is supported for training, checkpointing, and some VDB tests in the version 3.0 benchmark suite. Approximately one-sixth of the total submissions in this round utilized the S3 storage access layer.

“Further broadening our support for a diverse set of storage systems, including S3, in the MLPerf Storage benchmark suite gives organizations that are provisioning AI systems even greater ability to select and combine technologies to meet the specific technical requirements of their application,” said Curtis Anderson, MLPerf Storage working group co-chair. “As the scale of AI contexts reaches into the trillions, we expect object-based storage systems to emerge as a viable – and possibly preferred – alternative to filesystem-based storage. By enabling S3 support now, we are ensuring that stakeholders will have the performance information they need to make smart decisions.”

11 new organizations submitting to Version 3.0

Nineteen organizations submitted to this round of the benchmark. “We would particularly like to welcome the eleven first-time submitters: Azure, Everpure, HolmesAI, Nebius, NewFW, NVIDIA, OpenLake, Suzhou Zishan Longlin, TuringData, XSKY, and ZettaLane,” said David Kanter, Head of MLPerf. “The MLPerf Storage benchmark has been embraced widely by the storage community, with submitters representing both cloud-based and on-premises solution providers, as well as organizations developing storage systems and devices. That broad level of engagement shows that the MLPerf Storage benchmark is providing critical information for stakeholders involved in all aspects of provisioning AI systems.”

The performance results provide unique and important insights into the state of the storage industry – including power efficiency. On-premises submissions for the checkpointing write test achieved a median rate of 14 GB/second per watt, with a maximum of 201. Similarly, submissions for the UNet3D read test achieved a median of 34 GB/second per watt, with a maximum of 277. “There is a wide range of power efficiencies represented in the results,” said Anderson, “which is critical performance information for stakeholders buying storage solutions for AI applications. It also shows that there is ample room for further improvement, and we encourage all suppliers to optimize for that metric.”

The MLPerf Storage benchmark was created through a collaborative engineering process over five years by 35 leading storage solution providers and academic research groups. The open-source and peer-reviewed benchmark suite offers a level playing field for competition, driving innovation, performance, and energy efficiency across the industry. It also provides critical technical information for customers who are procuring and tuning AI training and inference systems.

We invite stakeholders to join the MLPerf Storage working group and help us continue to evolve the benchmark suite.

View the Results

To view the results for MLPerf Storage v3.0, please visit the Storage benchmark results. For more information on the results, please visit the Supplemental Discussion.

About MLCommons

MLCommons is the world’s leader in AI benchmarking. An open engineering consortium supported by over 125 members and affiliates, MLCommons has a proven record of bringing together academia, industry, and civil society to measure and improve AI. The foundation for MLCommons began with the MLPerf benchmarks in 2018, which rapidly scaled into a set of industry metrics for measuring machine learning performance and promoting transparency in machine learning techniques. Since then, MLCommons has continued to use collective engineering to build the benchmarks and metrics required for better AI – ultimately helping to evaluate and improve the accuracy, safety, speed, and efficiency of AI technologies.

For additional information on MLCommons and details on becoming a member, please visit MLCommons.org or email participation@mlcommons.org.

The post MLCommons Releases New MLPerf Storage v3.0 Benchmark Results appeared first on MLCommons.

Deployers, like banks, need to keep sensitive data safe. AI solution providers, like frontier labs, are very protective of model weights. And most critically from our perspective, benchmark providers must have both technical and legal mitigations in place to ensure the models being tested won’t compromise the integrity of their benchmarks by learning from them. Importantly, secrecy of the evaluation itself is not sufficient. A robust benchmark stewardship program is also a necessary component of operating a high-integrity benchmark over time.  

Demonstrating best-in-class benchmark integrity is why we participated in this first-ever, double-blind evaluation proof of concept with Google DeepMind, OpenMined, and AVERI. This approach keeps every party protected by design, rather than relying solely on legal contracts, as part of the first-ever double-blind evaluation of a closed-weight model.  Through this method, we have cryptographic guarantees that our evaluation components are able to be used without contaminating the benchmark or compromising its integrity. 

Our approach to benchmark integrity is what made AILuminate the obvious choice for this proof of concept. We provided a reserved subset of the AILuminate safety benchmark prompts to be used in running this evaluation, ensuring that no GDM model had ever been exposed to this specific evaluation set before. AVERI used OpenMined’s secure computation to run these prompts on a containerized instance of a Google DeepMind model, with cryptographically provable protections for both the model’s weights and our evaluation. 

In this proof of concept, AILuminate characterized an AI system’s reliability without exposing our test data to the developer, and without exposing the developer’s proprietary model weights to the instrument maker (MLCommons) or the auditor (AVERI).

We are eager to see this type of IP-safe, saturation-resistant testing become the standard for applied tests of AI models. In support of it, we’ve already done this sort of work on the most sensitive of data: health informatics.  Through our MedPerf program, we’ve been building Trusted Execution Environments (TEEs) just like these, which enable multiple parties to bring encrypted data together in one place, run high-integrity evaluations, and get results, all without seeing any other party’s sensitive information.

We’re working toward a near future where AI adopters can quickly and easily run trustworthy evaluations of AI systems for their purpose-specific deployments. An environment that structurally guarantees the confidentiality of model, data, and test paired with the assurance of never-before-seen prompts from an independent organization, provides the clearest path toward informed decision making about AI adoption, deployment, and reliable operation.

MLCommons is working toward a near future where AI adopters can quickly and easily run trustworthy evaluations for their purpose-specific deployments. We invite practitioners, researchers, and deployers to review the technical details and explore opportunities to engage with the AILuminate framework and the Benchmark Stewardship Program at https://mlcommons.org/get-involved/.

The post The key to trustworthy AI evaluation is secrecy by design appeared first on MLCommons.

A Retrieval-Augmented Generation (RAG) pipeline ingests and chunks source data, converts it into semantic vector embeddings for storage and retrieval, and then combines relevant retrieved data with a user query to generate an accurate, model-driven response. This benchmark measures this entire pipeline. It is built around two workloads: 

• An ingestion pipeline that builds the vector database from a document corpus. 
• A question-and-answering (QnA) pipeline answers queries over the vector database, iterating retrieve-and-reason steps across multiple hops until it has enough evidence. 

Together they capture what governs a real RAG deployment — multiple models of different roles and sizes, served across an iterative loop — none of which a single-model benchmark can measure.

It is the first multi-component MLPerf inference benchmark to score an entire RAG pipeline end to end. This post covers what makes this workload unique, the dataset and task, the models and metrics, and the reference implementation. 

Why End-to-End RAG?

RAG is increasingly popular and  for many deployments, essential. By grounding answers in documents retrieved at query time, it reduces hallucination, keeps answers up to date, and lets a general model use proprietary or private knowledge it was never trained on — exactly the properties enterprises need.

By nature, RAG is a multi-component system, and its behavior emerges from how those components are composed and served together, not from any one of them alone. Single-model LLM benchmarks can’t capture this: they score one model on one prompt (most often without tokenzier/de-tokenizer involved), so the pipeline-specific behavior that governs a real RAG deployment goes unmeasured, and the optimization opportunities of serving several models together never arise.

It is also a stepping stone toward benchmarking agentic AI. RAG in itself is one of the tools an agent reaches for, and its multi-component, multi-model serving challenges are a first taste of what a full agentic benchmark must measure — laying groundwork that future agentic benchmarks can build on.

And it opens optimization avenues we haven’t seen in prior LLM benchmarks. Because the pipeline runs several models with different roles and sizes — rather than one model on one prompt — a submitter has levers that a single-model benchmark simply doesn’t expose. A few of them:

– Model placement. How the components are mapped onto the system — e.g. the smaller and larger language models on separate accelerators, or the lightweight components on CPU with varying precision levels without compromising E2E Accuracy criteria.

– Co-residency. How to share one device among multiple models — via memory partitioning or hardware-level isolation.

– Scheduling a multi-stage fleet of calls. How to overlap many concurrent tasks as an effective pipeline across heterogeneous accelerators involving macro and micro batching across the pipeline and within each ingredient in the pipeline.

– Prefix caching. The multi-hop context grows each hop, so reusing the shared prefix trades compute-bound prefills for memory-bound. 

– System level optimization/KPI: Effective pipeline implementation across CPU/GPU/NIC/Storage to demonstrate system level KPI mimicking real world deployment scenarios                                                                                                       

Dataset and task selection

The task is multi-hop queries answering over a set of Wikipedia documents, using the FRAMES benchmark:

• 824 queries, each with a ground-truth answer and the Wikipedia article URLs required to answer it.
• 2,515 Wikipedia HTML articles, a frozen snapshot shipped with the benchmark so every submitter indexes the same corpus.
• ~107,000 passages — the articles chunked into 768-character passages with 32 characters overlap for embedding and indexing.

FRAMES was chosen because it is public and factual, with unambiguous ground truth, and its queries span diverse reasoning types — numerical, tabular, temporal, multiple-constraint, and post-processing.

Above all, its queries are multi-hop, which is what makes FRAMES a demanding test of a RAG system. A multi-hop queries can’t be answered from one passage — it requires chaining several facts that never appear together, so every stage has to pull its weight. For example (Figure 1),

> “The person who posted a photo with Rahul Ligma and Daniel Johnson at the headquarters of a social media company claims to have a certain syndrome, despite never receiving a formal diagnosis. Who was this syndrome named after?” → ground truth: Hans Asperger

No single passage holds the answer. The pipeline has to chain three facts across separate articles: who posted that photo (Elon Musk), the syndrome he says he has (Asperger’s), and whom it is named after (Hans Asperger) — reformulating its searches hop by hop until the evidence connects.

Figure 1: Example Multi-hop E2E RAG

The benchmark exercises this dataset through two independent pipelines, each its own MLPerf workload:

Ingestion (`e2e-rag-db`) runs once to build the database; QnA (`e2e-rag-qna`) is the scored end-to-end pipeline. 

Figure 2: Ingestion and QnA Pipeline for E2E RAG

The ingestion pipeline turns the corpus into a searchable vector database, which is a one-time operation. Parsing extracts the article body from each of the 2,515 Wikipedia HTML files shipped with the benchmark, stripping Wikipedia metadata such as references and navigation. Tables and lists are flattened into text row by row rather than dropped. Chunking slices that text into 768-character passages, each tagged with its original Wikipedia URL so it can be traced back to its page. The size was tuned: larger chunks mix in unrelated text, smaller ones split related facts apart. A 32-character overlap keeps facts that land on a boundary intact. Embedding encodes each passage into a 768-dimension vector, and the vectors are indexed into a FAISS HNSW graph for fast approximate similarity search. The result is the vector database that every QnA run queries against.

The QnA pipeline answers a query by looping over that database generated in the Ingestion pipeline (E2E-RAG-DB). Answering a query is not a single model call but a QnA task: the query runs several components through an iterative loop with sub-queries to reach an answer. A query rewriter first decomposes the query into up to three focused sub-queries. Each sub-query is embedded with the same embedder from ingestion, and used to retrieve its most similar passages from the database. The reranker then deduplicates these candidates and reorders them by relevance, keeping the strongest few. A document grader judges each retrieved passage for relevance and keeps only those that help, and a sufficiency checker decides whether the accumulated evidence can answer the query. If not, the query rewriter generates new sub-queries, reformulating and broadening its searches, and the loop repeats for up to five hops. Once the evidence is sufficient (or the hop limit is reached), answer generation produces the final answer grounded in the kept passages, or returns “Unknown” if the evidence falls short.

Model selection

All models can be downloaded from MLCommons-Storage

• intfloat/e5-base-v2 is the embedder. Trained for retrieval with separate query and passage encodings, it fits RAG’s query-to-document matching and performs strongly on standard benchmarks (MTEB, BEIR), all at a compact, widely adopted 110M parameters.
• ColBERTv2.0 is the reranker, using late interaction rather than the typical cross-encoder. Its token-level matching captures finer-grained relevance than compressing a passage into one vector, and generalizes better to new domains, which suits the private document collections common in enterprise use.
• GPT-OSS-120B handles the reasoning-heavy work of rewriting a query into good sub-queries, judging when the evidence is sufficient, and generating the final answer. It is open, and already widely deployed and well-optimized by MLPerf submitters.
• GPT-OSS-20B covers document grading, deciding whether a retrieved passage is relevant. This is a high-volume classification task, and pairing a smaller model here reflects real deployments that route easy work to right-sized models rather than sending everything to the largest one. It also shares the gpt-oss architecture with the 120B, so supporting it takes little extra effort.
• Llama-3.1-8B-Instruct is the judge, deliberately from a different family than the GPT-OSS models it grades to avoid self-preference bias. It scores answers after the accuracy run is finished.

Performance metrics

MLPerf Inference traditionally defines two serving scenarios, Offline and Server. This first instantiation focuses on Offline, where all requests are available at once and can be scheduled for maximum throughput; Server is left for future rounds. Each pipeline reports its own throughput metric: documents per second for ingestion, and tasks per second for QnA.

For ingestion, documents per second is the natural unit. Each document flows through parse, chunk, embed, and index, and the document is the fixed unit of work every submitter starts from, which makes documents per second directly comparable across systems. More details can be found in inference rules.                                                                                                                                                  

For QnA, the usual language-model metric, tokens per second, doesn’t fit: the pipeline runs two different-sized language models alongside non-language-model components, so a single token rate can’t represent the whole system. A per-hop metric like hops per second is hard to interpret, since a QnA task takes a variable number of hops. Tasks per second avoids both problems by measuring the unit that matters, one query answered end to end. It looks small next to other benchmarks because a single query can run up to five hops and a dozen or more language-model calls.

The QnA pipeline is also a complex, dynamic workload that is hard to make reproducible, on top of the inherent non-determinism of language-model generation. For performance runs, recorded reference inputs are provided for each stage of each hop, fixing the number of hops and the documents retrieved. Outputs are still generated normally but discarded, so the pipeline does real work while run-to-run variation stays minimal.

Accuracy metrics

Retrieval quality (against the required Wikipedia URLs) is not an official metric. It is a database-integrity check, used to confirm that a submitter’s independently built vector database behaves like the reference so that everyone is answering from the same corpus.                                                                                                            

Reference values over the full 824-query set:

A submission is valid if its answer accuracy is at least 97% of the reference accuracy.

Answer accuracy also varies with the kind of reasoning a query demands:

Answer accuracy also varies with the kind of reasoning a query demands. The breakdown below is insights only, and a query can carry more than one reasoning type.

Multiple-constraint queries score highest, since they mainly require gathering discrete facts, exactly what dense retrieval and a language model are good at. Numerical and tabular queries are hardest: the answer depends on pulling exact figures out of prose or tables and then computing over them, so a single misread number or a chunk boundary that splits a table fails the whole query. Temporal and post-processing queries sit in between, needing date arithmetic or a final transformation on top of correct retrieval.

Closing these gaps points naturally toward a more agentic pipeline: a knowledge graph or structured index the system can query for entities and relations, table-aware parsing and chunking so figures survive ingestion intact, and equipping the answer step with tools such as a code interpreter to handle arithmetic and unit conversion. Each is a step from today’s fixed retrieve-and-read loop toward an agent that chooses how to find and process what it needs.                                                                                                                                                   

Compliance

E2E-RAG-QnA requires a single compliance test, TEST09 (output-token-length verification) which re-runs the workload in LoadGen performance mode with the audit.config and compares the mean output token length of the answer-generator response against the reference. The only component visible to LoadGen against the reference implementation’s mean OSL of 273.81 tokens with a ±10% band (246.43 – 301.19). This guards against a submission cutting work by generating systematically shorter answers in performance mode than the reference does.

The check is a single aggregate on one pipeline stage, it does not provide any guard to retrieval, reranking, or intermediate query/sufficiency LLM calls. Future revisions could add distribution-level or per-component compliance coverage.

Optimization Opportunities

E2E RAG pipeline is intentionally designed to expose multiple layers of optimization opportunities across the full inference stack from the Vector DB Generation following the rules algorithmic criteria to inference serving framework, with dense and sparse kernels to KV-cache-aware scheduling. The reference implementation provides a clear but unoptimized baseline in several of these dimensions, leaving room for submitters to explore system and kernel improvements while preserving accuracy. 

• CPU Pipeline: MLPerf Loadgen provides the entire 824 queries/tasks at one-shot (in this offline scenario) to the serving framework. This provides a great opportunity for the submitters to explore how to spawn the tasks across several cores of the CPU with appropriate affinity to their corresponding accelerators with memory placement strategy for the E2E pipeline and within each of the ingredients in the pipeline. 
• GPU Partitioning with KV Cache Balancing: Effective Multi-model placement with low latency within accelerators and across accelerators to saturate all the ingredient accelerators participating in compute/comms/storage. This includes efficient low-latency high throughput CPU-GPU interaction, GPU-GPU/Accelerator Interaction/Compute vs. COMMS Balancing across the macro & micro batching to keep a tight pipeline. 
• Multi-model with different precisions: All the selected models can be operated in different precision as long as the E2E Accuracy criteria is met. This provides greater innovation opportunity to selectively choose the right precision that can provide best performance at system level. 

Conclusion

This is MLPerf’s first benchmark to measure an entire RAG pipeline end to end, rather than a single model in isolation. It captures what actually governs a deployed RAG system: multiple models of different roles and sizes, served together across an iterative, multi-hop loop.

This blazes the trail for the future Agentic RAG.  The retrieve-reason-decide loop at its core is the same loop that drives agents, and the natural next steps — tool use, structured retrieval, more autonomy in how the system finds and processes information — move it steadily toward a full agentic benchmark. We look forward to evolving it with the community with EndPoints/Agentic Intercept in near future

References

• Getting started with E2E-RAG reference implementation.
• Reference implementation. 
• MLPerf Inference Benchmark Data Download 
• A new GPT-OSS benchmark and DeepSeek R1 updates for latency-optimized reasoning – MLCommons 
• MLPerf Inference Rules for E2E-RAG
• FRAMES Paper 

Acknowledgements

We thank MLCommons E2E RAG Taskforce participants and the MLPerf Inference Working Group, MLCommons Platform Engineering team for their feedback, support and guidance in developing this benchmark. Our sincere appreciation to all the contributors across the industry & academia that was very instrumental to weigh-in evaluation of various proposals! We are grateful to everyone to make this first unique benchmark definition happen with collaboration across various time-zones.

We also thank the Multi-Turn Taskforce, whose ideas helped us address the non-determinism inherent in multi-hop benchmarking.

We are especially grateful to Satyapriya Krishna, one of the authors of the FRAMES dataset, who joined our discussions to offer clarifications and guidance. 

The post Introducing the MLPerf End-to-End RAG Inference Benchmark appeared first on MLCommons.

MLPerf Client measures how effectively PCs—from laptops and desktops to workstations—run AI workloads locally. With the release of v2.0, the benchmark now includes new Agentic AI and Image Generation capabilities alongside its established LLM benchmarking of real-world tasks like summarization, content creation, and code analysis, reporting metrics for both responsiveness and throughput.

New in MLPerf Client v2.0

Version 2.0 introduces a range of new features, refinements, and expanded support designed to keep pace with the rapid evolution of AI-enabled hardware and software:

Major Workload Changes

• New Image Generation Category: Adding this category significantly enhances the benchmark’s utility by enabling evaluation of generative visual capabilities. This category features Flux.2 klein 4B as an experimental test.
• New Agentic AI Category: Benchmarks agentic AI performance through Software Engineering (SWE) Agent and Data Analyst Agent scenarios. It reports end-to-end performance, including a breakdown of LLM inference and tool execution times.
• Updated LLM Inference tests: This release upgrades Phi 3.5 mini instruct in the required workloads to Phi 4 Mini Instruct, while introducing Qwen 3 8B as an experimental test. Base tasks now include an Intermediate Summarization task featuring an input prompt of roughly 4K tokens.

These updates mark another major step toward a comprehensive, cross-platform benchmark for client AI computing.

MLPerf Client is the product of a collaboration among technology leaders, including AMD, Intel, Microsoft, NVIDIA, Qualcomm Technologies, Inc., and top PC OEMs. These participants have contributed resources and expertise to the creation of this AI benchmark.

MLPerf Client v2.0 is freely available for download, and the source code is open for inspection and community contribution via the MLCommons GitHub repository. For downloads, documentation, and additional details, visit mlcommons.org.

GitHub releases (Windows, macOS, Linux): https://github.com/mlcommons/mlperf_client/releases

About MLCommons 

MLCommons is an open engineering consortium with a mission to make machine learning better for everyone. The organization produces industry-leading benchmarks, datasets, and best practices that span the full range of ML applications—from massive cloud training to resource-constrained edge devices. Its MLPerf benchmark suite has become the de facto standard for evaluating AI performance.

Learn more at www.mlcommons.org.

Press Contact: press@mlcommons.org

The post MLPerf Client v2.0 Expands AI PC Benchmarking with Image Generation and Agentic AI appeared first on MLCommons.

Some of those numbers are real. Some are what we’d call benchmark washing – the selective use of convenient results to imply performance, reliability, safety, or readiness that the evidence doesn’t actually support.

We build benchmarks for a living. We started MLPerf in 2018 – years before the AI evaluation explosion – and created the industry-standard benchmark for AI systems. We’ve spent years building the governance, audit trails, and integrity controls that make results defensible. We’ve also seen, up close, just about every way a benchmark can fail, be gamed, or be misused.

We don’t sit in your procurement meetings. We’re not claiming to know exactly what your evaluation workflow looks like from the inside. But we hear from enterprise teams that they need a way to tell the difference between a benchmark result they can trust and one they can’t. This is our attempt to help with that – not as a vendor selling you an assessment, but as a consortium that has learned some hard lessons about what makes benchmark evidence hold up and what makes it fall apart.

The Benchmark Trust Test

When someone hands you a benchmark result and asks you to make a decision based on it, here are the questions we’d ask. It isn’t realistic that every enterprise becomes a benchmarking expert – but every enterprise needs to become an intelligent benchmark consumer. We believe these questions are worth carrying into any conversation in which a benchmark score is used to support a consequential decision.

If they’re hard to answer, if the person presenting the result can’t answer them, or hasn’t thought about them… that’s noteworthy.

1. Does this benchmark actually measure what I’m deciding about?

A good benchmark starts with a decision, not a dataset. It should be clear what the benchmark is for – model selection, vendor shortlisting, release readiness, risk acceptance, cost-performance trade-off. We’ve seen the failure mode where a benchmark gets used because it’s available and familiar, not because it’s relevant to the decision at hand. The result can be perfectly accurate and still misleading. A high score on a knowledge benchmark doesn’t tell you much about reliability under production workload. A strong coding benchmark doesn’t prove secure code generation. 

2. Where did the data come from, and has it been kept clean?

The evaluation population, sampling method, labeling, provenance, and known limitations should be documented. More importantly – and this is a problem we think is more widespread than many people realize – the data should be checked for contamination. Has the system seen the test data during training? This isn’t a rare accident. Survey literature now treats test-set leakage as a structural, recurring threat. The data tells a clear story. 

In a recent analysis of a grade-school arithmetic benchmark, researchers discovered that accuracy can plunge by 13 points across several model families once a fresh, equivalent test set is introduced to eliminate contamination. Perhaps the most sobering evidence comes from software engineering. On the public SWE-Bench Verified, elite models frequently exceed 70%. However, when Scale AI’s research lab utilized its SWE-Bench Pro, which relies on private, proprietary repositories that models haven’t seen, those results evaporated. GPT-5’s performance dropped from 23% to less than 15%, while Claude Opus 4.1 sank from 23% to 18%. This represents a 55-point discrepancy between public and private evidence, all on a metric that teams are currently using to justify high-stakes investments in automated coding infrastructure.

If a benchmark can’t demonstrate contamination controls, a high score might reflect memorization rather than capability. And as the SWE-Bench example shows, the gap between those two things may be significant enough to impact such decisions.

3. Could someone else reproduce this result- and reconstruct what happened?

A trustworthy benchmark controls the conditions under which evidence is produced: model version, prompts, hyperparameters, hardware, random seeds, scoring method, post-processing. This isn’t aspirational. We’ve seen prompt formatting alone swing measured accuracy by tens of points, with format-based rankings barely correlating across models. A single number as a benchmarking result is both an artifact of conditions and a property of the system. If the benchmark can’t specify and reproduce those conditions, the score is more of a snapshot than a measurement.  

Reproducibility has two faces, and both matter. The first is the peer-review perspective: could another benchmarking team take the specification and re-run the evaluation themselves? This is the lens we apply in MLPerf – the rules exist so that a qualified reviewer can reconstruct the result and verify it was produced fairly. The second is the auditor perspective: could an independent assurance provider – someone who isn’t a benchmarking expert but needs to verify evidence for compliance, procurement, or risk purposes – trace what happened? Who ran the benchmark, when, on what system, with what inputs, how were outputs scored, what exceptions occurred, and how were results approved? This is the lens we apply in our AI Risk and Reliability work, where the audience is often an auditor or assurance provider rather than a fellow researcher.

Both perspectives ask the same underlying question: is there enough of an evidence trail that someone other than the person who produced the result can stand behind it? If that trail doesn’t exist, the result can’t be audited – and using a result that can’t be audited to inform a production decision carries additional risks..

4. Is the score telling you everything you need – or just the easiest thing to measure?

A single score is easy to communicate. It’s also easy to over-rely on. When a benchmark reports one number – usually accuracy or resolve rate – it’s worth asking what’s missing. Quality, latency, cost, reliability, safety, efficiency – these dimensions often matter as much as the headline metric, and they frequently trade off against each other. A model that scores highest on accuracy might also be the most expensive to run, the slowest in production, or the least robust to edge cases. We don’t think every benchmark needs to measure everything. But we do think it’s worth understanding what a benchmark isn’t measuring, and whether the gaps matter for the decision you’re making. A benchmark that reports only accuracy isn’t wrong – but a claim that attempts to extend that accuracy measurement into “the best system for your use case” is worth being skeptical of.

5. Is the system under test a passive subject – or could it be gaming the test?

This is a newer problem, and we’ll be honest: it surprised us. Frontier models can sandbag – strategically underperform to hit a target score. They exhibit evaluation awareness – they can often tell when they’re being tested and infer what’s being measured, which means test behavior may not transfer to production. In agentic settings, systems have been caught searching public repositories (and, making headlines recently, breaking into private ones) for a benchmark’s answers instead of solving the task. A benchmark that assumes the system is passive is operating on an assumption that, in our experience, no longer holds for capable models. Look for integrity controls: blinded protocols, held-out items, trajectory inspection, continuous stewardship of the evaluation – not just final scores.

6. If a model is scoring the results, has that model been validated?

As benchmarks move toward open-ended generation, many rely on an LLM to score another LLM’s output. This is scalable, but the judge is itself a fallible instrument – biased toward longer answers, particular styles, self-preference, and position in comparisons. If the benchmark uses automated scoring, we’d want to know that the judge has been anchored to human judgments on a documented sample, checked for bias and stability, and version-controlled. Where the judge can’t be made reliable, objective ground truth should carry the weight instead.

7. Is the benchmark still alive – or has it gone stale?

A benchmark isn’t trustworthy (or even particularly useful) if it was abandoned after launch. Models saturate benchmarks. Data distributions shift. Threat models change. A benchmark that discriminated between systems two years ago may not discriminate today. We believe this is one of the more underappreciated risks in enterprise settings: the benchmark keeps producing numbers, the numbers just stop carrying the same meaning. Look for a maintenance plan: scheduled reviews, drift analysis, contamination checks, retirement criteria. If the benchmark hasn’t been refreshed, its scores may be measuring adaptation to the test, not real capability.

Why This Is Getting Harder, Not Easier

We’ve been at this for years, and here’s what concerns us: these problems aren’t just present – they’re compounding. Contamination is structural. Leaderboards create incentives to optimize for the leaderboard. Safety benchmarks can measure capability, not safety. And once a benchmark saturates, it stops discriminating between systems but keeps producing confident-looking numbers. Each of these failures is documented. What’s harder to see is how they interact – a contaminated, saturated benchmark that’s been gamed by well-resourced providers and cited in a vendor deck is a stack of compounding distortions, each one invisible on its own.

The “I told you so” here isn’t directed at anyone in particular. It’s the recognition that these problems are now well-documented enough that pretending they don’t exist is itself a form of benchmark washing.

What Good Looks Like

Not all benchmarks are built for the same job. We find it useful to distinguish between two types.

A research benchmark is designed to explore – to test hypotheses, reveal emerging capabilities, create shared reference points, and push the frontier of what can be measured. BIG-bench, MMLU, HELM, and CheckList are examples. They’re built for discovery, speed, and comparability, not for operational decisions.

An industrialized benchmark is designed to support specific decisions – procurement, model selection, release gates, risk acceptance, regulatory claims. Its design, operation, evidence, and maintenance are robust enough to support production decisions and external accountability.

The distinction isn’t about size or sophistication. A narrow benchmark can be trustworthy and fit for industrial use if its purpose is clear, its limitations are documented, its execution is controlled, and its outputs are interpreted within appropriate boundaries. A sophisticated test set can be untrustworthy if it lacks governance, repeatability, and auditability.

An industrial benchmark has:

• Explicit scope and decision linkage – it knows what decision it informs and doesn’t claim more than it measures
• Governed data – documented provenance, contamination controls, and stated limitations
• Repeatable execution – versioned data, code, configuration, prompts, hardware, and seeds
• Awareness of what the score doesn’t cover – trade-offs between dimensions like cost, latency, and quality are surfaced where they matter, and claims don’t generalize beyond what the benchmark actually measures
• Honest uncertainty – variance and confidence are reported where they’re material, so “Model A beat Model B” is a claim that can survive scrutiny, not just a headline
• Auditability – complete evidence trails that an independent reviewer can reconstruct
• Integrity controls – blinding, refresh, red-teaming, and trajectory inspection for systems that can game the test
• Lifecycle maintenance – scheduled review, drift monitoring, retirement criteria, and controlled change logs

Our MLPerf family of benchmarks is our flagship example, and we’re proud of them. MLPerf Inference is on version 6.1, and each version encapsulates new learnings about how to build industry-grade benchmarks. Its rules address fairness, consistency of systems and frameworks, shared implementations, restrictions on non-determinism, prohibitions on benchmark detection, mandatory replicability, and audit processes. That level of discipline isn’t bureaucracy for its own sake. It’s what makes a result defensible when someone’s procurement decision, product claim, or regulatory filing depends on it.

AILuminate is rapidly catching up to MLPerf as our second family of industry-grade benchmarks. While MLPerf focuses on performance measurement (time-to-first token, concurrency, power consumption, etc.), AILuminate measures reliability, which involves more subjective topics such as safety and jailbreak resilience. We launched AILuminate in 2024, and that ongoing effort continues to build on our perspective of “What Good Looks Like.”

This doesn’t mean every benchmark needs to be MLPerf or AILuminate. We believe credible performance and reliability claims require comparable controls, scaled to the stakes. A benchmark for a low-risk internal assistant may need lightweight governance and monthly review. A benchmark supporting clinical, financial, or safety-critical deployment probably needs independent review, formal audit trails, subgroup analysis, and adversarial testing. The standard, as we see it, is proportionality. The evaluation rigor should match the consequences of potential failure states – and the benchmarks used in those evaluations should actually measure those failure states.

Research Benchmarks Still Matter

To be clear – this piece is not an argument against research benchmarks. We use them. We value them. They are where new measurement frontiers open, where capability gaps get revealed, and where the concepts that later become production controls are first discovered. They are the laboratory. Industrialized benchmarks are the operating plant. You need both.

The mistake isn’t using research benchmarks. The mistake is treating exploratory evidence as production assurance without industrializing the evaluation process around it. A research benchmark that’s transparent about its assumptions and limits is doing its job. A research benchmark whose results show up in a vendor deck as proof of production readiness is being misused – and that misuse is benchmark washing.

Why We’re Saying This

We’re a consortium, not a vendor. We don’t have a model to sell or a leaderboard to win. Our incentive is to make benchmark evidence more trustworthy, not to make any particular result look good. We think that’s worth something – and we think it’s the reason a piece like this is better coming from us than from anyone who has a model in the market.

We also think independent validation has real value. Not because we’re the only ones who can provide it, but because the principle itself – that you shouldn’t grade your own homework, and that the person measuring the system shouldn’t be the same person selling it – is one we’ve seen violated often enough to know it matters.

A benchmark result is only as meaningful as the conditions under which it was generated, the representativeness of what it measures, the integrity of the process that produced it, and the honesty with which its limitations are disclosed. If you can’t answer the eight questions above about a benchmark result you’re being asked to trust – ask. If the answers aren’t there, we believe the result probably isn’t either.

Further Reading

BetterBench – Stanford’s benchmark quality assessment framework, which evaluates 24 AI benchmarks against 46 lifecycle criteria. MLCommons AI Safety v0.5 is among the benchmarks assessed and scored well.

Goodhart’s Law Comes for Every Benchmark You Trust – Communications of the ACM, 2026. An accessible, sharp overview of contamination, leaderboard gaming, and what actually survives Goodhart’s Law.

The Leaderboard Illusion – Singh et al., 2025. The multi-institution analysis of how public AI leaderboards can be gamed through undisclosed private testing, selective score retraction, and unequal data access.

HELM: Holistic Evaluation of Language Models – Stanford CRFM. A living, multi-metric evaluation framework that demonstrates what “more than one number” looks like in practice, across accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency.

NIST AI Risk Management Framework – NIST, 2023. The US government’s voluntary framework for managing AI risks, with a “Measure” function that directly addresses evaluation, benchmarking, and monitoring.

Can We Trust AI Benchmarks? – European Commission Joint Research Centre, 2025. A comprehensive interdisciplinary meta-review of ~100 studies on benchmark shortcomings, covering construct validity, contamination, gaming, saturation, and commercial dynamics.

The post How to Tell When a Benchmark Is Worth Trusting appeared first on MLCommons.

MLPerf was originally designed to help a relatively small number of cloud providers and downstream system builders inform their AI hardware purchasing decisions. Today, procuring inference compute is a salient business decision for companies of all sizes – and it means evaluating options across neoclouds, cloud providers, and managed services simultaneously. These buyers need reliable, comparable, and independent performance benchmarks to inform their purchasing decisions, and these benchmarks must be dynamic enough to keep pace with an industry that launches new models weekly. 

MLPerf Endpoints is designed to meet this more diverse and complex demand for benchmarks to inform AI system procurement decisions. Four key principles are guiding our development of enterprise-buyer-centric benchmarks:  

1. Current: Results must keep pace with the market. Buyers can’t wait months for a benchmark round to include new hardware or models. 
2. Comprehensive: Buyers need benchmarks that cover the many competing inference providers, systems, and workloads available for purchase. 
3. Comparable: Buyers need to see apples-to-apples results across vendors, normalized for cost or power, to inform procurement decisions.
4. Commentary: To support buyers in these decisions, Endpoints provides additional context for benchmark results through visualizations, data filtering, and analysis. 

MLPerf Endpoints v0.7 is a foundation release with initial results from Coreweave, Google, Intel, KRAI, and Nvidia. We want to congratulate these members on their excellent results, spanning several orders of magnitude in performance across 3 benchmarks. This is the infrastructure on which we are building a more dynamic, comprehensive, and comparable inference benchmark suite for the data center. Endpoints currently supports automated submission pipelines, continuous review tooling, and dynamic visualization of results that you can see at mlcommons.endpoints.com. We are also evolving our benchmarking rules towards more buyer-centric benchmarks.

Later this year, we will deliver MLPerf Endpoints v1.0 with more buyer-centric rules, normalization, and an expanded set of benchmarks including agentic workloads and then open the rolling submission process to our broader membership. The rolling submission process ensures Endpoints will provide current and up-to-date results that move at the pace of the market. 

We’re grateful to our 30+ supporters who have helped guide the development of MLPerf Endpoints, including AMD, Argonne National Laboratory, Broadcom, Core 42, Dell, HPE, Lambda, Oracle,  and Red Hat. Their partnership has been invaluable in stress-testing our rules, processes, and improving our infrastructure. With this release, we are just getting started. 

If you’re a system or service provider, now is the time to get involved, help shape the rules, and plan your submission at a time of your choosing. Complete this form to let us know you’re interested in submitting, and what hardware you want to submit on. The community-developed rules are actively being refined for the 1.0 release later this year, and early participants in this effort will shape Endpoints’ direction. If you’re an enterprise buyer, Endpoints is being built for you, and we want to know how we can best support you and your organization. Let us know by emailing alejandro@mlcommons.org.

We deeply value the trust our members place in us as the benchmark of record for inference. We are excited to build this future of MLPerf with all of you. 

[1] A. Tschand, A. T. R. Rajan, S. Idgunji, et al., “MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from µWatts to MWatts for Sustainable AI,” in 2025 IEEE International Symposium on High-Performance Computer Architecture (HPCA), 2025. arXiv:2410.12032.

[2] D. Kanter, M. Ahmad, H. Kassa, and S. Rishab, “MLPerf Training v4.1 Results — Press Briefing,” MLCommons, Nov. 13, 2024. [Online]. Available: https://docs.google.com/presentation/d/1KSIJBvIV9OcswF1mVbhGRN0nUWbSgbYSoxu6dHCwajM/

The post MLPerf Endpoints v0.7: A Foundation Release appeared first on MLCommons.

A brain tumor (glioblastoma) is a rare disease with devastating outcomes for life expectancy. AI has the potential to improve the diagnosis and prognosis of brain tumor patients through a multitude of automated processes – including boundary definition/segmentation, tumor classification, and tumor quantification. Thanks to the Federated Tumor Segmentation (FeTS) consortium, coordinated by Dr. Spyridon Bakas at Indiana University School of Medicine and the Response Assessment in Neuro-Oncology (RANO) cooperative group, the MedPerf team demonstrated evaluation of a clinically valuable AI model – designed by Dr. Evan Calabrese’s team at Duke University – on real-world brain tumor MRI data.

With MedPerf on Google Cloud’s Confidential Computing, we aim to advance medical AI research worldwide while securely evaluating models on medical data. Using Google Cloud’s Confidential Space, a Trusted Execution Environment (TEE), medical data, model weights, and benchmarks are fully protected throughout the benchmarking process.

The Problem: Life-Threatening Disease, Locked-Down Data

Protecting both AI models and patient data in the medical space is paramount, driven by privacy, ethical, sovereign, and regulatory considerations. At the same time, measuring the performance of AI models on real-world patient data is critical for enabling trust and adoption of AI among clinicians, patients, regulators, and payors.

The MLCommons community built MedPerf to address exactly this gap. MedPerf uses a federated approach – introduced to healthcare by Dr. Bakas’ team – in which AI models move to the medical data owners (such as healthcare organizations, hospitals, and data brokers), rather than requiring data to move to the benchmark operator. This allows data owners to keep their data private while sharing only summarized results with the benchmark operator.

However, while this federated approach protects patient data, it does not fully address concerns around model theft, intellectual property, and benchmark integrity. Without additional guarantees, there is no assurance that model weights won’t leak or that benchmark results won’t be tampered with at the time of execution – raising serious concerns about trust and participation in the benchmark itself.

The Solution: MedPerf on Google Cloud Confidential Space

MedPerf’s integration with Google Cloud’s Confidential Space solves exactly this problem: protecting both model weights and benchmark results while simultaneously safeguarding medical data.

This integration brings four key benefits to MedPerf users:

1. A global test dataset for healthcare AI – enabling evaluation at unprecedented scale
2. Patient data is protected – data never leaves the data owner’s secure environment
3. Model IP is protected – model weights remain encrypted and inaccessible to unauthorized parties
4. No data sharing required – results are summarized and encrypted before transmission

The Demo: A Step-by-Step Secure Benchmark

A benchmark authority first publishes a benchmark on the MedPerf server, describing the benchmark’s aim, data preparation guidelines, and evaluation criteria. Data owners and AI model owners then register metadata for their private assets with the MedPerf server and link their registrations to the benchmark.

Private assets are encrypted using Google Cloud’s Cloud Key Management Service (Cloud KMS) and uploaded to a Google Cloud Storage bucket. Access to Cloud KMS and the bucket is governed by a Google Cloud workload identity pool, which allows access only if the request originates from a confidential virtual machine with specific measurements – such as the benchmark container hash and input hash.

When it’s time to run, data owners execute the benchmark container inside a Confidential Space virtual machine. The container receives cryptographic evidence of the trusted execution environment’s configuration, which is exchanged for access tokens from the model owner’s and data owner’s workload identity pools. These tokens retrieve and decrypt the model and dataset, inference is executed, results are encrypted with the data owner’s public key, and the encrypted results are returned to the data owner’s bucket.

The video below demonstrates a brain tumor segmentation AI model running on a clinical dataset in Google Cloud:

Watch the Demo

Voices from the Community

“My experience testing federated learning on Google Cloud has shown that the future of medical AI lies in secure, scalable, and collaborative cloud environments. Moving beyond the controlled lab setting to test these workflows in a production-ready infrastructure provided a unique opportunity to evaluate the performance and security of federated learning in real-world clinical applications. This collaboration between the FeTS community, MLCommons, and Google Cloud demonstrates how scalable cloud solutions can accelerate the development of high-precision diagnostic tools in neuroradiology.” – Dr. Yury Velichko, Northwestern University

“Federated Learning offers a paradigm shift in healthcare multi-site collaborations that expedites access to unprecedented amounts of knowledge from diverse patient populations, towards robust generalizable AI models with real clinical impact. Security and privacy are prime considerations to ensure that FL-related assets are always protected. Training and evaluating AI models on confidential compute infrastructure, using MLCommons’ MedPerf on Google Cloud, describes a complete exemplary solution for large-scale real-world clinically-relevant AI studies in healthcare, providing the ability to track how well an AI model continues to perform after it has received regulatory approval, for example, from the FDA.” – Dr. Spyridon Bakas, Indiana University School of Medicine

“One of the greatest barriers to trustworthy medical AI has been the tension between the need for real-world data and the need to protect it. The integration of MedPerf with Google Cloud’s Confidential Space is a tangible solution to that tension – one that the MLCommons community has worked hard to bring to life. We are proud to demonstrate what is possible when open collaboration meets production-grade security infrastructure.” – Alexandros Karargyris, MedPerf Lead, MLCommons

Join the MedPerf Community

MedPerf is an open-source effort developed by the medical AI community to address real community needs. Thanks to contributions from the MLCommons community, MedPerf has already supported impactful clinical AI studies, including the FeTS challenge and FeTS 2.0 for brain tumors.

We know we still have much more to do – and we need your help. We are looking for:

• Research investigators with the ambition to lead global clinical AI studies
• Computational experts who want to build disruptive AI on real-world medical data
• Infrastructure engineers to improve real-world benchmarking technologies
• Privacy and governance experts to provide input on medical data access controls and policy
• Regulatory experts to provide feedback on bringing regulatory alignment to real-world benchmarking

Acknowledgments

Contributors to the MedPerf confidential compute integration include Google Cloud team lead Nelly Porter, Dr. Spyridon Bakas from Indiana University School of Medicine, Dr. Yury Velichko from Northwestern University, and Dr. Amber Simpson from the University of Alberta (formerly at Queen’s University).

The post MedPerf Meets Google Cloud Confidential Computing: Secure AI Benchmarking for Brain Tumor Research appeared first on MLCommons.