The highlights of this round include a record number of submitters, new accelerators that significantly improve per-device performance, and a new multi-turn benchmark. Read on for our results analysis.

Submitters and Submissions

This round brought a record 30 submitters spanning silicon vendors, system builders (OEMs and ODMs), cloud and neocloud providers, and specialized inference-software companies.
Several results this round were joint submissions, where two organizations collaborated: Dell_AMD, Dell_MangoBoost, RedHat_Intel, and RedHat_Supermicro.

The table below provides a breakdown of the systems and scores submitted by each organization:

Altogether, 120 systems across the Datacenter and Edge suites, in both the Closed and Open divisions, were submitted.

Benchmarks

MLPerf Inference benchmarks are divided into Datacenter and Edge categories. Each benchmark supports several scenarios, which may be one or more of: Offline, Server, Interactive, SingleStream, or MultiStream. Submissions can also be classified as either Closed or Open. Closed submissions require using a model that is mathematically equivalent to the reference implementation, essentially holding the model fixed to enable direct comparisons. For more details on the MLPerf Inference benchmark setup, visit https://mlcommons.org/benchmarks/inference-datacenter/.

MLPerf Inference v6.1 features 10 Datacenter and 6 Edge benchmarks, two of which are new: End-to-End RAG and Agentic Edge Inference. Two benchmarks have been updated: For VLM (Visual Language Model based on Qwen3), an Interactive scenario has been newly defined, and in GPT-OSS-120B, the interactive scenario has been updated to allow speculative decoding.

MLPerf Inference submitters can choose to submit to any of the benchmarks. In the Datacenter category, the most popular benchmark is gpt-oss-120b, followed by Llama2-70b. This is the first time gpt-oss-120b has topped the list, after Llama2-70b was the most popular for the last few rounds. This demonstrates that the MLPerf benchmarking community (which highly values stability and comparability) is now fully embracing MoE models.

Performance improvement this round

To gauge performance improvements since the last round, we compared the best per-accelerator Offline and Server scores against those from v6.0.

The biggest gains are in VLM and DeepSeek R1 for both Offline and Server scenarios, significantly surpassing gains in other benchmarks. This is because those 2 benchmarks were submitted on a new Preview category system powered by Nvidia Vera Rubin. Other benchmarks’ top scores were achieved on the same hardware as in 6.0, so results reflect more gradual performance improvements from software stack and algorithmic optimizations.

Llama2-70b is the longest-running LLM in the MLPerf Inference suite, being introduced in early 2024, round v4.0. As such, it is a strong vehicle for showing LLM performance improvements over time. This is shown below in median per-accelerator performance for Server scenario submissions, which has improved 5.58x over 6 runs. Three main reasons contribute to the performance gains: 

1. Reduced numerical precision: Some v6.1 submissions use FP4 precision, whereas earlier rounds used FP8. Since MLPerf Inference has strong accuracy requirements, the performance gains did not come at the cost of accuracy loss.
2. New generation of accelerators. Accelerators that were used in the initial rounds of Llama2-70b benchmarks have been replaced with newer, more performant ones in the latest rounds.
3. Software performance improvements. Every submission round sees performance improvements even on the same hardware, a testament to the improvements our submitters bring and a reflection of continued improvement in the field.

A similar trend appears in the DeepSeek R1 benchmark, despite being around for less time. The figure below shows that best per-accelerator performance has increased by 2.7x in Offline and 5.7x in Server scenarios, respectively, within a year. Similarly, the Interactive scenario, which had only 2 submission rounds, increased by 2.7x over that period.

Results Highlights

Beyond performance improvements, results also contain additional important achievements. One continued trend is multi-node inference. In the last few rounds, the number of multi-node submissions has increased considerably, as shown in the figure below:

The upward trend started in round 4.1, 2 years ago, and this round reached an all-time record of 16 multi-node submissions. We also see the size of submitted systems increasing. Just last round, a record of 288 accelerators was set, only to be broken in this round by Cruose submissions 6.1-0026 and 6.1-0027, which used 512 accelerators. The latter of the submissions set a new record for the number of tokens/second generated in the MLPerf Inference benchmark by generating almost 5.8M tokens/second in GPT-OSS-120b offline test.

Another trend in MLPerf Inference is hybrid submissions that use different types of accelerators working together. These systems pose unique challenges because submitters must account for the different computational capabilities of the devices. This round contains two such submissions:

New Accelerators and System

v6.1 introduced a diverse set of newly submitted accelerators and systems — from on-device parts to a next-generation rack-scale platform — spanning multiple accelerator vendors and different scales of deployment.

Submitters in their own words

Many participating organizations submitted supplemental statements describing what they consider most significant about their v6.1 submission. Full statements are available in the results repository; here are the highlights.

Summary

As always, these results are most valuable when read in context: the Closed division remains the foundation for apples-to-apples comparison, while the Open division surfaces the optimization techniques that often preview the mainstream. We encourage readers to consult both the results tables and the submitter statements.

A round like this reflects an enormous collective effort. Our thanks go to every submitting organization — established vendors and first-time participants alike — and to the many Inference Working Group members who build, review, and audit these benchmarks and keep MLPerf rigorous, relevant, and fair.

The post Where the Industry Is Investing: A Look at MLPerf Inference v6.1 appeared first on MLCommons.

The open-source MLPerf Inference benchmark suite measures system performance in an architecture-neutral, representative, and reproducible manner. It creates a level playing field for competition, driving innovation, performance, and energy efficiency across the entire industry. The published results provide critical technical information, allowing customers who are procuring and deploying AI systems to make informed decisions based on trusted empirical data.

Evolution of the benchmark: new tests, new support for popular optimizations 

MLPerf Inference v6.1 introduces two new tests to the suite that reflect the industry’s evolution towards more complex, multi-step, and agentic deployments of AI inference technology, both in the datacenter and at the edge.

The End-to-End Retrieval-Augmented Generation (RAG) benchmark tests a complex pipeline of multiple distinct AI models for question answering. This workload decomposes the task into a sequence of steps: an embedding model turns a query into a vector, a retriever pulls candidate passages from a vector database, a re-ranker refines the list, and one or more LLMs reason over the data to produce an answer. The test generates performance data for two separate but related tasks: ingesting a document corpus to create a vector database, and query-answering against a pre-built vector database. More information on the End-to-End RAG benchmark is available here.

The Edge Agentic Inference benchmark reflects the shift from single-shot interactions toward complex multi-turn workloads such as agentic coding. Unlike single-shot inference, agentic workloads feature growing conversational history in which each query depends on those that came before, and the model merges evidence gathering with subsequent reasoning over that evidence, iterating as necessary to arrive at a final result. This is a much more powerful usage model and unlocks tremendous new capabilities that single-shot inference lacks, while radically increasing demands on AI compute. Agentic AI deployment is increasingly popular at the edge, where serving one user at a time with fixed memory, processing, power, and context creates unique performance challenges. This test adopts the framework and methodology from the soon-to-be-released MLPerf Agentic datacenter performance benchmark and specializes it with an edge model and quantization, a single-stream coding workload, latency metrics, and a statistically robust accuracy gate. It incorporates two tasks: measuring workload accuracy under time constraints and measuring the performance of a deterministic workload in which accuracy checks are intrinsic to the workload. More information on the Edge Agentic Inference benchmark is available here.

MLPerf Inference 6.1 also adds support for a performance optimization now in common use in production inference systems. Speculative Decoding, a popular optimization technique that predicts and verifies multiple tokens in a single forward pass, is now supported in the interactive scenario for two of the inference benchmarks, with added support in the GPT-OSS task. 

“We are working hard to ensure that the MLPerf Inference benchmark continues to reflect the scenarios that the AI community values most,” said Miro Hodak, MLPerf Inference working group co-chair. “We added the End-to-end RAG test because it’s clear that query-answering has evolved beyond simply an LLM trained on a corpus; stakeholders need to understand the real-world performance of the types of multi-step, multi-component pipelines that are being built today. Likewise, we added the Edge Agentic Inference test because complex inference systems with agentic properties are increasingly hosted on edge computing devices, creating a new set of performance challenges our customers face today. We are committed to providing timely and relevant performance data that reflects real-world production systems and the performance optimizations that are being deployed today.”

Submissions and Results Demonstrate Speed of AI Innovation

This round of the benchmark highlights the rapid evolution of AI technology and deployments, with submitted systems representing a range of hardware options across accelerators, boards, workstations, and servers. The results include five new processors or accelerators: AMD Ryzen AI Max+ 395, AMD Instinct MI350P, and Intel Arc Pro B70 are all available, and NVIDIA Rubin and NVIDIA Vera Rubin NVL72 is in preview.  They also include the largest system ever submitted to the MLPerf Inference benchmark, with 512 accelerators and two novel heterogeneous systems. The first heterogeneous system combines high-performance networking with accelerators from two different vendors, and the second system is geographically distributed – spanning the Pacific Ocean. 

Performance gains continue to compound and shatter records for AI accelerator capabilities. For the Visual Language Model (VLM) test, the best per-accelerator server scenario result for this benchmark round improved by 2.99x from v6.0 just six months ago. For the Deepseek R1 test, the best per-accelerator server scenario result was 5.7x better compared to v5.1 one year ago. This performance gain translates into more capabilities for end customers, potentially serving more users or enabling more sophisticated inference use cases.

“With the critical data from the MLPerf Inference benchmark, the AI community is once again proving that what can be measured can be improved,” said Frank Han, MLPerf Inference working group co-chair. “As workloads and the underlying systems increase in capability, scale, and complexity across a broader variety of form factors and physical configurations, we are clearly seeing a renewed interest in ensuring that inference can take full advantage of hardware, software, and architectural advances – both in the datacenter and at the edge. With performance data from the Inference v6.1 benchmark, customers can better understand the cost-benefit tradeoffs and make informed decisions on how to procure and deploy their AI systems.”

The AI community continues to embrace and invest in MLPerf Inference

The MLPerf Inference 6.1 benchmark received submissions from a record-high 30 participating organizations: AMD, ASUSTeK, Atlas Inference, Cisco, CoreWeave, Crusoe, Dell, Fujitsu Limited, GigaComputing, Google, Hewlett Packard Enterprise, Intel, Inventec Corporation, KRAI, Lambda, MangoBoost, Microsoft Azure, MiTAC, Nebius, NVIDIA, Oracle, Orrick, Quanta Cloud Technology, RedHat, ScitiX, Supermicro, Telecommunications Technology Association, VibeHPC, Wiwynn and individual contributor Naeem Khoshnevis. 

“I would like to welcome our six first-time submitters: Atlas Inference, Crusoe, Orrick Industries LLC, ScitiX, VibeHPC, and individual contributor Naeem Khoshnevis, ” said David Kanter, Head of MLPerf. “The groundswell of participation from the AI community tells us that our work is important and is making a difference to stakeholders.”

In addition, over 50% of submitters used MLPerf’s new API-centric harness, the foundation of the next-generation MLPerf Endpoints benchmark suite, in their submissions. The new harness uses a true client/server architecture over industry-standard APIs to communicate inference queries and results to the system under test, more accurately reflecting real-world datacenter deployment. “We’re excited to see submitters embrace our customer-centric tooling and workloads and partner with us to improve the state of industry-wide benchmarking for AI,” said Kanter. “Moving forward, MLPerf Endpoints will replace Inference in our family of benchmarks for the datacenter, and the quick uptake of our API-centric harness will contribute to making that transition seamless.”

View the results

To view the results for MLPerf Inference v6.1, please visit the Datacenter and Edge benchmark results pages and read details on individual submissions in our supplemental. For an interactive view of datacenter results, access the MLPerf benchmark results dashboard: https://mlcommons.org/visualizer. Additionally, submitters have provided supplemental statements to help contextualize and understand their submissions and results.

About MLCommons

MLCommons is the world’s leader in AI benchmarking. An open engineering consortium supported by over 130 members and affiliates, MLCommons has a proven record of bringing together academia, industry, and civil society to measure and improve AI. The foundation for MLCommons began with the MLPerf benchmarks in 2018, which rapidly grew into a set of industry metrics for measuring machine learning performance and promoting transparency in machine learning techniques. Since then, MLCommons has continued to use collective enMLCommons Sets Participation Record with New MLPerf Inference v6.1 Benchmark Results

For additional information on MLCommons and details on becoming a member, please visit MLCommons.org or email participation@mlcommons.org.

Press Inquiries: contact press@mlcommons.org

The post MLCommons Sets Participation Record with New MLPerf Inference v6.1 Benchmark Results appeared first on MLCommons.

A Retrieval-Augmented Generation (RAG) pipeline ingests and chunks source data, converts it into semantic vector embeddings for storage and retrieval, and then combines relevant retrieved data with a user query to generate an accurate, model-driven response. This benchmark measures this entire pipeline. It is built around two workloads: 

• An ingestion pipeline that builds the vector database from a document corpus. 
• A question-and-answering (QnA) pipeline answers queries over the vector database, iterating retrieve-and-reason steps across multiple hops until it has enough evidence. 

Together they capture what governs a real RAG deployment — multiple models of different roles and sizes, served across an iterative loop — none of which a single-model benchmark can measure.

It is the first multi-component MLPerf inference benchmark to score an entire RAG pipeline end to end. This post covers what makes this workload unique, the dataset and task, the models and metrics, and the reference implementation. 

Why End-to-End RAG?

RAG is increasingly popular and  for many deployments, essential. By grounding answers in documents retrieved at query time, it reduces hallucination, keeps answers up to date, and lets a general model use proprietary or private knowledge it was never trained on — exactly the properties enterprises need.

By nature, RAG is a multi-component system, and its behavior emerges from how those components are composed and served together, not from any one of them alone. Single-model LLM benchmarks can’t capture this: they score one model on one prompt (most often without tokenzier/de-tokenizer involved), so the pipeline-specific behavior that governs a real RAG deployment goes unmeasured, and the optimization opportunities of serving several models together never arise.

It is also a stepping stone toward benchmarking agentic AI. RAG in itself is one of the tools an agent reaches for, and its multi-component, multi-model serving challenges are a first taste of what a full agentic benchmark must measure — laying groundwork that future agentic benchmarks can build on.

And it opens optimization avenues we haven’t seen in prior LLM benchmarks. Because the pipeline runs several models with different roles and sizes — rather than one model on one prompt — a submitter has levers that a single-model benchmark simply doesn’t expose. A few of them:

– Model placement. How the components are mapped onto the system — e.g. the smaller and larger language models on separate accelerators, or the lightweight components on CPU with varying precision levels without compromising E2E Accuracy criteria.

– Co-residency. How to share one device among multiple models — via memory partitioning or hardware-level isolation.

– Scheduling a multi-stage fleet of calls. How to overlap many concurrent tasks as an effective pipeline across heterogeneous accelerators involving macro and micro batching across the pipeline and within each ingredient in the pipeline.

– Prefix caching. The multi-hop context grows each hop, so reusing the shared prefix trades compute-bound prefills for memory-bound. 

– System level optimization/KPI: Effective pipeline implementation across CPU/GPU/NIC/Storage to demonstrate system level KPI mimicking real world deployment scenarios                                                                                                       

Dataset and task selection

The task is multi-hop queries answering over a set of Wikipedia documents, using the FRAMES benchmark:

• 824 queries, each with a ground-truth answer and the Wikipedia article URLs required to answer it.
• 2,515 Wikipedia HTML articles, a frozen snapshot shipped with the benchmark so every submitter indexes the same corpus.
• ~107,000 passages — the articles chunked into 768-character passages with 32 characters overlap for embedding and indexing.

FRAMES was chosen because it is public and factual, with unambiguous ground truth, and its queries span diverse reasoning types — numerical, tabular, temporal, multiple-constraint, and post-processing.

Above all, its queries are multi-hop, which is what makes FRAMES a demanding test of a RAG system. A multi-hop queries can’t be answered from one passage — it requires chaining several facts that never appear together, so every stage has to pull its weight. For example (Figure 1),

> “The person who posted a photo with Rahul Ligma and Daniel Johnson at the headquarters of a social media company claims to have a certain syndrome, despite never receiving a formal diagnosis. Who was this syndrome named after?” → ground truth: Hans Asperger

No single passage holds the answer. The pipeline has to chain three facts across separate articles: who posted that photo (Elon Musk), the syndrome he says he has (Asperger’s), and whom it is named after (Hans Asperger) — reformulating its searches hop by hop until the evidence connects.

Figure 1: Example Multi-hop E2E RAG

The benchmark exercises this dataset through two independent pipelines, each its own MLPerf workload:

Ingestion (`e2e-rag-db`) runs once to build the database; QnA (`e2e-rag-qna`) is the scored end-to-end pipeline. 

Figure 2: Ingestion and QnA Pipeline for E2E RAG

The ingestion pipeline turns the corpus into a searchable vector database, which is a one-time operation. Parsing extracts the article body from each of the 2,515 Wikipedia HTML files shipped with the benchmark, stripping Wikipedia metadata such as references and navigation. Tables and lists are flattened into text row by row rather than dropped. Chunking slices that text into 768-character passages, each tagged with its original Wikipedia URL so it can be traced back to its page. The size was tuned: larger chunks mix in unrelated text, smaller ones split related facts apart. A 32-character overlap keeps facts that land on a boundary intact. Embedding encodes each passage into a 768-dimension vector, and the vectors are indexed into a FAISS HNSW graph for fast approximate similarity search. The result is the vector database that every QnA run queries against.

The QnA pipeline answers a query by looping over that database generated in the Ingestion pipeline (E2E-RAG-DB). Answering a query is not a single model call but a QnA task: the query runs several components through an iterative loop with sub-queries to reach an answer. A query rewriter first decomposes the query into up to three focused sub-queries. Each sub-query is embedded with the same embedder from ingestion, and used to retrieve its most similar passages from the database. The reranker then deduplicates these candidates and reorders them by relevance, keeping the strongest few. A document grader judges each retrieved passage for relevance and keeps only those that help, and a sufficiency checker decides whether the accumulated evidence can answer the query. If not, the query rewriter generates new sub-queries, reformulating and broadening its searches, and the loop repeats for up to five hops. Once the evidence is sufficient (or the hop limit is reached), answer generation produces the final answer grounded in the kept passages, or returns “Unknown” if the evidence falls short.

Model selection

All models can be downloaded from MLCommons-Storage

• intfloat/e5-base-v2 is the embedder. Trained for retrieval with separate query and passage encodings, it fits RAG’s query-to-document matching and performs strongly on standard benchmarks (MTEB, BEIR), all at a compact, widely adopted 110M parameters.
• ColBERTv2.0 is the reranker, using late interaction rather than the typical cross-encoder. Its token-level matching captures finer-grained relevance than compressing a passage into one vector, and generalizes better to new domains, which suits the private document collections common in enterprise use.
• GPT-OSS-120B handles the reasoning-heavy work of rewriting a query into good sub-queries, judging when the evidence is sufficient, and generating the final answer. It is open, and already widely deployed and well-optimized by MLPerf submitters.
• GPT-OSS-20B covers document grading, deciding whether a retrieved passage is relevant. This is a high-volume classification task, and pairing a smaller model here reflects real deployments that route easy work to right-sized models rather than sending everything to the largest one. It also shares the gpt-oss architecture with the 120B, so supporting it takes little extra effort.
• Llama-3.1-8B-Instruct is the judge, deliberately from a different family than the GPT-OSS models it grades to avoid self-preference bias. It scores answers after the accuracy run is finished.

Performance metrics

MLPerf Inference traditionally defines two serving scenarios, Offline and Server. This first instantiation focuses on Offline, where all requests are available at once and can be scheduled for maximum throughput; Server is left for future rounds. Each pipeline reports its own throughput metric: documents per second for ingestion, and tasks per second for QnA.

For ingestion, documents per second is the natural unit. Each document flows through parse, chunk, embed, and index, and the document is the fixed unit of work every submitter starts from, which makes documents per second directly comparable across systems. More details can be found in inference rules.                                                                                                                                                  

For QnA, the usual language-model metric, tokens per second, doesn’t fit: the pipeline runs two different-sized language models alongside non-language-model components, so a single token rate can’t represent the whole system. A per-hop metric like hops per second is hard to interpret, since a QnA task takes a variable number of hops. Tasks per second avoids both problems by measuring the unit that matters, one query answered end to end. It looks small next to other benchmarks because a single query can run up to five hops and a dozen or more language-model calls.

The QnA pipeline is also a complex, dynamic workload that is hard to make reproducible, on top of the inherent non-determinism of language-model generation. For performance runs, recorded reference inputs are provided for each stage of each hop, fixing the number of hops and the documents retrieved. Outputs are still generated normally but discarded, so the pipeline does real work while run-to-run variation stays minimal.

Accuracy metrics

Retrieval quality (against the required Wikipedia URLs) is not an official metric. It is a database-integrity check, used to confirm that a submitter’s independently built vector database behaves like the reference so that everyone is answering from the same corpus.                                                                                                            

Reference values over the full 824-query set:

A submission is valid if its answer accuracy is at least 97% of the reference accuracy.

Answer accuracy also varies with the kind of reasoning a query demands:

Answer accuracy also varies with the kind of reasoning a query demands. The breakdown below is insights only, and a query can carry more than one reasoning type.

Multiple-constraint queries score highest, since they mainly require gathering discrete facts, exactly what dense retrieval and a language model are good at. Numerical and tabular queries are hardest: the answer depends on pulling exact figures out of prose or tables and then computing over them, so a single misread number or a chunk boundary that splits a table fails the whole query. Temporal and post-processing queries sit in between, needing date arithmetic or a final transformation on top of correct retrieval.

Closing these gaps points naturally toward a more agentic pipeline: a knowledge graph or structured index the system can query for entities and relations, table-aware parsing and chunking so figures survive ingestion intact, and equipping the answer step with tools such as a code interpreter to handle arithmetic and unit conversion. Each is a step from today’s fixed retrieve-and-read loop toward an agent that chooses how to find and process what it needs.                                                                                                                                                   

Compliance

E2E-RAG-QnA requires a single compliance test, TEST09 (output-token-length verification) which re-runs the workload in LoadGen performance mode with the audit.config and compares the mean output token length of the answer-generator response against the reference. The only component visible to LoadGen against the reference implementation’s mean OSL of 273.81 tokens with a ±10% band (246.43 – 301.19). This guards against a submission cutting work by generating systematically shorter answers in performance mode than the reference does.

The check is a single aggregate on one pipeline stage, it does not provide any guard to retrieval, reranking, or intermediate query/sufficiency LLM calls. Future revisions could add distribution-level or per-component compliance coverage.

Optimization Opportunities

E2E RAG pipeline is intentionally designed to expose multiple layers of optimization opportunities across the full inference stack from the Vector DB Generation following the rules algorithmic criteria to inference serving framework, with dense and sparse kernels to KV-cache-aware scheduling. The reference implementation provides a clear but unoptimized baseline in several of these dimensions, leaving room for submitters to explore system and kernel improvements while preserving accuracy. 

• CPU Pipeline: MLPerf Loadgen provides the entire 824 queries/tasks at one-shot (in this offline scenario) to the serving framework. This provides a great opportunity for the submitters to explore how to spawn the tasks across several cores of the CPU with appropriate affinity to their corresponding accelerators with memory placement strategy for the E2E pipeline and within each of the ingredients in the pipeline. 
• GPU Partitioning with KV Cache Balancing: Effective Multi-model placement with low latency within accelerators and across accelerators to saturate all the ingredient accelerators participating in compute/comms/storage. This includes efficient low-latency high throughput CPU-GPU interaction, GPU-GPU/Accelerator Interaction/Compute vs. COMMS Balancing across the macro & micro batching to keep a tight pipeline. 
• Multi-model with different precisions: All the selected models can be operated in different precision as long as the E2E Accuracy criteria is met. This provides greater innovation opportunity to selectively choose the right precision that can provide best performance at system level. 

Conclusion

This is MLPerf’s first benchmark to measure an entire RAG pipeline end to end, rather than a single model in isolation. It captures what actually governs a deployed RAG system: multiple models of different roles and sizes, served together across an iterative, multi-hop loop.

This blazes the trail for the future Agentic RAG.  The retrieve-reason-decide loop at its core is the same loop that drives agents, and the natural next steps — tool use, structured retrieval, more autonomy in how the system finds and processes information — move it steadily toward a full agentic benchmark. We look forward to evolving it with the community with EndPoints/Agentic Intercept in near future

References

• Getting started with E2E-RAG reference implementation.
• Reference implementation. 
• MLPerf Inference Benchmark Data Download 
• A new GPT-OSS benchmark and DeepSeek R1 updates for latency-optimized reasoning – MLCommons 
• MLPerf Inference Rules for E2E-RAG
• FRAMES Paper 

Acknowledgements

We thank MLCommons E2E RAG Taskforce participants and the MLPerf Inference Working Group, MLCommons Platform Engineering team for their feedback, support and guidance in developing this benchmark. Our sincere appreciation to all the contributors across the industry & academia that was very instrumental to weigh-in evaluation of various proposals! We are grateful to everyone to make this first unique benchmark definition happen with collaboration across various time-zones.

We also thank the Multi-Turn Taskforce, whose ideas helped us address the non-determinism inherent in multi-hop benchmarking.

We are especially grateful to Satyapriya Krishna, one of the authors of the FRAMES dataset, who joined our discussions to offer clarifications and guidance. 

The post Introducing the MLPerf End-to-End RAG Inference Benchmark appeared first on MLCommons.

MLPerf was originally designed to help a relatively small number of cloud providers and downstream system builders inform their AI hardware purchasing decisions. Today, procuring inference compute is a salient business decision for companies of all sizes – and it means evaluating options across neoclouds, cloud providers, and managed services simultaneously. These buyers need reliable, comparable, and independent performance benchmarks to inform their purchasing decisions, and these benchmarks must be dynamic enough to keep pace with an industry that launches new models weekly. 

MLPerf Endpoints is designed to meet this more diverse and complex demand for benchmarks to inform AI system procurement decisions. Four key principles are guiding our development of enterprise-buyer-centric benchmarks:  

1. Current: Results must keep pace with the market. Buyers can’t wait months for a benchmark round to include new hardware or models. 
2. Comprehensive: Buyers need benchmarks that cover the many competing inference providers, systems, and workloads available for purchase. 
3. Comparable: Buyers need to see apples-to-apples results across vendors, normalized for cost or power, to inform procurement decisions.
4. Commentary: To support buyers in these decisions, Endpoints provides additional context for benchmark results through visualizations, data filtering, and analysis. 

MLPerf Endpoints v0.7 is a foundation release with initial results from Coreweave, Google, Intel, KRAI, and Nvidia. We want to congratulate these members on their excellent results, spanning several orders of magnitude in performance across 3 benchmarks. This is the infrastructure on which we are building a more dynamic, comprehensive, and comparable inference benchmark suite for the data center. Endpoints currently supports automated submission pipelines, continuous review tooling, and dynamic visualization of results that you can see at mlcommons.endpoints.com. We are also evolving our benchmarking rules towards more buyer-centric benchmarks.

Later this year, we will deliver MLPerf Endpoints v1.0 with more buyer-centric rules, normalization, and an expanded set of benchmarks including agentic workloads and then open the rolling submission process to our broader membership. The rolling submission process ensures Endpoints will provide current and up-to-date results that move at the pace of the market. 

We’re grateful to our 30+ supporters who have helped guide the development of MLPerf Endpoints, including AMD, Argonne National Laboratory, Broadcom, Core 42, Dell, HPE, Lambda, Oracle,  and Red Hat. Their partnership has been invaluable in stress-testing our rules, processes, and improving our infrastructure. With this release, we are just getting started. 

If you’re a system or service provider, now is the time to get involved, help shape the rules, and plan your submission at a time of your choosing. Complete this form to let us know you’re interested in submitting, and what hardware you want to submit on. The community-developed rules are actively being refined for the 1.0 release later this year, and early participants in this effort will shape Endpoints’ direction. If you’re an enterprise buyer, Endpoints is being built for you, and we want to know how we can best support you and your organization. Let us know by emailing alejandro@mlcommons.org.

We deeply value the trust our members place in us as the benchmark of record for inference. We are excited to build this future of MLPerf with all of you. 

[1] A. Tschand, A. T. R. Rajan, S. Idgunji, et al., “MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from µWatts to MWatts for Sustainable AI,” in 2025 IEEE International Symposium on High-Performance Computer Architecture (HPCA), 2025. arXiv:2410.12032.

[2] D. Kanter, M. Ahmad, H. Kassa, and S. Rishab, “MLPerf Training v4.1 Results — Press Briefing,” MLCommons, Nov. 13, 2024. [Online]. Available: https://docs.google.com/presentation/d/1KSIJBvIV9OcswF1mVbhGRN0nUWbSgbYSoxu6dHCwajM/

The post MLPerf Endpoints v0.7: A Foundation Release appeared first on MLCommons.

For this round we debut the Qwen3.6-27B model (released April 22, 2026), served as a Q4_K_M GGUF quantization (reference implementation’s format – submitters may use any allowed quantization that meets the accuracy threshold) on a single edge accelerator, paired with two complementary workloads: the Berkeley Function Calling Leaderboard v4 (BFCL v4) for a deterministic, judge-free accuracy gate, and a recorded agentic-coding replay for single-stream performance. Together they capture the messy reality of on-device agents: long, growing tool calling conversations that must fit within a hard context wall and feel responsive to a single interactive user. 

The submission deadline is July 31, 2026. We invite all hardware vendors, edge-device makers,  and inference-system experts to submit results and help raise the bar for on-device agentic inference. 

Why Agentic AI at the Edge 

The shift from single-shot text generation to multi-turn agentic workloads is one of the most consequential trends in applied AI. A real agentic session is not one prompt and one answer – it is a trajectory where each turn appends tool outputs and model responses to a growing conversation history, the model alternates between short tool calls and longer reasoning, and every turn depends strictly on the one before it. 

The datacenter MLPerf Agentic benchmark (coming up in September) measures this at scale – large mixture-of-experts models, long-horizon trajectories whose cumulative multi-turn context runs to 100K+ tokens, and a concurrency sweep that produces a per-GPU Pareto frontier. The  edge regime is the opposite problem, and it is what this benchmark targets: 

• A fixed memory and power budget: A single small accelerator (with a limited amount of memory) must hold weights, KV cache, and activations. Full-precision frontier models do not fit, so the model is quantized.

• A single in-flight request: Edge serving is dominated by one interactive user, not thousands of concurrent sessions.  

• A fixed served context window. The reference fixes the served context at 32K tokens – a controlled benchmark parameter, not a device ceiling – so long trajectories can exhaust it. This makes context growth and truncation first-order, measured effects of the agentic workload rather than something that varies with per-device memory. 

• Single-user latency, not aggregate throughput. On a single-slot edge device, throughput and latency are reciprocal to each other, so the metrics that matter are per-turn TTFT,  TPOT, and end-to-end turn latency on one device – not the aggregate QPS/token throughput a multi-stream datacenter server optimizes for. 

This benchmark inherits the multi-turn agentic methodology – terminology, deterministic replay,  the JSONL dataset schema, and the inline accuracy check – from the datacenter agentic specification and specializes it to the edge: an edge model and quantization, a single-stream load pattern, latency-centric metrics, and a statistically robust accuracy gate. 

Model Selection 

The reference model is Qwen3.6-27B (https://qwen.ai/blog?id=qwen3.6-27b), served with  llama.cpp (commit cfff1fc) as a Q4_K_M GGUF quantization (https://huggingface.co/unsloth/Qwen3.6-27B-GGUF). Qwen3.6 was selected because it is an open (Apache 2.0), capable tool-calling model with native MTP speculative-decoding head that punches above its size – a dense 27B that Alibaba reports (https://qwen.ai/blog?id=qwen3.6- 27b) beats the prior 397B-MoE flagship on major coding benchmarks (77.2% SWE-bench  Verified), yet deploys easily: official weights on Hugging Face and ModelScope plus community  GGUF builds (e.g. unsloth) that run directly under llama.cpp. 

Quantization is the defining edge choice. Q4_K_M is a 4-bit K-quant “Medium” tier: weights are stored at 4 bits/value (a 4× memory reduction versus BF16), attention blocks are kept at higher precision than the feed-forward blocks, and importance-weighted (imatrix) calibration preserves the highest-impact weights. The result fits the 27B model in ~16.5 GB of VRAM  (versus ~54 GB for BF16) at a ~2–5% accuracy cost – the difference between “runs on an edge  GPU” and “does not.” The exact GGUF used for the reference runs is Qwen3.6-27B-Q4_K_M.gguf from unsloth/Qwen3.6-27B-GGUF. 

Reference sampling parameters used are listed in the table below:

max_new_tokens 1024 

Reasoning is deliberately disabled on this tool-calling workload; it lowers single-turn accuracy while costing far more time (see Accuracy Metric section). The benchmark has been validated across a range of edge accelerators from multiple vendors, confirming that the workload and methodology are portable. 

The Benchmark Task 

The benchmark uses two datasets with distinct roles. 

Accuracy dataset: The Berkeley Function Calling Leaderboard v4 (BFCL v4) tests whether the model calls functions correctly, deterministically, and without an LLM judge. It spans single-turn requests (one prompt → one structured tool call) in three categories – non_live and live (graded by AST match against gold labels) and hallucination (a binary check that the model refuses to call a function when the available tools are irrelevant) – and optional multi-turn agentic conversations against in-process Python simulators. The categories are drawn from the public gorilla-llm/gorilla-eval-set (auto-downloaded at runtime). The scored gate is the single turn set only, per-category sampled to a stable ~995-sample point estimate: non_live 72%  (~712 samples), live17% (~171), and hallucination11% (~112), with a subset_floor of 25 so any subset of ≤ 25 entries is taken in full. 

Performance dataset. A subset of the MLPerf Agentic benchmark – recorded multi-turn agentic coding trajectories (SWE-bench-style tasks drawn from real repositories e.g., astropy), replayed deterministically as a performance workload, and scaled for single-device edge serving. The reference set comprises 20 conversations / 1,007 turns, sized so that no trajectory overflows the 32K-token served context (peak input length ~23.5K tokens). Because nothing overflows, every turn completes: the run is valid with zero dropped turns, and a single pass at concurrency 1 (one request in flight at a time) completes in a reasonable time on a single edge device. The dataset also serves as its own ground truth – each trajectory’s recorded tool calls drive a zero-cost inline accuracy check (multiset IOU of the executed calls) that runs during the latency measurement,  so correctness and latency come from the same pass. 

Example Request and Response 

A single-turn function-calling request is sent to the model’s OpenAI-compatible  /v1/chat/completions endpoint, carrying the available tool schemas:

}, 

 { 

 "role": "user", 

 "content": "What is the current temperature in San Francisco, in Celsius?" 

 } 

 ], 

 "tools": [ 

 { 

 "type": "function", 

 "function": { 

 "name": "get_current_weather", 

 "description": "Get the current weather for a given location.", 

 "parameters": { 

 "type": "object", 

 "properties": { 

 "location": { 

 "type": "string", 

 "description": "City and state, e.g. 'San Francisco, CA'" 

 }, 

 "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] } 

 }, 

 "required": ["location", "unit"] 

 } 

 } 

 } 

 ] 

} 

The model responds with a structured tool call: 

{ 

 "choices": [ 

 { 

 "message": { 

 "role": "assistant", 

 "tool_calls": [ 

 { 

 "type": "function",

 "function": { 

 "name": "get_current_weather", 

 "arguments": "{\"location\": \"San Francisco, CA\", \"unit\": \"celsius\"}"  } 

 } 

 ] 

 }, 

 "finish_reason": "tool_calls" 

 } 

 ] 

}

BFCL v4’s AST checker compares the predicted call – function name and each argument – against the gold label, so scoring is exact and reproducible. In the agentic-coding performance  workload, the same mechanism applies to multi-turn trajectories: the model emits bash tool  calls, the runner records them, and the inline checker scores the multiset intersection-over union (IoU) of the canonicalized shell executables against the recorded ground-truth trajectory.

Performance Metrics 

The edge benchmark reports single-stream, per-turn latency on a single accelerator rather than a per-GPU throughput frontier. A single simulated user replays trajectories closed-loop: send a turn, wait for the full streamed response, pause for the pre-recorded inter-turn delay, send the next turn – exactly one in-flight request at all times (target_concurrency = 1, one worker, one connection), matching the single-slot llama.cpp server. The reported metrics are TTFT (time to first token), TPOT (time per output token), end-to-end latency per turn, and ISL/OSL  distributions, as p50/p90/p99/max. 

Accuracy Metric 

Accuracy is the scored BFCL v4 single-turn gate, chosen for statistical robustness: the full set of single-turn categories is sampled to ~995 samples, completing in a reasonable time on an edge device.  Single-turn was chosen because its score is stable at large n and across devices, whereas a small multi-turn gate is brittle – an n=57 multi-turn gate carries a ~±11.6 pp 95% confidence interval,  and Q4 quantization-kernel differences cause per-entry verdicts to flip across devices. Multi-turn is retained as an optional, non-scored reference artifact. 

Reference single-turn accuracy (Qwen3.6-27B-Q4_K_M, seed 42, temperature 0; hardware independent, and identical across two seeded runs with a freshly restarted server each pass): 

There are two gated metrics: Overall, the sample-weighted accuracy across all ~995 single-turn samples, and Normalized, the category-balanced mean of the three category scores (so each category counts equally regardless of size). The per-category rows are components, not gates. 

The pass/fail criterion is a 3% one-sided band applied to both gated metrics: a submission passes only if its Overall and Normalized single-turn scores are each ≥ 0.97 × their respective references, with no upper bound (a higher score never fails).  

Reasoning off is the reference submission config. On this tool-calling workload, enabling server-side reasoning gives no accuracy benefit and costs far more wall-clock: on the accuracy gate, reasoning-off scores 86.23% versus reasoning-on’s 78.19% – an 8-point drop – and on the performance replay, reasoning-on moves inline IoU by only +0.004 (0.6374 vs 0.6335) while taking ~60% longer run-time.  

Join the Benchmark 

This is your opportunity to show how your edge hardware and software stack runs real multi-turn tool-calling workloads – on both accuracy and latency. Submissions for MLPerf Inference v6.1 are open to MLCommons members. Join MLCommons to submit your results by July 31, 2026.

• Model: Qwen/Qwen3.6-27B, served as a Q4_K_M GGUF – Qwen3.6-27B-Q4_K_M.gguf from  unsloth/Qwen3.6-27B-GGUF. 

• Accuracy dataset: gorilla-llm/gorilla-eval-set (BFCL v4 single-turn), auto-downloaded at  runtime. 

• Reference implementation: mlcommons/endpoints – examples/10_Edge_Agentic_Example/.  A single mandated config (online_edge_full_run.yaml) runs performance and accuracy back-to-back against the same server; –accuracy-only runs the gate alone.

• Submission deadline: July 31, 2026. 

The reference implementation drives any OpenAI-compatible Chat Completions endpoint, so you can benchmark arbitrary inference systems – not just the llama.cpp reference server.  Anyone with an edge accelerator can reproduce the reference accuracy and latency numbers by following the example README. We look forward to your submissions! 

Acknowledgements 

We thank the MLCommons Edge LLM Taskforce and the MLPerf Inference Working Group for their feedback and guidance in developing this benchmark.

The post Call for Submission: Edge Agentic Inference Benchmark for MLPerf Inference v6.1 appeared first on MLCommons.

The MLPerf Inference benchmark suite must evolve alongside AI deployment patterns. Early inference benchmarks focused on image classification, object detection, speech recognition, recommendation, and single-turn language generation. Those workloads remain important, but they no longer cover one of the fastest-growing ways large language models are used in production: multi-turn agentic inference.

For example, a coding assistant is far more complex than a single query. The agent will read an issue, inspect files, run commands, observe failures, edit code, and iterate potentially many times. Similarly, a workflow agent gathers customer information, calls tools, interprets results, asks follow-up questions, and continues until the task is resolved. In both cases, the workload is a trajectory: a sequence of dependent turns where each request includes the conversation history up to that point.

This changes the serving problem in four concrete ways:

• Context grows across the trajectory, so prefill and KV-cache pressure increase over time.
• KV-cache reuse is now a critical serving optimization for performance and efficiency.
• Output lengths vary widely, from compact tool calls to long reasoning traces.
• Turn dependencies make throughput a measure of closed-loop progress, not independent request rate.

Figure 1. Illustration of an agentic inference scenario involving multiple turns.

The Agentic Inference benchmark adds this class of workload to the MLPerf Endpoints framework. It keeps the general MLPerf measurement principles intact while defining the workload-specific pieces: model choice, dataset composition, multi-turn load generation, output validation, and constraints for optimizations such as prefix caching and speculative decoding.

Terminology

• Turn: one client-issued user or tool request and the model response generated for that request.
• Trajectory: the ordered sequence of turns for one task or simulated user; later turns include the accumulated conversation history.

Model selection

The benchmark needs long-context, thinking-capable LLMs that stress the serving behavior of agentic applications: growing context, KV-cache reuse, variable output lengths, and strict turn dependencies. For this benchmark, we chose Kimi K3 and Qwen3.6-35B-A3B. Kimi K3 brings state-of-the-art coding capability and a large model profile representative of leading agentic systems; Qwen3.6-35B-A3B is more compact and brings a new Gated DeltaNet (GDN) architecture and exceptional coding performance for its size. This gives the benchmark coverage across distinct serving behaviors, architectural choices, and speculative-decoding paths. Both models are evaluated with the same methodology and dataset. To avoid ambiguity, each model produces its own result; the two models are not run together or combined into a single score.

Table 1. Model metadata and benchmark settings

Dataset and task selection

The benchmark dataset combines two agentic domains that exercise different infrastructure bottlenecks. Together, they contain 613 multi-turn trajectories: 113 agentic coding trajectories and 500 agentic workflow trajectories. Across the reference dataset, these trajectories contain 30,335 client-issued turns and 30,328 assistant turns. By using real collected traces for benchmarking, the workload simulates real-world behavior in the serving stack, including how speculative decoding and expert-rank balancing behave under realistic multi-turn traffic.

Table 2. Dataset composition and mean token statistics

The agentic coding traces come from the DeepSWE dataset by DataCurve (datacurve.ai), which contains software engineering tasks built around repository-level bugs and feature requests. A typical trace begins with a user request describing an issue, followed by an assistant that investigates the repository through a series of bash commands. The agent searches files, reads code, runs tests, observes errors, edits the implementation, and iterates. These traces are deep: the median trajectory contains dozens of turns, and the conversation history grows steadily as command outputs, file contents, and test logs accumulate.

The Workato agentic workflow traces come from enterprise customer-support and orchestration scenarios. They represent interactions where a simulated customer asks for help and the agent uses tools to retrieve orders, track shipments, check policies, escalate cases, or resolve account questions. These trajectories are usually shallower than the coding traces, but they include a much larger shared system prompt with many tool definitions and business rules. The agentic workflow traces were provided by Workato, the agentic control and execution plane for the enterprise. The data are synthetic traces modeled on Workato’s production experience orchestrating customer support agents for enterprise customers.

The two domains are intentionally different:

• Coding traces grow aggressively over many turns.
• Workflow traces begin with a large common prefix and grow more slowly.
• Coding stresses KV-cache capacity and long-context scheduling.
• Workflow stresses shared-prefix reuse and routing locality.
• The combined workload prevents systems from optimizing for only one agentic traffic shape and stresses context-aware routing.

Client Design

We introduced multi-turn support to MLPerf Endpoints so the benchmark can drive full end-to-end agentic workloads against standard serving stacks. Submitters simply need to spin up an OpenAI-compatible endpoint using vLLM, SGLang, TensorRT-LLM, or another serving framework, then point the client at that endpoint for full end-to-end benchmarking.

The client handles the multi-turn behavior:

• Closed-loop replay: one active conversation issues one turn at a time and waits for the complete model response before the next turn.
• Target concurrency: the load generator controls the number of active users or conversations without breaking turn dependencies.
• Inter-turn delay: dataset-provided waits before tool or user turns preserve realistic pacing without counting delay as model serving latency.
• Conversation-aware routing: a stable X-Session-ID header is sent for each trajectory so routers can preserve KV-cache locality.
• Cache salting: deterministic salt markers around the system prompt allow valid within-trajectory reuse while blocking invalid cross-trajectory reuse to ensure the benchmark is representative of production workloads.
• Deterministic prompt reconstruction: future prompts are built from the pre-recorded dataset rather than from live model output, which keeps performance runs reproducible while still measuring generated outputs.
• Generated token cache clearing: In order to enable fair benchmarking of all platforms, the generated tokens are cleared out of the KV cache by introducing a whitespace character – ensuring that the performance is independent of the system that was used for generating the traces.

Performance metrics

The primary performance chart is a Pareto curve where each point is a fixed-concurrency benchmark result. Submitters need to submit a full Pareto curve as defined in the MLPerf Endpoints rules.

• Y-axis: output tokens per second per system, measuring aggregate serving throughput.
• X-axis: E2E Interactivity, computed as sum(output tokens per turn) / sum(E2E turn time in seconds), where each turn is measured from request issuance through receipt of the complete response.

The two axes represent complementary views of the serving system. The x-axis, E2E Interactivity, output tokens per second per user, captures how quickly an individual agent progresses through its task; moving right means faster task completion. The y-axis, output tokens per second per system, captures the total work completed across all active agents; moving up means greater aggregate throughput and higher system utilization. As concurrency increases, the system may complete more total work while each individual agent progresses more slowly. The Pareto curve makes this tradeoff visible.

Figure 2. Chart illustrating agentic inference performance

Accuracy metrics

Accuracy is enforced as a three-level hierarchy so the Pareto curve measures usable agentic progress rather than responses that are shorter, lower quality, or easier to serve.

• OSL mean value: prevents systems from gaining throughput by truncating responses or shifting the answer-length distribution. The check requires generated outputs to preserve the expected mean output sequence length for the workload.
• Inline accuracy: catches behavior drift during the exact performance configuration being reported. It operates on the outputs produced by the performance run itself and compares them against ground-truth values preserved in the dataset, including coding tool-call action checks and workflow intent-code checks, so no separate inference pass is required.
• Standalone accuracy: verifies task-level capability beyond turn-level similarity. The standalone evaluation will use 200 tasks from the SWE-bench Verified dataset to check whether the submitted model configuration can solve representative tasks end to end.

All three levels must be run for every point on the Pareto curve, and each score must meet a pre-defined threshold. Thresholds are defined separately for each level and for both Kimi K3 and Qwen3.6-35B-A3B.

Table 3. Tentative accuracy values, subject to change.

Reference implementation

Submitters only need to spin up an OpenAI-compatible server using vLLM, SGLang, TensorRT-LLM, or another serving framework, then point the MLPerf Endpoints client at that endpoint to run the full benchmark. The MLPerf Endpoints Agentic Inference README includes the example configs, command lines, and setup steps.

Conclusion

Agentic inference is one of the fastest growing applications of AI today and it’s far more complicated than existing single-turn text generation. Agentic workflows have serving patterns with unique constraints: growing context, strict turn dependencies, tool-mediated delays, shared prefixes, long-tail trajectories, and output quality checks that must run on the same configuration used for performance.

The Agentic Inference benchmark brings those constraints into MLPerf in a reproducible form. By combining deep coding trajectories with shared-prefix workflow trajectories, it measures whether serving systems can make progress for real multi-turn users while still using hardware efficiently.

As agentic applications become a larger share of production LLM traffic, the industry needs benchmarks that measure the systems work that actually matters: preserving cache locality, scheduling long contexts, maintaining output quality, and balancing aggregate throughput with per-user progress. This benchmark is a step toward that measurement standard. If your team is building serving infrastructure for multi-turn agents, join MLCommons to access the reference implementation and submit results

References

• MLPerf Endpoints repository.
• MLPerf Endpoints Inference Rules and Pareto collection methodology.
• MLPerf Endpoints Agentic Inference reference implementation.
• SWE-bench Verified benchmark.
• Workato agentic workflow and enterprise task definitions.
• DeepSWE Dataset by Datacurve.

The post Agentic Inference for MLPerf Inference appeared first on MLCommons.

Five of the eleven datacenter tests in MLPerf Inference v6.0 are new or updated, and the release also includes a new object-detection test for edge systems. The major changes include:

●      A new, open-weight large-language model benchmark based on GPT-OSS 120B that can be used for mathematics, scientific reasoning, and coding;

●      An expanded DeepSeek-R1 advanced-reasoning benchmark, including an interactive scenario that permits speculative decoding;

●      DLRMv3, the third generation of our recommender benchmark and now the first sequential recommendation benchmark test in the suite, which is thoroughly modernized based on generous engineering contributions from Meta, a world leader in recommender systems;

●      The suite’s first text-to-video generation benchmark;

●      A new vision-language model (VLM) benchmark that transforms unstructured multimodal data from Shopify’s extensive product Catalog into structured metadata;

●      An upgraded single-shot object detection benchmark for edge scenarios based on Ultralytics’ YOLOv11 Large model.

“This is the most significant revision of the Inference benchmark suite that we’ve ever done,” said Frank Han, Technical Staff, Systems Development Engineering at Dell Technologies and MLPerf Inference Working Group Co-chair. “The decision to update so many benchmarks in this round was prompted by the extraordinary enthusiasm and collaboration from our members, who contributed an unprecedented amount of engineering effort and IP toward building new inference benchmarks. Adding these new tests allows MLPerf Inference to better keep pace with the breakneck pace of evolution in AI models and techniques so that our benchmarks are relevant and representative of real-world deployments.”

The open-source MLPerf Inference benchmark suite measures system performance in an architecture-neutral, representative, and reproducible manner. The goal is to create a level playing field for competition that drives innovation, performance, and energy efficiency for the entire industry. The published results provide critical technical information for customers who are procuring and tuning AI systems. 

“We thank Meta, Shopify and Ultralytics for their substantial collaboration with us in making these changes to the MLPerf Inference benchmark suite and for contributing their datasets, task definitions and expertise,” said Miro Hodak, Senior Member of Technical Staff at AMD and MLPerf Inference Working Group Co-chair. “These partnerships were essential in ensuring that the tests include scenarios and workloads that represent the current state of the industry.”

“MLPerf Inference benchmarks play a vital role in driving transparency and accountability across the AI industry,” said Glenn Jocher, CEO & Founder of Ultralytics. “At Ultralytics, rigorous, reproducible benchmarking is central to how we develop and validate our Ultralytics YOLO models — ensuring developers and organizations can make informed decisions about real-world performance. We’re proud to be part of an ecosystem that holds the entire field to a higher standard.”

“Commerce is one of the most complex domains in AI, yet researchers rarely have data that reflects that complexity,” said Kshetrajna Raghavan, Principal Engineer, Applied ML at Shopify. “Shopify is uniquely positioned to address this, sitting at the intersection of millions of merchants and billions of products. Sharing this taxonomy allows the whole field to evolve.”

New tools for submitters and consumers

With Inference 6.0, submitters have the option to use a newly available harness to complete benchmark tests. The new system, LoadGen++, allows LLMs to run with a serving-style software stack, which is familiar from typical deployments today. “LoadGen++ is a significant upgrade from its predecessor, and represents an important investment by MLCommons that will allow us to stay nimble as we continue to produce benchmark tests that track the state of the art,” said Han.

In addition, the Inference 6.0 results can be viewed in a new online dashboard on the MLCommons site. The dashboard brings new levels of interactivity to viewing results, including advanced filtering and customized performance graphs.

Large-scale, multi-node systems gaining attention 

The submissions to Inference 6.0 demonstrate that technology providers want to showcase the performance of scaled-up, multi-node systems running real-world inference workloads. This round recorded a new high for multi-node system submissions, a 30% increase over the Inference 5.1 benchmark six months ago. Moreover, 10% of all of the submitted systems in Inference 6.0 had more than ten nodes, compared to only 2% in the previous round. The largest system submitted in Inference 6.0 featured 72 nodes and 288 accelerators, quadrupling the number of nodes in the largest system in the previous round. 

“As more AI applications have moved into production and wide availability, the demand for large-scale, high-performance systems to run them has grown,” says Hodak. “At the same time, multi-node systems bring a unique set of technical challenges beyond those of single-node systems, requiring configuration and optimization of system architectures, network interconnects, data storage, and software layers. Stakeholders are eagerly stepping up to meet these challenges and run inference workloads at scale.”

The AI community continues to embrace and invest in MLPerf Inference

The MLPerf Inference 6.0 benchmark received submissions from a total of 24 participating organizations: AMD, ASUSTeK, Cisco, CoreWeave, Dell, GATEOverflow, GigaComputing, Google, Hewlett Packard Enterprise, Intel, Inventec Corporation, KRAI, Lambda, Lenovo, MangoBoost, MiTAC, Nebius, Netweb Technologies India Limited, NVIDIA, Oracle, Quanta Cloud Technology, Red Hat, Stevens Institute of Technology, and Supermicro. 

“I would like to welcome our first-time submitters, Inventec Corporation, Netweb Technologies India Limited, and Stevens Institute of Technology,” said Han. “The AI ecosystem is large and diverse, and it continues to grow and evolve rapidly. On behalf of MLCommons, I want to also thank our members, our contributors, and our partners including Meta, Shopify, and Ultralytics, for collaborating with us to build and shepherding forward the most comprehensive and relevant performance benchmark suite for AI inference. Together, we are ensuring that stakeholders in our community have valuable, real-world information that helps them to make better decisions.”

View the results

To view the results for MLPerf Inference v6.0, please visit the Datacenter and Edge benchmark results pages. For an interactive view of datacenter results, access the new benchmark results dashboard: https://mlcommons.org/visualizer. 

About MLCommons

MLCommons is the world’s leader in AI benchmarking. An open engineering consortium supported by over 130 members and affiliates, MLCommons has a proven record of bringing together academia, industry, and civil society to measure and improve AI. The foundation for MLCommons began with the MLPerf benchmarks in 2018, which rapidly grew into a set of industry metrics for measuring machine learning performance and promoting transparency in machine learning techniques. Since then, MLCommons has continued to use collective engineering to build the benchmarks and metrics required for better AI – ultimately helping to evaluate and improve the accuracy, safety, speed, and efficiency of AI technologies.

For additional information on MLCommons and details on becoming a member, please visit MLCommons.org or email participation@mlcommons.org.

Press Inquiries: contact press@mlcommons.org

The post MLCommons Releases New MLPerf Inference v6.0 Benchmark Results appeared first on MLCommons.

The MLPerf® Inference v6.0 release marks a significant expansion in our coverage of the open-weight large language model (LLM) landscape. As the industry moves toward more specialized and capable open models, the benchmarks must evolve to reflect these shifts in deployment strategies and model architectures.

In this round, the Reasoning LLM task force introduces two major additions to the suite:

• GPT-OSS 120B: a new benchmark based on a popular open-source, high-capability model which excels at mathematics, scientific reasoning, and coding. It features a large-scale mixture-of-experts (MOE) architecture of 117B total parameters. 
• DeepSeek-R1 interactive scenario: Building upon the existing DeepSeek-R1 benchmark, we add a low-latency-constrained interactive workload targeting real-time reasoning applications. This workload also features the first standard for speculative decoding in MLPerf.

A new benchmark with GPT-OSS 120B

GPT-OSS 120B is a popular new open-source, high-capability model, featuring a mixture-of-experts (MoE) architecture with 117B total parameters (5.1B active per token). This model natively supports configurable reasoning effort levels and is deployed across a diverse range of complex, knowledge-intensive workflows including advanced coding, competition mathematics, and graduate-level scientific logic. 

Frontier models are often deployed across a wide spectrum of production workloads—from fast, routine requests to complex, multi-step problem solving. To accurately reflect this duality, we introduced a split-dataset strategy: 

• Performance dataset of routine, low-effort usage tasks (e.g., summarization)
• Accuracy dataset of difficult reasoning problems across coding, scientific knowledge and math.

Across both modes, all inputs utilize the OpenAI Harmony chat format, allowing us to natively control the model’s reasoning effort (low, medium, or high) directly through system prompts.

Dataset selection for GPT-OSS

For the first time in MLPerf inference benchmark we have decided to separate performance and accuracy datasets. In all existing MLPerf Inference benchmarks, a single dataset is used for both. However, separating the two brings flexibility that allows agility in benchmark definition for this workload and also in the future. Additionally, the two tasks are different, and the separation allows the use of an optimal dataset for each task. 

A key consideration in the task force deliberation has been to ensure consistency between performance and accuracy runs. This issue was resolved by adding a new compliance test that verifies accuracy while running in the performance mode. 

Accuracy mode (high reasoning effort)

To ensure the benchmark sticks to a fair accuracy baseline, we curated a composite dataset requiring high reasoning effort.

• Max output length: 32,768 tokens.
• Evaluation strategy: Pass @ 1 with k repeats.
• Datasets:

• AIME 2024: Advanced mathematics problems. Metric: Exact Match.
• LiveCodeBench v6: Real-time coding tasks. Metric: Pass/Fail.
• GPQA-Diamond: Graduate-level science QA. Metric: Correct/Not Correct.

The evaluation and dataset curation was based on OpenAI’s official evaluation scripts. We created a feature branch (feat/mlperf_integration) that:

1. Enabled inference on tokenized inputs (HarmonySampler), and
2. Added evaluation for LiveCodeBench v6

To curate the accuracy dataset, we then collected input traces from running `gpt_oss.evals` and ran multiple times to get a reliable accuracy threshold on AIME25, LCB_V6 and GPQA_Diamond.

Accuracy targets for GPT-OSS

To qualify for submission, implementations must meet or exceed the following accuracy targets on the curated accuracy dataset:

Performance mode (low reasoning effort)

For measuring pure inference speed (tokens/second), we utilize a dataset sampled from ccdv/pubmed-summarization.

• Task: PubMed health article summarization.
• Configuration: “Low reasoning effort” via Harmony format.
• Sequence lengths: Max output length is set to 10,240 tokens.
• Metrics: Throughput and Latency.
• Mean input sequence length: 5,000 tokens
• Mean output sequence length: 1,250 tokens

Performance metrics

The performance metrics evaluate the systems under strict latency constraints depending on the deployment scenario.

GPT-OSS constraints:

• Interactive Scenario: 99th percentile TTFT <= 2.0s; TPOT <= 15ms.
• Server Scenario: 99th percentile TTFT <= 3.0s; TPOT <= 80ms.

Accuracy metrics

For GPT-OSS, accuracy is evaluated strictly on the accuracy mode dataset. AIME 2024 is evaluated via Exact Match, LiveCodeBench v6 via Code Execution (Pass/Fail), and GPQA-Diamond via Correct/Not Correct mapping.

Compliance checks

Because this benchmark uses separate datasets for accuracy and performance evaluations, new compliance checks were introduced to ensure accuracy when running in performance modes. The tests are as follows:

1. TEST07: This test verifies the accuracy of a performance run by using the GPQA dataset, one of the three accuracy datasets. Using all three datasets would be too computationally expensive, and the task force decided to use a subset for enforcing the accuracy verification.
2. TEST09: This test verifies that the mean output length generated from the performance dataset is within 10% of the output length from reference implementation.  

Reference implementation

The official reference implementations for the MLPerf Inference v6.0 benchmarks provide the necessary code and instructions to run the end-to-end evaluations.

• GPT-OSS 120B: GitHub link

A new workload: the DeepSeek-R1 interactive scenario

Building on the DeepSeek-R1 benchmark introduced in v5.1, we added an interactive scenario designed to represent the growing demand for low-latency responses in advanced reasoning uses cases like math, knowledge and reasoning, and complex coding tasks. The dataset (LiveCodeBench, MATH500, AIME, GPQA-Diamond, and MMLU-Pro) remains identical to the server scenario, with a minimum query count of 4,388, but the workload shifts to tighter bounded response times than server scenario.

Performance metrics

The performance metrics evaluate the systems under strict latency constraints depending on the deployment scenario.

DeepSeek-R1 Interactive Constraints & Speculative Decoding:

• New interactive scenario (Poisson arrival): 99th percentile TTFT <= 1.5s; TPOT <= 15ms.
• (Existing) server scenario (Poisson arrival): 99th percentile TTFT <= 2s, TPOT <= 80ms.

To meet the demanding latency requirements of the DeepSeek-R1 Interactive scenario, we are enabling speculative decoding for this specific workload. Implementations must use the official DeepSeek-R1 MTP (Multi-Token Prediction) Head with EAGLE-style decoding:

• Algorithm: EAGLE-style decoding with deepseek-ai/deepseek-r1 MTP head.
• Configuration: speculative-num-steps=3, speculative-eagle-topk=1.0.
• Prohibitions: Implementations cannot artificially manipulate acceptance rates. Techniques such as continued pre-training of the MTP head, quantization of the MTP head weights, or post-training adjustments (fine-tuning, RLHF) are strictly disallowed.
• See official rules here: mlperf-inference/policies

Accuracy metrics

For DeepSeek-R1 interactive, the accuracy metrics remain identical to the v5.1 server submission (exact match for math/QA and code execution for LiveCodeBench), ensuring the speculative decoding implementation does not degrade the model’s reasoning capabilities.

Reference implementation

The official reference implementations for the MLPerf Inference v6.0 benchmarks provide the necessary code and instructions to run the end-to-end evaluations.

• DeepSeek-R1: GitHub link

Conclusion

With MLPerf Inference v6.0, we continue to push the boundaries of what a standardized benchmark suite can measure. By introducing a split-dataset approach for GPT-OSS and a rigorous, latency-constrained Interactive scenario for DeepSeek-R1, we are providing the industry with the tools needed to evaluate the next generation of AI applications—from high-throughput summarization to real-time agentic reasoning.

We invite the community to explore the reference implementations and to participate in future rounds as we continue to track the rapid evolution of large language models.

For additional information on MLCommons and details on becoming a member, please visit MLCommons.org or email participation@mlcommons.org.

The post A new GPT-OSS benchmark and DeepSeek R1 updates for latency-optimized reasoning appeared first on MLCommons.

At GTC, MLCommons^® co-founder David Kanter unveiled the answer: MLPerf^® Endpoints, a ground-up rethinking of how the industry’s benchmark of record measures generative AI performance. With over 125 member organizations, more than 90,000 reproducible results to date, and recognition by IEEE and ISO/IEC SC42, MLPerf already underpins critical procurement decisions across government, industry, and academia. Endpoints is designed to keep that trust intact while adapting to a landscape that looks fundamentally different from what it was just two years ago. You can try it out here.

Why Traditional Approaches Need To Change

Traditional MLPerf inference benchmarks used a tightly coupled architecture: the load generator and model server ran as a single local process with shared dependencies. That worked well for classical ML, but generative AI deployments are API-first — whether on-prem, in the cloud, or via managed cloud endpoints. 

Meanwhile, measuring GenAI performance is harder than it looks. Real serving combines accuracy, latency, throughput, and sequence length into a non-linear, multi-dimensional surface. Long-tail queries, variable arrival patterns, and tight SLAs interact in ways that simple scenarios miss entirely.

An API-Centric Architecture

MLPerf Endpoints replaces the monolithic design with a decoupled client that communicates with any model-serving API endpoint via standard interfaces such as HTTP or gRPC. The benchmark client is lightweight and production-ready; the system under test is simply a URL. This means zero-effort integration for submitters — point the client at your endpoint and run. The architecture also enables benchmarking of managed cloud services alongside bare-metal deployments on an equal footing, something the previous framework could not easily support.

Under the hood, a new scalable load generator uses separate worker processes, pre-warmed connection pools, and ZeroMQ-based IPC to ensure the harness itself never becomes the bottleneck, even when testing rack-scale systems.

Pareto Curves and Step Functions: New Visualisation for New Metrics and Easy Comparison

One of the most compelling innovations is how results are presented. Each benchmark run varies concurrency and captures key metrics, including TTFT (time to first token), throughput (tokens per second), interactivity (tokens per second per user), and response latency. Submitters tune parallelism and batch settings for each operating point, and the visualizer plots these metrics as a Pareto curve (e.g., throughput vs. interactivity)—giving buyers an immediate picture of the real-world trade-offs, such as between serving more users and keeping each user’s experience responsive.

Crucially, MLPerf Endpoints uses step functions rather than interpolated trend lines. GenAI performance is highly non-linear; interpolating between measured points can suggest performance levels that were never actually achieved, masking memory overflows or P99 latency spikes. Step functions show only verified operating points, eliminating what the presentation aptly called “paper performance.” Customers can easily compare these step functions against each other and match these verified points to their own use cases — high concurrency during the day, best possible interactivity at night.

Rolling Submissions: Benchmarking at the Speed of Software Updates

Perhaps the boldest change is operational. MLPerf has historically published results on a fixed bi-annual schedule, depending on the benchmark offering (Training, Inference, Storage, etc). In a market where major model releases land every few weeks, that cadence is too slow for buyers writing RFPs and vendors launching hardware. Starting in Q2 2026, MLPerf Endpoints will move to continuous rolling submissions: submitters can publish peer-reviewed, audited results at any time. Incremental submissions let vendors start with a baseline Pareto curve and iteratively add more operating points as their software stack matures.

The approach is inspired by proven methodologies from other industry-standard bodies such as SPEC and TPC, adapted to the world of AI. Peer review and audit requirements will remain fully intact to deliver the robustness that the industry demands.

What Comes Next

The first MLPerf Endpoints v0.5 demonstration feature results from AMD, Google, Intel, KRAI, and NVIDIA, backed by over 30 supporting organisations, including Argonne National Laboratory, Broadcom, Dell, HPE, Lambda, Lenovo, Oracle, Red Hat, and the University of Florida. The results include models such as DeepSeek-R1, GPT OSS 120B, Llama 3.1 8B, QWEN 3 Coder 480B and more, running on nearly a dozen different systems.

Looking ahead, MLCommons is inviting the broader ecosystem to shape what comes next. Enterprise and IT buyers can join the advisory council. OEMs, CSPs, and ODMs can contribute results to the rolling leaderboard. Model developers and API providers can integrate next-generation SOTA models and build managed roadmaps. Researchers can anchor reproducible baselines using the Endpoints framework. New models — especially popular and commercially relevant ones — are continuously evaluated for inclusion. You can try it out for yourself here.

Get involved: MLPerf Endpoints rolling submissions open in Q2 2026. To participate, contribute, or learn more, visit https://mlcommons.org/benchmarks/endpoints/ or join our working group.

The post Standardizing Generative AI Service Evaluation: An API-Centric Benchmarking Approach appeared first on MLCommons.

The MLPerf Inference benchmark has evolved to an industry standard for measuring the performance of artificial intelligence (AI) infrastructure by creating a fair benchmarking platform and incorporating diverse workloads, including vision, speech, and natural language processing. MLCommons’ effort to stay relevant with the latest AI workloads is evident not only in the introduction of new models in the Generative AI space but also in upgrades to legacy workloads. The YOLO Task Force was formed to upgrade the RetinaNet benchmark in the edge suite to Ultralytics YOLO11, a more modern, state-of-the-art detection model. 

RetinaNet has been a solid, academically sound benchmark for single-shot detection for years, but there are several reasons to upgrade to a more modern YOLO (You Only Look Once) variant for the object detection workload [1]. YOLO has experienced rapid growth across research and real-world applications, driven by accelerated innovation, frequent releases, and strong community adoption – positioning it as one of the most effective and exciting models for modern object detection workloads. YOLO11, released by Ultralytics in September 2024, introduces substantial architectural and training improvements – achieving higher accuracy with fewer parameters and offering model variants ranging from YOLO11n (nano) to YOLO11x (extra large) that support diverse compute-accuracy trade-offs [2]. In contrast, RetinaNet has received fewer major updates in recent years, leading to diminished development momentum and reduced community adoption. Meanwhile, the YOLO family continues to evolve rapidly, reflecting cutting-edge advances and the prevailing trends in the AI industry’s object detection domain. 

Model selection 

Before YOLO, state-of-the-art detectors were “two-stage” systems that first proposed regions of interest and then classified them [1]. The YOLO model was first introduced in 2015 by Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi at the University of Washington. It fundamentally changed computer vision by treating object detection as a single regression problem—predicting bounding boxes and class probabilities simultaneously in a single pass—whereas previous models like R-CNN required thousands of separate passes per image [3]. The “single-shot” approach sacrificed a small amount of accuracy for a massive speedup, enabling real-time detection at 45 frames per second. Community-driven YOLO model improvements from organizations like Ultralytics improved subsequent versions.

Our initial challenge was balancing the proven stability of established versions with the cutting-edge accuracy of the latest releases. While legacy versions like Ultralytics YOLOv8 have solidified their place as the industry standard due to their robust anchor-free design and broad community support, we ultimately focused our evaluation on YOLO11 – and even peeked at the nascent Ultralytics YOLO26 – to ensure the benchmark remains future-proof. 

The technical analysis revealed that YOLO11 offers a significant leap in parameter efficiency and raw accuracy. For our benchmark, we selected the YOLO11l (large) variant, which achieves a 53.4% mAP on the COCO dataset, outperforming the YOLOv8l baseline’s 52.9%. The Mean Average Precision (mAP) score serves as the ultimate arbiter of quality because it balances precision (number of correct detections) and recall (objects found). This is achieved while maintaining a highly competitive footprint of 25.3 million parameters, a refinement achieved by replacing the older C2f modules with more efficient C3k2 blocks and integrating C2PSA (Cross-Stage Partial Spatial Attention), which enhances the model’s focus on salient regions without a proportional increase in computational cost [4]. 

Beyond the YOLO family, we weighed alternative modern architectures to ensure a comprehensive selection process. EfficientDet was noted for its strong accuracy-to-FLOPs ratio via the BiFPN (Bi-directional Feature Pyramid Network), though it often lacks the raw throughput required for high-velocity production environments [5]. Similarly, transformer-based detectors like DETR and Deformable successors are good for their streamlined, NMS-free training pipelines and superior global context [6]. Ultimately, YOLO11 Large was selected for its unparalleled production throughput and its ability to act as a rigorous stress test for hardware interconnects and data-loading pipelines. Choosing YOLO11 allows the MLPerf benchmark to reflect real-world deployment patterns while pushing vendors to optimize for high-efficiency, attention-augmented convolutional neural networks. 

Fig. 1. Schematic diagram of YOLO11 showing the Backbone, Neck, and Head components (adapted from A. T. Khan and S. M. Jensen, LEAF-Net, Dec. 2024). 

Dataset 

Choosing the correct dataset was another fundamental decision in the YOLO Inference benchmark integration task, as it serves as the ground truth for our benchmark’s validity. We selected COCO 2017 (Common Objects in Context) because it was used to train YOLO models and remains the gold standard for object detection. With 80 object categories and over 1.5 million instances, we can ensure that the model won’t just memorize shapes but also truly understand spatial relationships and the varied hidden components in realistic images [7]. The model was not trained by the group; we used a subset of the COCO validation dataset and verified that it maintained its original accuracy.

However, distributing a large-scale dataset for an open benchmark such as MLPerf poses legal and compliance challenges, particularly regarding commercial distribution rights. While the COCO annotations are open, some images from sources are marked “Non-Commercial,” which means they are not compliant with commercial benchmarking. To address this, the Task Force developed a custom filtering pipeline to create a safe subset of the full COCO 2017 dataset. This ensures that the final dataset used by our partners is fully distributable and legally safe for both academia and industry, without compromising the benchmark’s statistical integrity. 

The MLPerf Subset 

Table 1: The COCO MLPerf subset shows the total number of safe images compared to the full dataset.

Loadgen integration

After creating the  COCO dataset, we had to ensure consistent accuracy scores in our YOLO LoadGen integration. MLCommons LoadGen (Load Generator) is a reusable C++ library with Python bindings that effectively and fairly measures the performance of ML inference systems by generating standardized query traffic patterns to ping the model across scenarios such as SingleStream, MultiStream, and Offline. The LoadGen API records all queries and responses for verification and summarizes whether performance latency constraints are met. While remaining model-agnostic and not specifically handling accuracy evaluation, the LoadGen API generates files needed for model-specific accuracy metric calculations [8].

Originally, the YOLO11 implementation generated a standard predictions.json file. While this format is sufficient for general COCO validation, it was incompatible with the COCO MLPerf Accuracy script for several reasons:

• Class mapping inconsistency: YOLO models traditionally use an 80-class index (0-79) derived from the COCO dataset. However, for the MLPerf accuracy evaluation, we wanted to use the 91-category COCO mapping, where class IDs are non-sequential.
• Coordinate normalization: The standard YOLO outputs are often absolute pixel coordinates or in the XYWH (standard x, y coordinates, width, height) format. For our accuracy evaluation, we wanted to use a specific serialized payload: a 7-element float array containing [index, y_min, x_min, y_max, x_max, score, class]. Crucially, these coordinates must be normalized (between 0.0 and 1.0) relative to the original image dimensions.
• Buffer serialization: Unlike a standard JSON dump, the MLPerf LoadGen requires results to be serialized into a byte buffer through the QuerySamplesComplete API.

Once we identified and resolved the issues above, we achieved matching mAP results to the Ultralytics reference using our YOLO LoadGen implementation. We got it to work with the following strategy:

• Class re-indexing: We implemented a robust mapping layer using the official COCO 80-to-91 conversion array. This ensures that every detection reported by YOLO is immediately translated into the category ID space recognized by the accuracy script.

Target ID = COCO_{80_to_91}[Model Class Index]

• Coordinate transformation: The Runner.enque logic was rewritten to handle image geometry dynamically. We first extracted the original height (H) and width (W) from the Ultralytics results object to perform a real-time normalization of the XYXY bounding box:

y_min = box[1] / H

x_min = box[0] / W

y_max = box[3] / H

x_max = box[2] / W

• LoadGen native serialization: We utilized the struct.pack(“7f”, …) method to create a binary payload. This payload is then passed directly into lg.QuerySampleResponse, allowing the MLPerf LoadGen to handle the logging in its native binary format. This change not only fixed the accuracy reporting but also reduced I/O overhead during performance runs by eliminating the need to write to an intermediate file.

By shifting this logic directly into the LoadGen runner, we achieved seamless integration, with mlperf_log_accuracy.json generated to the exact schema required by the accuracy-coco.py script. This ensures that mAP calculations are bit-accurate and directly comparable with other submissions in the MLPerf Inference leaderboard.

Performance evaluation 

For the edge suite, the following performance metrics are measured for the standard MLPerf scenarios: 

1. Offline, where the goal is to measure peak throughput (samples/sec) by processing the entire dataset as a single batch. In this benchmark scenario, LoadGen will send all queries to the System Under Test (SUT) at the start.
2. Single-Stream focuses on the absolute minimum latency for a single image – critical for real-time edge responses. LoadGen sends the queries to the SUT as soon as the previous one completes. The duration of this scenario is set to 1024 queries and 60 seconds.
3. Multi-Stream simulates multiple channels and measures the maximum number of concurrent streams the system under test can support. LoadGen sends queries to the SUT using the same method as Single-Stream, but the run duration is longer, at either 270,336 queries or 600 seconds.

As a single-stage detector with an optimized detection head, YOLO11 eliminates several post-processing and anchor-heavy computation stages present in RetinaNet, ultimately demonstrating a significant improvement in end-to-end inference 

Accuracy metrics 

While RetinaNet provided strong detection accuracy due to its focal loss formulation and multi-scale FPN backbone, YOLO11 achieves superior mAP while also delivering on latency [9]. Our implementation uses the mAP@50-95 standard, a metric that computes the average precision across ten Intersection over Union (IoU) thresholds (0.5-0.95). By doing so, we want to see that the model not only identifies the correct class but also localizes it with high pixel-level accuracy. We found this to be a challenging yet fair set of parameters for the YOLO11l model to maintain high throughput while ensuring that its bounding box predictions are nearly identical to the ground truth. 

We established two distinct target accuracy thresholds: yolo-95 and yolo-99. These represent 95% and 99% of the state-of-the-art reference score, respectively. The yolo-95 threshold serves as the “default” mode, allowing for submissions that prioritize system speed and throughput for standard production needs. In contrast, the yolo-99 threshold represents the “high accuracy” standard for mission-critical applications, requiring the model to converge almost perfectly to the reference accuracy. The mAP scores for the two modes are determined by taking the YOLO11l mAP score from Ultralytics and multiplying it by 0.95 and 0.99 for the default and high accuracy modes respectively. The table below shows a score of 53.4 for YOLO11l from Ultralytics, which translates to 0.534 * 0.95 for the yolo-95 version and 0.534 * 0.99 for the yolo-99 version. By offering these two tiers, the MLPerf benchmark allows hardware vendors to showcase different optimization profiles. 

Table 2: Parameters and accuracy comparison of YOLO11 model variants.

Reference implementation 

The reference implementation for YOLO11 uses the official Ultralytics YOLO11 inference code. For readers interested in running the model in their own environment, we recommend following this MLCommons reference implementation, which includes the code and instructions for running the end-to-end benchmark, starting with the dataset and model downloads. 

Conclusion 

In summary, the transition from RetinaNet to YOLO11 marks a pivotal evolution in the MLPerf Inference benchmark, as the upgrade to a better model reflects industry trends. By adopting the YOLO11l variant, we are challenging hardware vendors to optimize for attention-augmented components like the C2PSA block. This shift ensures that our performance metrics are not just theoretical numbers, but actionable data points that translate directly to the efficiency and responsiveness of real-world AI use cases. 

Furthermore, the introduction of the yolo-95 and yolo-99 accuracy tiers provides a nuanced framework for AI infrastructure evaluation. It allows organizations to showcase the raw speed of aggressive quantization at the 95% threshold while maintaining a path for mission-critical, high-fidelity deployments at the 99% level. This benchmark will continue to serve as a positive step forward for the community, driving innovation and ensuring that the next generation of AI hardware is built to handle the complexity and scale of computer vision. MLCommons will continue working with domain experts and industry members to update the object detection models, ensuring that the MLPerf benchmark remains current and accurately reflects evolving industry trends.

Acknowledge

We would like to express our gratitude to the following individuals for their help and guidance during the development of this benchmark:

• Anandhu Sooraj (MLCommons)
• Arjun Suresh (MLCommons)
• Ashutosh Dhar (NVIDIA)
• Karl Pietri (MLCommons)
• Miro Hodak (AMD)
• Reilly Fairbanks (MLCommons)
• Scott Wasson (MLCommons)
• Zhihan Jiang (NVIDIA)

References

1. J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You Only Look Once: Unified, Real-Time Object Detection,” arXiv preprint arXiv:1506.02640, Jun. 2015. [Online]. Available: https://arxiv.org/abs/1506.02640
2. Ultralytics, YOLO11 Model Documentation. [Online]. Available: https://docs.ultralytics.com/models/yolo11/
3. R. Girshick, “Rich feature hierarchies for accurate object detection and semantic segmentation,” arXiv preprint arXiv:1311.2524, Nov. 2013. [Online]. Available: https://arxiv.org/abs/1311.2524
4. S. Author, T. Author, and L. Author, “YOLOv1 to YOLOv11: A Comprehensive Survey of Real-Time Object Detection Innovations and Challenges,” arXiv, Aug. 2025. [Online]. Available: https://www.arxiv.org/pdf/2508.02067
5. M. Tan, R. Pang, and Q. V. Le, “EfficientDet: Scalable and Efficient Object Detection,” arXiv preprint arXiv:1911.09070, Nov. 2019. [Online]. Available: https://arxiv.org/abs/1911.09070
6. N. Carion et al., “End-to-End Object Detection with Transformers,” arXiv preprint arXiv:2005.12872, May 2020. [Online]. Available: https://arxiv.org/abs/2005.12872
7. T. Lin et al., “Microsoft COCO: Common Objects in Context,” arXiv preprint arXiv:1405.0312, May 2014. [Online]. Available: https://arxiv.org/abs/1405.0312
8. E. Hall et al., “MLPerf Inference Benchmark,” arXiv, Nov. 2019. [Online]. Available: https://arxiv.org/pdf/1911.02549
9. T.-Y. Lin et al., “Focal Loss for Dense Object Detection,” arXiv preprint arXiv:1708.02002, Aug. 2017. [Online]. Available: https://arxiv.org/abs/1708.02002
10. **A. T. Khan and S. M. Jensen, “The schematic diagram of YOLOv11 illustrating its three core components: Backbone, Neck, and Head,” inLEAF-Net: A Unified Framework for Leaf Extraction and Analysis in Multi-Crop Phenotyping Using YOLOv11, preprint, Dec. 2024. Fig. 5. [Online Image]. Available:https://www.researchgate.net/figure/The-schematic-diagram-of-YOLOv11-illustrating-its-three-core-components-Backbone-Neck_fig1_386467184

The post YOLO for the MLPerf Inference v6.0 Edge Suite appeared first on MLCommons.

The MLCommonsⓇ MLPerfⓇ Inference benchmark suite is an industry standard for measuring the performance of machine learning (ML) and artificial intelligence (AI) workloads from diverse domains including vision, speech, and natural language processing. For each of these domains, the suite includes a carefully selected set of workloads that represents the state of the art in the industry across different application segments. These benchmarks not only provide key information to consumers for making application deployment and budget decisions but also enable vendors to deliver critical workload optimizations within certain practical constraints to their customers.

Over the past year, we have seen rapid advancements in the capabilities of video generative models like OpenAI Sora2. Gone are the days of hobbyists generating uncanny clips of Will Smith eating spaghetti; now professional artists create entire workflows around the capabilities of these models. As they now transition from being mere curiosities to becoming core parts of creative workflows, the need for a standardized benchmark has become clear. And thus the MLPerf Text-to-Video Task Force was convened to incorporate a dedicated video generation benchmark into the MLPerf suite.

Model selection

For this benchmark we picked the Wan2.2-T2V-A14B-Diffusers model (released July 2025) by Alibaba, as it was one of the best open weights models on the Text-to-Video leaderboard at the time. The model has been fully open sourced under an Apache 2.0 licence and can be run via Huggingface Diffusers.

The Wan2.2 model is run as a pipeline of 3 models:

1. The UMT5 XXL Text Encoder from Google, used to encode prompts
2. The Wan2.2 A14B Diffusion Transformer, used to generate a latent video representation
3. The Wan2.2 VAE Decoder, used to decode the latent video into a series of frames.

The crucial architectural feature of the Wan2.2-T2V-A14B-Diffusers model is that it is a type of Mixture of Experts model. However, unlike the standard MoE architecture, there is no gating network used to route tokens to experts, instead this model consists of 2 experts which are activated sequentially during the denoising process. The first expert is known as the “High Noise Expert” and is active during the early stages of denoising, after which the model switches to the “Low Noise Expert” which completes the denoising process.

Counterintuitively, most video generation models do not generate videos frame by frame, instead they generate the entire video at once, by denoising a massive video latent. This video latent usually represents both a spatial and temporal segment of the video, for example the Wan2.2 latent represents an area of 32×32 pixels across 4 frames. This means that generating a 5 second video at 720p resolution and 16fps would require a sequence length of 19,320. This leads to the model being heavily compute bound.

Performance metrics

One of the key difficulties we encountered when designing this benchmark was deciding which performance metrics to use. The text-to-video task is computationally expensive with long runtimes, with many systems taking multiple minutes per query.

One consequence of this performance profile is that we had to limit the length, resolution and number of videos generated to ensure that this benchmark remained feasible, whilst also demonstrating frontier capabilities. Therefore we made the following decisions.

• Configuration: We limited the length of the generated videos to 5 seconds, whilst fixing the resolution at 720p. This meant that we would generate 81 720×1280 frames at 16fps.
• Runtime Target: To ensure the benchmark remains accessible to a wide range of submitters, we reduced the dataset size to a practical subset (100 out of 248 samples for performance mode, keep 248 for accuracy mode).

Replacing Server with SingleStream

One of the key changes we’re introducing in this benchmark is the replacement of the Server scenario with a SingleStream scenario for latency measurement. We based this decision on the large amount of compute needed to generate a single video, meaning that videos often take multiple minutes to generate. 

This poses a problem with the Server scenario where the system is assumed to be able to operate in near real-time. In practice this would mean that the System Under Test would quickly become overloaded with requests, meaning that most requests would spend the majority of their time waiting to be processed instead of being processed. This in turn would mean that the final latency measurements wouldn’t accurately reflect the hardware’s performance.

To solve this problem, we replaced the Server scenario with a SingleStream one, where we would only measure the time taken to process requests, ignoring all wait times.

Dataset and Accuracy Metric and task selection

We selected VBench as the official dataset and accuracy framework for this benchmark. We reached this decision after a comparative analysis of available options, prioritizing licensing feasibility, robustness, and ease of adoption.

We evaluated a wide range of datasets at the beginning, including OpenVid-1M, VidGen-1M, WebVid-10M, and ActivityNet. A primary filter for selection was commercial viability, as MLPerf submissions often come from industry partners for commercial hardware validation.

• VidGen-1M and WebVid-10M were disqualified due to restrictive licensing (e.g., Non-Commercial or Research-Only terms), which leads to legal risks for benchmarking.
• OpenVid-1M offered a permissible license (CC BY 4.0) but functioned solely as a dataset and lacked an integrated evaluation framework.

VBench distinguished itself by offering a holistic solution that combined a diverse prompt set with a pre-validated scoring suite. Unlike raw datasets that would require us to independently develop and validate separate accuracy metrics (such as FVD or IS), VBench provided:

• Comprehensive Metrics: A suite of 16 distinct quality dimensions, including Subject Consistency, Motion Smoothness, and Aesthetic Quality.
• Standardized Prompts: A curated list of ~950 prompts designed to stress-test specific generation capabilities.
• Determinism: Experiments confirmed that VBench scores remained stable across different hardware backends (NVIDIA, AMD) with fixed seeds, a critical requirement for cross-vendor fairness.
• Widespread Adoption: VBench is a widely used benchmark by video model builders and has been cited in several technical reports, including that of the Wan model family.

While VBench provided the most robust framework, its default configuration was computationally expensive, requiring over 80 hours for a full inference pass. To align with MLPerf’s accuracy check runtime, we adapted VBench by:

• Subsetting the Dataset: We agreed to reduce to average the 6 key metrics: Subject Consistency, Background Consistency, Motion Smoothness, Dynamic Degree, Appearance Style, Scene.
  This selection focused on the most discriminatory dimensions, such as Dynamic Degree, Multiple Objects, and Scene Quality, while removing metrics that were redundant, static, or computationally trivial for datacenter-class hardware.
• Reducing Dataset Size: By focusing on these 6 metrics, the dataset was reduced to a statistically significant subset (248 samples), striking the necessary balance between rigorous accuracy validation and manageable submission runtimes.

VBench was chosen by the taskforce, because it offered the only commercially viable, legally cleared, and methodologically complete framework that could be adapted to meet the rigorous runtime constraints of the MLPerf Inference benchmark.

Reference implementation

To ensure a fair and reproducible benchmark, the reference implementation for the Text-to-Video task is built on a standardized open-source foundation. 

Here is the reference setup:

• Model Architecture: We utilize the Wan2.2-T2V-A14B-Diffusers model (hosted by Wan-AI). This is a 14-billion parameter Diffusion Transformer designed for high-quality video generation. 
• Precision & Compute: The reference implementation runs in BF16 (BFloat16) precision. This choice reflects modern datacenter standards, balancing numerical stability with efficient memory usage.
  • Reference Accuracy Score: 70.48 (VBench).
  • Minimum Accuracy Threshold: 69.77 (99% of reference).
• Generation Pipeline: The reference pipeline is adapted from the Hugging Face Diffusers library, ensuring broad compatibility and ease of use.
  • Input: Text prompt + Fixed Latent Tensor (to ensure deterministic outputs for debugging and verification).
  • Scheduler: Uses the UniPCMultistepScheduler (aligned with the Wan2.2 default) to optimize step efficiency.
  • Reduced diffusion steps (per TF discussion): 20 steps in diffuse process.
  • Output: 720px1280p resolution at 16 frames per second, 81 frames in a video.
• Containerization: To simplify deployment, the entire reference stack including Python dependencies, CUDA 12.1 libraries, and VBench evaluation tools is provided as a Docker container. This allows submitters to “build and run” the benchmark with a single launch.sh script.

Conclusion

With the introduction of the Text-to-Video task in MLPerf Inference v6.0, we have taken a significant step to generative video workloads. This benchmark provides the industry with a reliable, reproducible approach to measure the rapidly evolving capabilities of both hardware and software.

The architectural decisions behind this inaugural benchmark reflect the the real video generation tasks today:

• Model Selection: We selected Wan2.2-A14B-Diffusers, a powerful open-weights T2V model that represents the state-of-the-art in open-source generation.
• Performance: We adopted SingleStream as the primary metric, and keep the standard Offline scenarios. The high-fidelity video generation task is currently a compute-bound, high-latency task which fits best for the current use cases .
• Accuracy: We integrated VBench as our evaluation framework, ensuring that performance optimizations do not come at the cost of visual fidelity, motion coherence, or prompt adherence.

This release represents a foundational baseline, but the field is moving fast. As generation latencies drop from minutes to seconds, we expect that the benchmark will evolve to include Server Mode scenarios to reflect in real-time use cases.

The post Bringing Text-to-Video to MLPerf Inference v6.0 appeared first on MLCommons.

Scaling compute, guided by neural scaling laws, has dramatically reduced the need for manual feature engineering in natural language processing (NLP) and computer vision by using large-scale attention-based transformer models to automatically learn rich representations from data [1][2]. A similar trend is transforming deep learning-based recommendation systems, which have traditionally relied on architectures built from multi-layer perceptrons (MLPs), graph neural networks (GNNs), and embedding tables [3][4][5]. Recently, large sequential and generative models have been successfully deployed in online content recommendation platforms, delivering substantial improvements in model quality [5][7][8][9][10][11][12][13]. Given the global scale and importance of recommendation systems [6], incorporating such large-scale sequential recommendation models into the MLPerf Inference benchmark suite helps support ongoing infrastructure development.

We introduce DLRMv3, the first sequential recommendation inference benchmark in MLPerf’s DLRM family. DLRMv3 is built around an HSTU-based [5] architecture for ranking, capturing the dominant compute patterns of modern recommendation workloads: long input sequences, attention-heavy computation, and large embedding tables. Compared to the existing DLRM benchmark (DLRMv2 [14]), DLRMv3 increases the model size by 20X from 50GB to 1TB and raises the per-candidate compute by 6500X from 40M FLOP to 260 GFLOP over a span of just three years, aligning the MLPerf Inference suite with contemporary production-scale recommendation workloads, highlighting the increasing demand for compute. This higher compute regime is motivated by reported HSTU scaling behavior where higher model compute yields improved recommendation quality in production, enabling realistic resource burden–accuracy trade-off evaluation.

Task selection

Modern recommendation systems are typically deployed as multi-stage pipelines that separate candidate retrieval from ranking and, in some cases, additional refinement stages such as re-ranking or business-logic post-processing [15][16]. In a common design, a retrieval model first selects a small subset of relevant items from a very large corpus, optimizing for high recall, coverage, and strict latency and memory constraints at scale [15][7]. A downstream ranking model then scores these candidates using richer features and a more expressive architecture, optimizing for fine-grained user engagement metrics (e.g., click-through rate (CTR), watch time, satisfaction) under somewhat looser but still production-critical latency and throughput constraints [15][16][18]. This separation of stages is now standard in large-scale industrial systems, including those used for web, video, and social content recommendation.

DLRMv3 focuses on the ranking stage of this pipeline. Ranking models typically dominate the overall ML compute budget in production recommendation systems and are a focus of innovation in model architecture (e.g., attention-based sequential models and larger embedding tables), making them especially relevant for hardware and systems benchmarking. Focusing on ranking also maintains continuity with prior MLPerf DLRM benchmarks which target click-through-rate prediction.

Formally, given a user’s interaction history (e.g., a sequence of previously viewed or engaged items) and a candidate item, the DLRMv3 model predicts the probability of a desired outcome such as a click, like, or watch. This probabilistic prediction task is directly aligned with earlier DLRM benchmarks, which also model CTR-style binary outcomes.

Model selection

Figure 1. Model Architecture of different DLRM models.

We introduce an HSTU-based architecture as MLPerf’s third-generation deep learning recommendation benchmark (DLRMv3). In the DLRM lineage, DLRMv1 is built from MLPs and embedding tables with simple dot-product feature interaction, and DLRMv2 adds a deep-cross network component for richer explicit feature crossing. DLRMv3 introduces a new sequential feature transformation, interaction, and extraction component based on the hierarchical sequential transduction unit (HSTU) [5] while retaining a single, large embedding table and a top MLP for final prediction (Figure 1(c)).

HSTU-style architectures have been shown in production systems to effectively model long user interaction histories and improve recommendation quality relative to traditional MLP/DCN-based models at comparable or higher compute resources. They also reflect a set of compute characteristics that increasingly dominate modern recommendation inference—long sequences, attention-heavy computation, and large embedding tables—making HSTU a representative and forward-looking choice for a system-level benchmark.

In the table below, we compare model configurations among different generations of DLRM. The 260 GFLOP is computed as 2 * layers * (UIH_length * UIH_length * EmbDim / 2 + UIH_length * EmbDim * EmbDim * 4 + UIH_length * EmbDim * EmbDim * 3), which accounts for attention FLOP as well as the pre- and post-attention GEMMs. Note that this 260 GFLOP “per-candidate” number is an effective normalization: in a typical ranking request, the HSTU encoder processes the shared user-interaction-history (UIH) sequence once, and its output is reused when scoring a candidate set (2K candidates in DLRMv3), so the dominant UIH encoding computational cost is amortized across candidates rather than repeated 2K times. Moreover, because DLRMv3 uses a streaming time-series setup, deployments can reuse UIH-related KV states across consecutive timestamps for the same user, avoiding recomputation of the UIH encoding and reducing redundant dense compute by roughly 80–90% in steady state.

To better align DLRMv3 with MLPerf Inference benchmarking goals and practical constraints, we introduce two intentional deviations from the original HSTU paper setup. These changes are motivated by the need for hardware-friendly, widely implementable benchmarking that still captures the important compute patterns of sequential recommendation models.

Action embedding preprocessing: The original HSTU model uses contextual interleaving of action embeddings, where contextual features and user actions are interleaved in the input sequence. This arrangement provides richer context to the model to learn dependencies between user behavior and item features. However, interleaving doubles the effective sequence length, which in turn significantly increases the computational cost. In DLRMv3, we omit action interleaving in the baseline and instead use a simplified input sequence where action embeddings are directly combined with contextual embeddings without expanding the sequence length. We choose this design because 1) the synthetic benchmark dataset (detailed in the next section) is intended for performance measurement and does not contain a sufficiently rich set of action features to justify the added modeling complexity and doubled sequence length, and 2) the non-interleaved option provides a more balanced accuracy–efficiency tradeoff that is appropriate for a standardized inference benchmark.

Time/position encoding: The original HSTU model uses a relative positional bias in the attention mechanism, implemented as Mask(SiLU(QKT)+bias)V, which helps capture relative temporal relationships between tokens and can improve accuracy. DLRMv3 instead adopts absolute time/position encoding, where a position-dependent bias is added to the query, key, and value vectors, and attention is computed as Mask(SiLU(QKT))V. We use absolute bias because relative bias introduces kernel optimization challenges and can slow down attention computation on many processors, whereas absolute encodings are widely supported, easier to optimize, and yield more predictable performance. 

Dataset selection

Unlike DLRMv1 and DLRMv2, DLRMv3 formulates recommendation as a sequential transduction task over long user interaction histories and a very large item set. To be representative of modern production workloads, a suitable benchmark dataset needs to satisfy several properties simultaneously: (1) reasonably long user-interaction histories per request (thousands of events), so that sequential models and attention layers are meaningfully exercised; (2) a very large item set, consistent with the single large embedding table used in DLRMv3 (hash size on the order of one billion); and (3) a streaming structure, where the items viewed by a given user and the user’s preferences evolve over time and inference requests can be replayed in timestamp order. To the best of our knowledge, no existing public recommendation dataset provides all of these characteristics at the required scale.

For this reason, DLRMv3 uses a synthetic dataset specifically designed to match the model and system characteristics of large-scale sequential recommendation. The generator simulates five million users interacting with a billion items over time. Items are partitioned evenly into 128 categories, and each user is randomly assigned four categories that define their long-term interests. We generate 100 timestamps per user. The first 90 timestamps are used for training, and the remaining 10 timestamps are reserved for inference. For each user and each timestamp, with 70% probability, we generate a user interaction history (UIH) and a candidate sequence, and with 30% probability, the user has no request at that timestamp. The average UIH sequence length per (user, timestamp) is approximately 100, and the candidate sequence length is 2K. The generated sequences respect the streaming setup: at earlier timestamps, only a prefix of item IDs in each category can be sampled, and the maximum reachable ID grows linearly over time, mimicking new items entering the system.

When generating each user’s sequence, item categories for each position are sampled using a Dirichlet-process-style mechanism, following the generation algorithm in Section C of [5]. This setup yields a “rich get richer” dynamic where previously visited categories are more likely to be revisited. For each sampled category, item, and timestamp, we generate a continuous user-item rating (used as an action feature) by combining (1) a category- and time-dependent rating profile that varies smoothly over time via cosine functions with (2) an intrinsic per-item rating drawn once from a global 1-5 distribution representing item quality. The final rating is their average and is then used to derive labels for prediction.

Before converging on the streaming generator described above, we also experimented with the fractal expansion [19] approach for scaling the user-item interaction matrix from a smaller dataset. In brief, fractal expansion applies randomized kronecker products to a small interaction matrix to increase the dataset size while certain static structural properties (e.g., item popularity distribution and user engagement distribution) are preserved. However, fractal expansion is fundamentally designed for static datasets and does not naturally produce the streaming structure required by DLRMv3. In particular, DLRMv3 needs temporally ordered requests with evolving user histories and an expanding reachable item universe (to mimic new items entering the system) so that inference can be replayed in timestamp order. Fractal expansion does not inherently model properties such as per-user sequences over time and preference drift. Due to this fact, we did not select fractal expansion for DLRMv3 and instead designed a generator that simulates the streaming and sequential characteristics of online recommendation workloads.

Checkpoint Training

During the benchmarking creation process, we decided to use a 1TB float16 embedding table to store the one-billion item set in order to reflect the production use cases for such a model. This checkpoint is later trained on a single NVIDIA HGX B200 with eight GPUs and the fully open-sourced Generative Recommenders repo. Because each B200 has 180GB HBM capacity, the 1TB item table can  be fully sharded to eight GPUs with enough capacity remaining in the GPUs’  HBM to store weights, activations, and gradients for the training loop to run. 

Performance and accuracy metrics

The performance metrics chosen for the DLRMv3 benchmark follow the MLPerf Inference conventions and are closely aligned with those used for DLRMv2:

1. Offline scenario: Query processing throughput measured as queries per second (QPS) with no latency constraint beyond completing the full query set as quickly as possible.
   
2. Server scenario: QPS measured under an end-to-end latency constraint on the system under test (SUT) reflecting an online serving environment where requests arrive over time and must be processed under strict tail-latency targets.

To better reflect real-world deployment demands for large-scale recommendation ranking, we set the 99th-percentile SUT latency threshold to 80 ms in the server scenario.

Unlike other MLPerf Inference benchmarks, DLRMv3 uses a streaming setup with explicit timestamped requests. As a result, there is a request ordering constraint: requests associated with earlier timestamps must be served before those with later timestamps. Because MLPerf LoadGen does not natively enforce temporal ordering, this constraint may need to be implemented via SUT-side scheduling or request handling logic to ensure that streaming semantics are preserved while optimizing for throughput and tail latency.

For accuracy verification, we provide a trained reference checkpoint with a model size of 1TB in float16. Accuracy is evaluated on 0.1% subset of the inference dataset, comprising 34,996 requests across 10 inference timestamps. We report normalized entropy (NE), accuracy, and area under the ROC curve (AUC) as the primary quality metrics. For the reference implementation, the observed metrics are:

NE: 86.69%,

Accuracy: 69.65%,

AUC: 78.66%.

These values serve as the target baseline for accuracy compliance in the DLRMv3 benchmark. For accepted submissions, all three metrics (NE, Accuracy, AUC) must be within 99.9% of the reference implementation values.

Reference implementation

We provide a reference implementation of the DLRMv3 benchmark at

https://github.com/mlcommons/inference/tree/master/recommendation/dlrm_v3. The reference implementation covers model implementation, synthetic data generation, and performance/accuracy evaluation under the MLPerf Inference rules, as detailed below:

Model implementation: we provide a PyTorch implementation of the DLRMv3 HSTU-based ranking model, including a large embedding table consistent with the 1B-hash and 512-dimension configuration, a HSTU sequential encoder stack with the DLRMv3 design choices, and customized Triton kernels for HSTU operations.

Synthetic data pipeline: we provide the synthetic data generation script as well as a streaming data loader that constructs per-request model inputs (user interaction history + candidate set) and labels, matching the benchmark specification.

Reference benchmark: the reference benchmark contains three main runtime components:

1. Data producer: this part takes in sample/request ID and performs batching and outputs item IDs of batched samples. This part is executed on the CPU, and the reference implementation supports both single- and multi-threading in Python.
   
2. Sparse forward: this part takes in item ID lists and outputs embedding lookup outputs. This part is executed on the CPU, as datacenter systems typically contain large CPU memory to store the 1TB embedding table. The reference implementation uses PyTorch’s default embedding lookup operation without any quantization (in the float16 datatype). The implementation serves as a baseline for future improvements, e.g. distributed embedding lookup, GPU lookup, etc.
   
3. Dense forward: this part takes in batched embeddings and runs the HSTU encoder forward in the bfloat16 datatype to generate predictions. The implementation employs data parallelism across GPUs on a single host by distributing input batches to eight GPUs using Python multi-processing. On each GPU, the implementation uses customized Triton kernels to improve efficiency. This implementation serves as a baseline for future improvements, including more efficient kernels and low-precision quantization.

To illustrate the performance of the DLRMv3 reference implementation, we report end-to-end results for different numbers of NVIDIA H100 GPUs on a single host with batch size equal to 10. In each configuration, we increase QPS until the system approaches saturation while still maintaining reasonable latency.

A breakdown of query time at 8 GPUs highlights where time is spent in the baseline:

The results show sub-linear scaling in throughput (QPS) with up to eight GPU devices: going from one to eight GPUs increases QPS by 3.3X (300 to 1000), while average latency drops from 78 ms to 60 ms. At eight GPUs, the end-to-end P99 remains slightly above 80 ms due to batching, data movement, and Python/CPU-side overheads. This outcome is expected and consistent with the goals of the reference implementation. The current code is a Python-based baseline, not an aggressively optimized C++/production server, and it is explicitly not targeting the best possible performance. The numbers therefore demonstrate substantial headroom for submitters to optimize request scheduling, embedding placement, kernel optimization, and inference-server integration to satisfy the P99 latency constraint.

Optimization opportunities

DLRMv3 is intentionally designed to expose multiple layers of optimization opportunities across the inference stack–from the serving framework to dense and sparse kernels to KV-cache-aware scheduling. The reference implementation provides a clear but unoptimized baseline in several of these dimensions, leaving room for submitters to explore system and kernel improvements while preserving accuracy.

Inference server library. The current reference pipeline is implemented in Python with a simple CPU/GPU orchestration model. In production, ranking models of this scale are typically deployed behind high-performance inference serving frameworks. There is significant room to improve request batching, thread and stream management, and SUT-side scheduling by integrating with an optimized inference server library. 

KV cache integration. DLRMv3’s sequential HSTU architecture naturally supports KV caching, and there are two main dimensions where caching can yield meaningful gains:

1. In-request cache (M‑FALCON-style microbatching). As discussed in the HSTU paper [5], when serving large candidate sets (e.g., >1K candidates), M‑FALCON applies microbatching over the candidates while reusing the KV cache for the shared user interaction history. This strategy does not change the model’s FLOP count requirements or accuracy, but it can materially improve performance by amortizing candidate processing and overlapping CPU and GPU work within a single request. In a DLRMv3 setting, this arrangement can reduce per-request latency without altering the underlying model.

2. Across-request cache. For users who appear in multiple timestamps (as in the DLRMv3 streaming setup), the KV cache corresponding to their user interaction history can be reused across requests. This avoids recomputing the UIH encoding for each timestamp, reducing redundant FLOP by around 90% and improving both throughput and latency. Careful cache management and invalidation policies are needed to ensure correctness under streaming semantics.
   

Together, in-request KV microbatching (M‑FALCON style) and across-request KV caching offer complementary optimization levers: the former amortizes compute over large candidate sets and enables finer-grained CPU/GPU overlap within a request, while the latter reduces repeated UIH compute across consecutive requests from the same user. The reference implementation does not yet exploit these techniques, leaving them as explicit optimization opportunities for submitters targeting high-performance DLRMv3 deployments.

Dense optimization. The dense path in the reference implementation (HSTU encoder + top MLP) already uses Triton kernels and bf16, but there is still substantial room to improve. Further optimizations could include:

• more efficient attention and feed-forward kernels tailored to DLRMv3’s sequence lengths and dimensions;
• better utilization of vendor-specific libraries and exploration of lower-precision formats (e.g., FP8, INT8) under acceptable accuracy loss.
• optimized multi-GPU scaling within a node via improved load balancing, overlapping of data transfers with compute, and more sophisticated pipeline parallelism.

Sparse optimization. The sparse forward stage currently uses TorchRec embeddings on the CPU, with bf16 and no quantization, to support 1TB size in a simple, portable way. This is a natural baseline, but production systems often employ a combination of GPU-based embedding lookup, distributed sharding, hierarchical caching (e.g., HBM + DRAM), and quantization/compression. Submitters can explore moving embedding tables to accelerators, applying different types of quantization, or introducing model-parallel embeddings, as long as DLRMv3 accuracy targets are met. Improving the embedding pipeline can reduce end-to-end latency and CPU bottlenecks for large-scale deployments.

Conclusion

DLRMv3 extends the MLPerf Inference recommendation suite into the regime of large-scale sequential models, matching the size, structure, and serving patterns of the state-of-the-art production systems. By combining an HSTU-based ranking architecture, a billion-item synthetic streaming dataset, and a reference implementation that cleanly separates sparse and dense computation, DLRMv3 exposes the challenges of modern recommender inference: long user histories, attention-heavy sequence modeling, massive embeddings, and tight tail-latency targets under streaming workloads.

Notably, DLRMv3 increases the model memory footprint by 20X (from 50GB to 1TB) and the per-candidate compute by 6500X (from 40 MFLOP to 260 GFLOP) compared to DLRMv2, reflecting the rapid evolution of recommendation models over just three years. This dramatic scaling in compute and memory enables richer representations and more expressive models, ultimately delivering greater capability and accuracy to the community. By benchmarking these advances, DLRMv3 highlights how performance optimization is driving the frontier of recommender systems.

The benchmark is intentionally designed to leave room for innovation across hardware and software stacks–from inference servers and kernel optimizations to embedding pipelines and KV-cache-aware scheduling, while providing clear accuracy and performance baselines. We hope DLRMv3 will serve as a foundation for the community to evaluate, compare, and advance systems for next-generation recommendation workloads.

Contributors

Lucy Liao (Meta), Yu He (Stanford), Ze Yang (Meta), Zhao Zhu (Meta), Chunxing Yin (Meta), Rafay Khurram (Meta), Yuanjun Yao (Meta), Han Li (Meta), Daisy Shi He (Meta), Shilin Ding (Meta)

References

[1] Scaling Laws for Neural Language Models. https://arxiv.org/abs/2001.08361.

[2] Scaling Vision Transformers. https://arxiv.org/abs/2106.04560. 

[3] Scaling Law for Recommendation Models: Towards General-purpose User Representations.  https://arxiv.org/abs/2111.11294 

[4] Understanding Scaling Laws for Recommendation Models. https://arxiv.org/abs/2208.08489 

[5] Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations. https://arxiv.org/abs/2402.17152v2

[6] Recommender system is the single largest software engine on the planet — Jensen Huang. Q4’23 earning report. https://www.youtube.com/watch?v=txOv_pi-_R4&t=2020s 

[7] LONGER: Scaling Up Long Sequence Modeling in Industrial Recommenders. https://arxiv.org/abs/2505.04421 

[8] Towards Large-scale Generative Ranking. https://arxiv.org/abs/2505.04180 

[9] MTGR: Industrial-Scale Generative Recommendation Framework in Meituan. https://arxiv.org/abs/2505.18654 

[10] From Features to Transformers: Redefining Ranking for Scalable Impact. https://arxiv.org/abs/2502.03417 

[11] Hi-Gen: Generative Retrieval For Large-Scale Personalized E-commerce Search. https://arxiv.org/abs/2404.15675 

[12] HLLM: Enhancing Sequential Recommendations via Hierarchical Large Language Models for Item and User Modeling. https://arxiv.org/abs/2409.12740 

[13] Recommender Systems with Generative Retrieval. https://arxiv.org/abs/2305.05065 

[14] https://github.com/mlcommons/inference/blob/master/recommendation/dlrm_v2/pytorch/README.md 

[15] Deep Neural Networks for YouTube Recommendations, https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45530.pdf 

[16] Deep Learning Based Recommender System: A Survey and New Perspectives, https://arxiv.org/abs/1707.07435 

[17] Sampling-Bias-Corrected Neural Modeling for Large Corpus Item Recommendations, https://research.google/pubs/sampling-bias-corrected-neural-modeling-for-large-corpus-item-recommendations/ 

[18] Recommending What Video to Watch Next: A Multitask Ranking System, https://daiwk.github.io/assets/youtube-multitask.pdf 

[19] Scalable realistic recommendation datasets through fractal expansions, https://arxiv.org/pdf/1901.08910 

The post DLRMv3: Generative recommendation benchmark in MLPerf Inference appeared first on MLCommons.