MLPerf was originally designed to help a relatively small number of cloud providers and downstream system builders inform their AI hardware purchasing decisions. Today, procuring inference compute is a salient business decision for companies of all sizes – and it means evaluating options across neoclouds, cloud providers, and managed services simultaneously. These buyers need reliable, comparable, and independent performance benchmarks to inform their purchasing decisions, and these benchmarks must be dynamic enough to keep pace with an industry that launches new models weekly. 

MLPerf Endpoints is designed to meet this more diverse and complex demand for benchmarks to inform AI system procurement decisions. Four key principles are guiding our development of enterprise-buyer-centric benchmarks:  

1. Current: Results must keep pace with the market. Buyers can’t wait months for a benchmark round to include new hardware or models. 
2. Comprehensive: Buyers need benchmarks that cover the many competing inference providers, systems, and workloads available for purchase. 
3. Comparable: Buyers need to see apples-to-apples results across vendors, normalized for cost or power, to inform procurement decisions.
4. Commentary: To support buyers in these decisions, Endpoints provides additional context for benchmark results through visualizations, data filtering, and analysis. 

MLPerf Endpoints v0.7 is a foundation release with initial results from Coreweave, Google, Intel, KRAI, and Nvidia. We want to congratulate these members on their excellent results, spanning several orders of magnitude in performance across 3 benchmarks. This is the infrastructure on which we are building a more dynamic, comprehensive, and comparable inference benchmark suite for the data center. Endpoints currently supports automated submission pipelines, continuous review tooling, and dynamic visualization of results that you can see at mlcommons.endpoints.com. We are also evolving our benchmarking rules towards more buyer-centric benchmarks.

Later this year, we will deliver MLPerf Endpoints v1.0 with more buyer-centric rules, normalization, and an expanded set of benchmarks including agentic workloads and then open the rolling submission process to our broader membership. The rolling submission process ensures Endpoints will provide current and up-to-date results that move at the pace of the market. 

We’re grateful to our 30+ supporters who have helped guide the development of MLPerf Endpoints, including AMD, Argonne National Laboratory, Broadcom, Core 42, Dell, HPE, Lambda, Oracle,  and Red Hat. Their partnership has been invaluable in stress-testing our rules, processes, and improving our infrastructure. With this release, we are just getting started. 

If you’re a system or service provider, now is the time to get involved, help shape the rules, and plan your submission at a time of your choosing. Complete this form to let us know you’re interested in submitting, and what hardware you want to submit on. The community-developed rules are actively being refined for the 1.0 release later this year, and early participants in this effort will shape Endpoints’ direction. If you’re an enterprise buyer, Endpoints is being built for you, and we want to know how we can best support you and your organization. Let us know by emailing alejandro@mlcommons.org.

We deeply value the trust our members place in us as the benchmark of record for inference. We are excited to build this future of MLPerf with all of you. 

[1] A. Tschand, A. T. R. Rajan, S. Idgunji, et al., “MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from µWatts to MWatts for Sustainable AI,” in 2025 IEEE International Symposium on High-Performance Computer Architecture (HPCA), 2025. arXiv:2410.12032.

[2] D. Kanter, M. Ahmad, H. Kassa, and S. Rishab, “MLPerf Training v4.1 Results — Press Briefing,” MLCommons, Nov. 13, 2024. [Online]. Available: https://docs.google.com/presentation/d/1KSIJBvIV9OcswF1mVbhGRN0nUWbSgbYSoxu6dHCwajM/

The post MLPerf Endpoints v0.7: A Foundation Release appeared first on MLCommons.

For this round we debut the Qwen3.6-27B model (released April 22, 2026), served as a Q4_K_M GGUF quantization (reference implementation’s format – submitters may use any allowed quantization that meets the accuracy threshold) on a single edge accelerator, paired with two complementary workloads: the Berkeley Function Calling Leaderboard v4 (BFCL v4) for a deterministic, judge-free accuracy gate, and a recorded agentic-coding replay for single-stream performance. Together they capture the messy reality of on-device agents: long, growing tool

calling conversations that must fit within a hard context wall and feel responsive to a single interactive user. 

The submission deadline is July 31, 2026. We invite all hardware vendors, edge-device makers,  and inference-system experts to submit results and help raise the bar for on-device agentic inference. 

Why Agentic AI at the Edge 

The shift from single-shot text generation to multi-turn agentic workloads is one of the most consequential trends in applied AI. A real agentic session is not one prompt and one answer – it is a trajectory where each turn appends tool outputs and model responses to a growing conversation history, the model alternates between short tool calls and longer reasoning, and every turn depends strictly on the one before it. 

The datacenter MLPerf Agentic benchmark (coming up in September) measures this at scale – large mixture-of-experts models, long-horizon trajectories whose cumulative multi-turn context runs to 100K+ tokens, and a concurrency sweep that produces a per-GPU Pareto frontier. The  edge regime is the opposite problem, and it is what this benchmark targets: 

• A fixed memory and power budget: A single small accelerator (with a limited amount of memory) must hold weights, KV cache, and activations. Full-precision frontier models do not fit, so the model is quantized.

• A single in-flight request: Edge serving is dominated by one interactive user, not thousands of concurrent sessions.  

• A fixed served context window. The reference fixes the served context at 32K tokens – a controlled benchmark parameter, not a device ceiling – so long trajectories can exhaust it. This makes context growth and truncation first-order, measured effects of the agentic workload rather than something that varies with per-device memory. 

• Single-user latency, not aggregate throughput. On a single-slot edge device, throughput and latency are reciprocal to each other, so the metrics that matter are per-turn TTFT,  TPOT, and end-to-end turn latency on one device – not the aggregate QPS/token throughput a multi-stream datacenter server optimizes for. 

This benchmark inherits the multi-turn agentic methodology – terminology, deterministic replay,  the JSONL dataset schema, and the inline accuracy check – from the datacenter agentic specification and specializes it to the edge: an edge model and quantization, a single-stream load pattern, latency-centric metrics, and a statistically robust accuracy gate. 

Model Selection 

The reference model is Qwen3.6-27B (https://qwen.ai/blog?id=qwen3.6-27b), served with  llama.cpp (commit cfff1fc) as a Q4_K_M GGUF quantization (https://huggingface.co/unsloth/Qwen3.6-27B-GGUF). Qwen3.6 was selected because it is an open (Apache 2.0), capable tool-calling model with native MTP speculative-decoding head that punches above its size – a dense 27B that Alibaba reports (https://qwen.ai/blog?id=qwen3.6- 27b) beats the prior 397B-MoE flagship on major coding benchmarks (77.2% SWE-bench  Verified), yet deploys easily: official weights on Hugging Face and ModelScope plus community  GGUF builds (e.g. unsloth) that run directly under llama.cpp. 

Quantization is the defining edge choice. Q4_K_M is a 4-bit K-quant “Medium” tier: weights are stored at 4 bits/value (a 4× memory reduction versus BF16), attention blocks are kept at higher precision than the feed-forward blocks, and importance-weighted (imatrix) calibration preserves the highest-impact weights. The result fits the 27B model in ~16.5 GB of VRAM  (versus ~54 GB for BF16) at a ~2–5% accuracy cost – the difference between “runs on an edge  GPU” and “does not.” The exact GGUF used for the reference runs is Qwen3.6-27B-Q4_K_M.gguf from unsloth/Qwen3.6-27B-GGUF. 

Reference sampling parameters used are listed in the table below:

max_new_tokens 1024 

Reasoning is deliberately disabled on this tool-calling workload; it lowers single-turn accuracy while costing far more time (see Accuracy Metric section). The benchmark has been validated across a range of edge accelerators from multiple vendors, confirming that the workload and methodology are portable. 

The Benchmark Task 

The benchmark uses two datasets with distinct roles. 

Accuracy dataset: The Berkeley Function Calling Leaderboard v4 (BFCL v4) tests whether the model calls functions correctly, deterministically, and without an LLM judge. It spans single-turn requests (one prompt → one structured tool call) in three categories – non_live and live (graded by AST match against gold labels) and hallucination (a binary check that the model refuses to call a function when the available tools are irrelevant) – and optional multi-turn agentic conversations against in-process Python simulators. The categories are drawn from the public gorilla-llm/gorilla-eval-set (auto-downloaded at runtime). The scored gate is the single turn set only, per-category sampled to a stable ~995-sample point estimate: non_live 72%  (~712 samples), live17% (~171), and hallucination11% (~112), with a subset_floor of 25 so any subset of ≤ 25 entries is taken in full. 

Performance dataset. A subset of the MLPerf Agentic benchmark – recorded multi-turn agentic coding trajectories (SWE-bench-style tasks drawn from real repositories e.g., astropy), replayed deterministically as a performance workload, and scaled for single-device edge serving. The reference set comprises 20 conversations / 1,007 turns, sized so that no trajectory overflows the 32K-token served context (peak input length ~23.5K tokens). Because nothing overflows, every turn completes: the run is valid with zero dropped turns, and a single pass at concurrency 1 (one request in flight at a time) completes in a reasonable time on a single edge device. The dataset also serves as its own ground truth – each trajectory’s recorded tool calls drive a zero-cost inline accuracy check (multiset IOU of the executed calls) that runs during the latency measurement,  so correctness and latency come from the same pass. 

Example Request and Response 

A single-turn function-calling request is sent to the model’s OpenAI-compatible  /v1/chat/completions endpoint, carrying the available tool schemas:

}, 

 { 

 "role": "user", 

 "content": "What is the current temperature in San Francisco, in Celsius?" 

 } 

 ], 

 "tools": [ 

 { 

 "type": "function", 

 "function": { 

 "name": "get_current_weather", 

 "description": "Get the current weather for a given location.", 

 "parameters": { 

 "type": "object", 

 "properties": { 

 "location": { 

 "type": "string", 

 "description": "City and state, e.g. 'San Francisco, CA'" 

 }, 

 "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] } 

 }, 

 "required": ["location", "unit"] 

 } 

 } 

 } 

 ] 

} 

The model responds with a structured tool call: 

{ 

 "choices": [ 

 { 

 "message": { 

 "role": "assistant", 

 "tool_calls": [ 

 { 

 "type": "function",

 "function": { 

 "name": "get_current_weather", 

 "arguments": "{\"location\": \"San Francisco, CA\", \"unit\": \"celsius\"}"  } 

 } 

 ] 

 }, 

 "finish_reason": "tool_calls" 

 } 

 ] 

}

BFCL v4’s AST checker compares the predicted call – function name and each argument – against the gold label, so scoring is exact and reproducible. In the agentic-coding performance  workload, the same mechanism applies to multi-turn trajectories: the model emits bash tool  calls, the runner records them, and the inline checker scores the multiset intersection-over union (IoU) of the canonicalized shell executables against the recorded ground-truth trajectory.

Performance Metrics 

The edge benchmark reports single-stream, per-turn latency on a single accelerator rather than a per-GPU throughput frontier. A single simulated user replays trajectories closed-loop: send a turn, wait for the full streamed response, pause for the pre-recorded inter-turn delay, send the next turn – exactly one in-flight request at all times (target_concurrency = 1, one worker, one connection), matching the single-slot llama.cpp server. The reported metrics are TTFT (time to first token), TPOT (time per output token), end-to-end latency per turn, and ISL/OSL  distributions, as p50/p90/p99/max. 

Accuracy Metric 

Accuracy is the scored BFCL v4 single-turn gate, chosen for statistical robustness: the full set of single-turn categories is sampled to ~995 samples, completing in a reasonable time on an edge device.  Single-turn was chosen because its score is stable at large n and across devices, whereas a small multi-turn gate is brittle – an n=57 multi-turn gate carries a ~±11.6 pp 95% confidence interval,  and Q4 quantization-kernel differences cause per-entry verdicts to flip across devices. Multi-turn is retained as an optional, non-scored reference artifact. 

Reference single-turn accuracy (Qwen3.6-27B-Q4_K_M, seed 42, temperature 0; hardware independent, and identical across two seeded runs with a freshly restarted server each pass): 

There are two gated metrics: Overall, the sample-weighted accuracy across all ~995 single-turn samples, and Normalized, the category-balanced mean of the three category scores (so each category counts equally regardless of size). The per-category rows are components, not gates. 

The pass/fail criterion is a 3% one-sided band applied to both gated metrics: a submission passes only if its Overall and Normalized single-turn scores are each ≥ 0.97 × their respective references, with no upper bound (a higher score never fails).  

Reasoning off is the reference submission config. On this tool-calling workload, enabling server-side reasoning gives no accuracy benefit and costs far more wall-clock: on the accuracy gate, reasoning-off scores 86.23% versus reasoning-on’s 78.19% – an 8-point drop – and on the performance replay, reasoning-on moves inline IoU by only +0.004 (0.6374 vs 0.6335) while taking ~60% longer run-time.  

Join the Benchmark 

This is your opportunity to show how your edge hardware and software stack runs real multi-turn tool-calling workloads – on both accuracy and latency. Submissions for MLPerf Inference v6.1 are open to MLCommons members. Join MLCommons to submit your results by July 31, 2026.

• Model: Qwen/Qwen3.6-27B, served as a Q4_K_M GGUF – Qwen3.6-27B-Q4_K_M.gguf from  unsloth/Qwen3.6-27B-GGUF. 

• Accuracy dataset: gorilla-llm/gorilla-eval-set (BFCL v4 single-turn), auto-downloaded at  runtime. 

• Reference implementation: mlcommons/endpoints – examples/10_Edge_Agentic_Example/.  A single mandated config (online_edge_full_run.yaml) runs performance and accuracy back-to-back against the same server; –accuracy-only runs the gate alone.

• Submission deadline: July 31, 2026. 

The reference implementation drives any OpenAI-compatible Chat Completions endpoint, so you can benchmark arbitrary inference systems – not just the llama.cpp reference server.  Anyone with an edge accelerator can reproduce the reference accuracy and latency numbers by following the example README. We look forward to your submissions! 

Acknowledgements 

We thank the MLCommons Edge LLM Taskforce and the MLPerf Inference Working Group for their feedback and guidance in developing this benchmark.

The post Call for Submission: Edge Agentic Inference Benchmark for MLPerf Inference v6.1 appeared first on MLCommons.

The MLPerf Inference benchmark suite must evolve alongside AI deployment patterns. Early inference benchmarks focused on image classification, object detection, speech recognition, recommendation, and single-turn language generation. Those workloads remain important, but they no longer cover one of the fastest-growing ways large language models are used in production: multi-turn agentic inference.

For example, a coding assistant is far more complex than a single query. The agent will read an issue, inspect files, run commands, observe failures, edit code, and iterate potentially many times. Similarly, a workflow agent gathers customer information, calls tools, interprets results, asks follow-up questions, and continues until the task is resolved. In both cases, the workload is a trajectory: a sequence of dependent turns where each request includes the conversation history up to that point.

This changes the serving problem in four concrete ways:

• Context grows across the trajectory, so prefill and KV-cache pressure increase over time.
• KV-cache reuse is now a critical serving optimization for performance and efficiency.
• Output lengths vary widely, from compact tool calls to long reasoning traces.
• Turn dependencies make throughput a measure of closed-loop progress, not independent request rate.

Figure 1. Illustration of an agentic inference scenario involving multiple turns.

The Agentic Inference benchmark adds this class of workload to the MLPerf Endpoints framework. It keeps the general MLPerf measurement principles intact while defining the workload-specific pieces: model choice, dataset composition, multi-turn load generation, output validation, and constraints for optimizations such as prefix caching and speculative decoding.

Terminology

• Turn: one client-issued user or tool request and the model response generated for that request.
• Trajectory: the ordered sequence of turns for one task or simulated user; later turns include the accumulated conversation history.

Model selection

The benchmark needs long-context, thinking-capable LLMs that stress the serving behavior of agentic applications: growing context, KV-cache reuse, variable output lengths, and strict turn dependencies. For this benchmark, we chose Kimi K2.6 and Qwen3.6-35B-A3B. Kimi K2.6 brings state-of-the-art coding capability and a large model profile representative of leading agentic systems; Qwen3.6-35B-A3B is more compact and brings a new Gated DeltaNet (GDN) architecture and exceptional coding performance for its size. This gives the benchmark coverage across distinct serving behaviors, architectural choices, and speculative-decoding paths. Both models are evaluated with the same methodology and dataset. To avoid ambiguity, each model produces its own result; the two models are not run together or combined into a single score.

Table 1. Model metadata and benchmark settings

Dataset and task selection

The benchmark dataset combines two agentic domains that exercise different infrastructure bottlenecks. Together, they contain 613 multi-turn trajectories: 113 agentic coding trajectories and 500 agentic workflow trajectories. Across the reference dataset, these trajectories contain 30,335 client-issued turns and 30,328 assistant turns. By using real collected traces for benchmarking, the workload simulates real-world behavior in the serving stack, including how speculative decoding and expert-rank balancing behave under realistic multi-turn traffic.

Table 2. Dataset composition and mean token statistics

The agentic coding traces come from the DeepSWE dataset by DataCurve (datacurve.ai), which contains software engineering tasks built around repository-level bugs and feature requests. A typical trace begins with a user request describing an issue, followed by an assistant that investigates the repository through a series of bash commands. The agent searches files, reads code, runs tests, observes errors, edits the implementation, and iterates. These traces are deep: the median trajectory contains dozens of turns, and the conversation history grows steadily as command outputs, file contents, and test logs accumulate.

The Workato agentic workflow traces come from enterprise customer-support and orchestration scenarios. They represent interactions where a simulated customer asks for help and the agent uses tools to retrieve orders, track shipments, check policies, escalate cases, or resolve account questions. These trajectories are usually shallower than the coding traces, but they include a much larger shared system prompt with many tool definitions and business rules. The agentic workflow traces were provided by Workato, the agentic control and execution plane for the enterprise. The data are synthetic traces modeled on Workato’s production experience orchestrating customer support agents for enterprise customers.

The two domains are intentionally different:

• Coding traces grow aggressively over many turns.
• Workflow traces begin with a large common prefix and grow more slowly.
• Coding stresses KV-cache capacity and long-context scheduling.
• Workflow stresses shared-prefix reuse and routing locality.
• The combined workload prevents systems from optimizing for only one agentic traffic shape and stresses context-aware routing.

Client Design

We introduced multi-turn support to MLPerf Endpoints so the benchmark can drive full end-to-end agentic workloads against standard serving stacks. Submitters simply need to spin up an OpenAI-compatible endpoint using vLLM, SGLang, TensorRT-LLM, or another serving framework, then point the client at that endpoint for full end-to-end benchmarking.

The client handles the multi-turn behavior:

• Closed-loop replay: one active conversation issues one turn at a time and waits for the complete model response before the next turn.
• Target concurrency: the load generator controls the number of active users or conversations without breaking turn dependencies.
• Inter-turn delay: dataset-provided waits before tool or user turns preserve realistic pacing without counting delay as model serving latency.
• Conversation-aware routing: a stable X-Session-ID header is sent for each trajectory so routers can preserve KV-cache locality.
• Cache salting: deterministic salt markers around the system prompt allow valid within-trajectory reuse while blocking invalid cross-trajectory reuse to ensure the benchmark is representative of production workloads.
• Deterministic prompt reconstruction: future prompts are built from the pre-recorded dataset rather than from live model output, which keeps performance runs reproducible while still measuring generated outputs.
• Generated token cache clearing: In order to enable fair benchmarking of all platforms, the generated tokens are cleared out of the KV cache by introducing a whitespace character – ensuring that the performance is independent of the system that was used for generating the traces.

Performance metrics

The primary performance chart is a Pareto curve where each point is a fixed-concurrency benchmark result. Submitters need to submit a full Pareto curve as defined in the MLPerf Endpoints rules.

• Y-axis: output tokens per second per system, measuring aggregate serving throughput.
• X-axis: output tokens per second per user, computed as output tokens per trajectory / (E2E time – total delays).

The two axes represent complementary views of the serving system. The x-axis, output tokens per second per user, captures how quickly an individual agent progresses through its task; moving right means faster task completion. The y-axis, output tokens per second per system, captures the total work completed across all active agents; moving up means greater aggregate throughput and higher system utilization. As concurrency increases, the system may complete more total work while each individual agent progresses more slowly. The Pareto curve makes this tradeoff visible.

Figure 2. Chart illustrating agentic inference performance

Accuracy metrics

Accuracy is enforced as a three-level hierarchy so the Pareto curve measures usable agentic progress rather than responses that are shorter, lower quality, or easier to serve.

• OSL mean value: prevents systems from gaining throughput by truncating responses or shifting the answer-length distribution. The check requires generated outputs to preserve the expected mean output sequence length for the workload.
• Inline accuracy: catches behavior drift during the exact performance configuration being reported. It operates on the outputs produced by the performance run itself and compares them against ground-truth values preserved in the dataset, including coding tool-call action checks and workflow intent-code checks, so no separate inference pass is required.
• Standalone accuracy: verifies task-level capability beyond turn-level similarity. The standalone evaluation will use 200 tasks from the SWE-bench Verified dataset to check whether the submitted model configuration can solve representative tasks end to end.

All three levels must be run for every point on the Pareto curve, and each score must meet a pre-defined threshold. Thresholds are defined separately for each level and for both Kimi K2.6 and Qwen3.6-35B-A3B.

Table 3. Tentative accuracy values, subject to change.

Reference implementation

Submitters only need to spin up an OpenAI-compatible server using vLLM, SGLang, TensorRT-LLM, or another serving framework, then point the MLPerf Endpoints client at that endpoint to run the full benchmark. The MLPerf Endpoints Agentic Inference README includes the example configs, command lines, and setup steps.

Conclusion

Agentic inference is one of the fastest growing applications of AI today and it’s far more complicated than existing single-turn text generation. Agentic workflows have serving patterns with unique constraints: growing context, strict turn dependencies, tool-mediated delays, shared prefixes, long-tail trajectories, and output quality checks that must run on the same configuration used for performance.

The Agentic Inference benchmark brings those constraints into MLPerf in a reproducible form. By combining deep coding trajectories with shared-prefix workflow trajectories, it measures whether serving systems can make progress for real multi-turn users while still using hardware efficiently.

As agentic applications become a larger share of production LLM traffic, the industry needs benchmarks that measure the systems work that actually matters: preserving cache locality, scheduling long contexts, maintaining output quality, and balancing aggregate throughput with per-user progress. This benchmark is a step toward that measurement standard. If your team is building serving infrastructure for multi-turn agents, join MLCommons to access the reference implementation and submit results

References

• MLPerf Endpoints repository.
• MLPerf Endpoints Inference Rules and Pareto collection methodology.
• MLPerf Endpoints Agentic Inference reference implementation.
• SWE-bench Verified benchmark.
• Workato agentic workflow and enterprise task definitions.
• DeepSWE Dataset by Datacurve.

The post Agentic Inference for MLPerf Inference appeared first on MLCommons.

Five of the eleven datacenter tests in MLPerf Inference v6.0 are new or updated, and the release also includes a new object-detection test for edge systems. The major changes include:

●      A new, open-weight large-language model benchmark based on GPT-OSS 120B that can be used for mathematics, scientific reasoning, and coding;

●      An expanded DeepSeek-R1 advanced-reasoning benchmark, including an interactive scenario that permits speculative decoding;

●      DLRMv3, the third generation of our recommender benchmark and now the first sequential recommendation benchmark test in the suite, which is thoroughly modernized based on generous engineering contributions from Meta, a world leader in recommender systems;

●      The suite’s first text-to-video generation benchmark;

●      A new vision-language model (VLM) benchmark that transforms unstructured multimodal data from Shopify’s extensive product Catalog into structured metadata;

●      An upgraded single-shot object detection benchmark for edge scenarios based on Ultralytics’ YOLOv11 Large model.

“This is the most significant revision of the Inference benchmark suite that we’ve ever done,” said Frank Han, Technical Staff, Systems Development Engineering at Dell Technologies and MLPerf Inference Working Group Co-chair. “The decision to update so many benchmarks in this round was prompted by the extraordinary enthusiasm and collaboration from our members, who contributed an unprecedented amount of engineering effort and IP toward building new inference benchmarks. Adding these new tests allows MLPerf Inference to better keep pace with the breakneck pace of evolution in AI models and techniques so that our benchmarks are relevant and representative of real-world deployments.”

The open-source MLPerf Inference benchmark suite measures system performance in an architecture-neutral, representative, and reproducible manner. The goal is to create a level playing field for competition that drives innovation, performance, and energy efficiency for the entire industry. The published results provide critical technical information for customers who are procuring and tuning AI systems. 

“We thank Meta, Shopify and Ultralytics for their substantial collaboration with us in making these changes to the MLPerf Inference benchmark suite and for contributing their datasets, task definitions and expertise,” said Miro Hodak, Senior Member of Technical Staff at AMD and MLPerf Inference Working Group Co-chair. “These partnerships were essential in ensuring that the tests include scenarios and workloads that represent the current state of the industry.”

“MLPerf Inference benchmarks play a vital role in driving transparency and accountability across the AI industry,” said Glenn Jocher, CEO & Founder of Ultralytics. “At Ultralytics, rigorous, reproducible benchmarking is central to how we develop and validate our Ultralytics YOLO models — ensuring developers and organizations can make informed decisions about real-world performance. We’re proud to be part of an ecosystem that holds the entire field to a higher standard.”

“Commerce is one of the most complex domains in AI, yet researchers rarely have data that reflects that complexity,” said Kshetrajna Raghavan, Principal Engineer, Applied ML at Shopify. “Shopify is uniquely positioned to address this, sitting at the intersection of millions of merchants and billions of products. Sharing this taxonomy allows the whole field to evolve.”

New tools for submitters and consumers

With Inference 6.0, submitters have the option to use a newly available harness to complete benchmark tests. The new system, LoadGen++, allows LLMs to run with a serving-style software stack, which is familiar from typical deployments today. “LoadGen++ is a significant upgrade from its predecessor, and represents an important investment by MLCommons that will allow us to stay nimble as we continue to produce benchmark tests that track the state of the art,” said Han.

In addition, the Inference 6.0 results can be viewed in a new online dashboard on the MLCommons site. The dashboard brings new levels of interactivity to viewing results, including advanced filtering and customized performance graphs.

Large-scale, multi-node systems gaining attention 

The submissions to Inference 6.0 demonstrate that technology providers want to showcase the performance of scaled-up, multi-node systems running real-world inference workloads. This round recorded a new high for multi-node system submissions, a 30% increase over the Inference 5.1 benchmark six months ago. Moreover, 10% of all of the submitted systems in Inference 6.0 had more than ten nodes, compared to only 2% in the previous round. The largest system submitted in Inference 6.0 featured 72 nodes and 288 accelerators, quadrupling the number of nodes in the largest system in the previous round. 

“As more AI applications have moved into production and wide availability, the demand for large-scale, high-performance systems to run them has grown,” says Hodak. “At the same time, multi-node systems bring a unique set of technical challenges beyond those of single-node systems, requiring configuration and optimization of system architectures, network interconnects, data storage, and software layers. Stakeholders are eagerly stepping up to meet these challenges and run inference workloads at scale.”

The AI community continues to embrace and invest in MLPerf Inference

The MLPerf Inference 6.0 benchmark received submissions from a total of 24 participating organizations: AMD, ASUSTeK, Cisco, CoreWeave, Dell, GATEOverflow, GigaComputing, Google, Hewlett Packard Enterprise, Intel, Inventec Corporation, KRAI, Lambda, Lenovo, MangoBoost, MiTAC, Nebius, Netweb Technologies India Limited, NVIDIA, Oracle, Quanta Cloud Technology, Red Hat, Stevens Institute of Technology, and Supermicro. 

“I would like to welcome our first-time submitters, Inventec Corporation, Netweb Technologies India Limited, and Stevens Institute of Technology,” said Han. “The AI ecosystem is large and diverse, and it continues to grow and evolve rapidly. On behalf of MLCommons, I want to also thank our members, our contributors, and our partners including Meta, Shopify, and Ultralytics, for collaborating with us to build and shepherding forward the most comprehensive and relevant performance benchmark suite for AI inference. Together, we are ensuring that stakeholders in our community have valuable, real-world information that helps them to make better decisions.”

View the results

To view the results for MLPerf Inference v6.0, please visit the Datacenter and Edge benchmark results pages. For an interactive view of datacenter results, access the new benchmark results dashboard: https://mlcommons.org/visualizer. 

About MLCommons

MLCommons is the world’s leader in AI benchmarking. An open engineering consortium supported by over 130 members and affiliates, MLCommons has a proven record of bringing together academia, industry, and civil society to measure and improve AI. The foundation for MLCommons began with the MLPerf benchmarks in 2018, which rapidly grew into a set of industry metrics for measuring machine learning performance and promoting transparency in machine learning techniques. Since then, MLCommons has continued to use collective engineering to build the benchmarks and metrics required for better AI – ultimately helping to evaluate and improve the accuracy, safety, speed, and efficiency of AI technologies.

For additional information on MLCommons and details on becoming a member, please visit MLCommons.org or email participation@mlcommons.org.

Press Inquiries: contact press@mlcommons.org

The post MLCommons Releases New MLPerf Inference v6.0 Benchmark Results appeared first on MLCommons.

The MLPerf® Inference v6.0 release marks a significant expansion in our coverage of the open-weight large language model (LLM) landscape. As the industry moves toward more specialized and capable open models, the benchmarks must evolve to reflect these shifts in deployment strategies and model architectures.

In this round, the Reasoning LLM task force introduces two major additions to the suite:

• GPT-OSS 120B: a new benchmark based on a popular open-source, high-capability model which excels at mathematics, scientific reasoning, and coding. It features a large-scale mixture-of-experts (MOE) architecture of 117B total parameters. 
• DeepSeek-R1 interactive scenario: Building upon the existing DeepSeek-R1 benchmark, we add a low-latency-constrained interactive workload targeting real-time reasoning applications. This workload also features the first standard for speculative decoding in MLPerf.

A new benchmark with GPT-OSS 120B

GPT-OSS 120B is a popular new open-source, high-capability model, featuring a mixture-of-experts (MoE) architecture with 117B total parameters (5.1B active per token). This model natively supports configurable reasoning effort levels and is deployed across a diverse range of complex, knowledge-intensive workflows including advanced coding, competition mathematics, and graduate-level scientific logic. 

Frontier models are often deployed across a wide spectrum of production workloads—from fast, routine requests to complex, multi-step problem solving. To accurately reflect this duality, we introduced a split-dataset strategy: 

• Performance dataset of routine, low-effort usage tasks (e.g., summarization)
• Accuracy dataset of difficult reasoning problems across coding, scientific knowledge and math.

Across both modes, all inputs utilize the OpenAI Harmony chat format, allowing us to natively control the model’s reasoning effort (low, medium, or high) directly through system prompts.

Dataset selection for GPT-OSS

For the first time in MLPerf inference benchmark we have decided to separate performance and accuracy datasets. In all existing MLPerf Inference benchmarks, a single dataset is used for both. However, separating the two brings flexibility that allows agility in benchmark definition for this workload and also in the future. Additionally, the two tasks are different, and the separation allows the use of an optimal dataset for each task. 

A key consideration in the task force deliberation has been to ensure consistency between performance and accuracy runs. This issue was resolved by adding a new compliance test that verifies accuracy while running in the performance mode. 

Accuracy mode (high reasoning effort)

To ensure the benchmark sticks to a fair accuracy baseline, we curated a composite dataset requiring high reasoning effort.

• Max output length: 32,768 tokens.
• Evaluation strategy: Pass @ 1 with k repeats.
• Datasets:

• AIME 2024: Advanced mathematics problems. Metric: Exact Match.
• LiveCodeBench v6: Real-time coding tasks. Metric: Pass/Fail.
• GPQA-Diamond: Graduate-level science QA. Metric: Correct/Not Correct.

The evaluation and dataset curation was based on OpenAI’s official evaluation scripts. We created a feature branch (feat/mlperf_integration) that:

1. Enabled inference on tokenized inputs (HarmonySampler), and
2. Added evaluation for LiveCodeBench v6

To curate the accuracy dataset, we then collected input traces from running `gpt_oss.evals` and ran multiple times to get a reliable accuracy threshold on AIME25, LCB_V6 and GPQA_Diamond.

Accuracy targets for GPT-OSS

To qualify for submission, implementations must meet or exceed the following accuracy targets on the curated accuracy dataset:

Performance mode (low reasoning effort)

For measuring pure inference speed (tokens/second), we utilize a dataset sampled from ccdv/pubmed-summarization.

• Task: PubMed health article summarization.
• Configuration: “Low reasoning effort” via Harmony format.
• Sequence lengths: Max output length is set to 10,240 tokens.
• Metrics: Throughput and Latency.
• Mean input sequence length: 5,000 tokens
• Mean output sequence length: 1,250 tokens

Performance metrics

The performance metrics evaluate the systems under strict latency constraints depending on the deployment scenario.

GPT-OSS constraints:

• Interactive Scenario: 99th percentile TTFT <= 2.0s; TPOT <= 15ms.
• Server Scenario: 99th percentile TTFT <= 3.0s; TPOT <= 80ms.

Accuracy metrics

For GPT-OSS, accuracy is evaluated strictly on the accuracy mode dataset. AIME 2024 is evaluated via Exact Match, LiveCodeBench v6 via Code Execution (Pass/Fail), and GPQA-Diamond via Correct/Not Correct mapping.

Compliance checks

Because this benchmark uses separate datasets for accuracy and performance evaluations, new compliance checks were introduced to ensure accuracy when running in performance modes. The tests are as follows:

1. TEST07: This test verifies the accuracy of a performance run by using the GPQA dataset, one of the three accuracy datasets. Using all three datasets would be too computationally expensive, and the task force decided to use a subset for enforcing the accuracy verification.
2. TEST09: This test verifies that the mean output length generated from the performance dataset is within 10% of the output length from reference implementation.  

Reference implementation

The official reference implementations for the MLPerf Inference v6.0 benchmarks provide the necessary code and instructions to run the end-to-end evaluations.

• GPT-OSS 120B: GitHub link

A new workload: the DeepSeek-R1 interactive scenario

Building on the DeepSeek-R1 benchmark introduced in v5.1, we added an interactive scenario designed to represent the growing demand for low-latency responses in advanced reasoning uses cases like math, knowledge and reasoning, and complex coding tasks. The dataset (LiveCodeBench, MATH500, AIME, GPQA-Diamond, and MMLU-Pro) remains identical to the server scenario, with a minimum query count of 4,388, but the workload shifts to tighter bounded response times than server scenario.

Performance metrics

The performance metrics evaluate the systems under strict latency constraints depending on the deployment scenario.

DeepSeek-R1 Interactive Constraints & Speculative Decoding:

• New interactive scenario (Poisson arrival): 99th percentile TTFT <= 1.5s; TPOT <= 15ms.
• (Existing) server scenario (Poisson arrival): 99th percentile TTFT <= 2s, TPOT <= 80ms.

To meet the demanding latency requirements of the DeepSeek-R1 Interactive scenario, we are enabling speculative decoding for this specific workload. Implementations must use the official DeepSeek-R1 MTP (Multi-Token Prediction) Head with EAGLE-style decoding:

• Algorithm: EAGLE-style decoding with deepseek-ai/deepseek-r1 MTP head.
• Configuration: speculative-num-steps=3, speculative-eagle-topk=1.0.
• Prohibitions: Implementations cannot artificially manipulate acceptance rates. Techniques such as continued pre-training of the MTP head, quantization of the MTP head weights, or post-training adjustments (fine-tuning, RLHF) are strictly disallowed.
• See official rules here: mlperf-inference/policies

Accuracy metrics

For DeepSeek-R1 interactive, the accuracy metrics remain identical to the v5.1 server submission (exact match for math/QA and code execution for LiveCodeBench), ensuring the speculative decoding implementation does not degrade the model’s reasoning capabilities.

Reference implementation

The official reference implementations for the MLPerf Inference v6.0 benchmarks provide the necessary code and instructions to run the end-to-end evaluations.

• DeepSeek-R1: GitHub link

Conclusion

With MLPerf Inference v6.0, we continue to push the boundaries of what a standardized benchmark suite can measure. By introducing a split-dataset approach for GPT-OSS and a rigorous, latency-constrained Interactive scenario for DeepSeek-R1, we are providing the industry with the tools needed to evaluate the next generation of AI applications—from high-throughput summarization to real-time agentic reasoning.

We invite the community to explore the reference implementations and to participate in future rounds as we continue to track the rapid evolution of large language models.

For additional information on MLCommons and details on becoming a member, please visit MLCommons.org or email participation@mlcommons.org.

The post A new GPT-OSS benchmark and DeepSeek R1 updates for latency-optimized reasoning appeared first on MLCommons.

At GTC, MLCommons^® co-founder David Kanter unveiled the answer: MLPerf^® Endpoints, a ground-up rethinking of how the industry’s benchmark of record measures generative AI performance. With over 125 member organizations, more than 90,000 reproducible results to date, and recognition by IEEE and ISO/IEC SC42, MLPerf already underpins critical procurement decisions across government, industry, and academia. Endpoints is designed to keep that trust intact while adapting to a landscape that looks fundamentally different from what it was just two years ago. You can try it out here.

Why Traditional Approaches Need To Change

Traditional MLPerf inference benchmarks used a tightly coupled architecture: the load generator and model server ran as a single local process with shared dependencies. That worked well for classical ML, but generative AI deployments are API-first — whether on-prem, in the cloud, or via managed cloud endpoints. 

Meanwhile, measuring GenAI performance is harder than it looks. Real serving combines accuracy, latency, throughput, and sequence length into a non-linear, multi-dimensional surface. Long-tail queries, variable arrival patterns, and tight SLAs interact in ways that simple scenarios miss entirely.

An API-Centric Architecture

MLPerf Endpoints replaces the monolithic design with a decoupled client that communicates with any model-serving API endpoint via standard interfaces such as HTTP or gRPC. The benchmark client is lightweight and production-ready; the system under test is simply a URL. This means zero-effort integration for submitters — point the client at your endpoint and run. The architecture also enables benchmarking of managed cloud services alongside bare-metal deployments on an equal footing, something the previous framework could not easily support.

Under the hood, a new scalable load generator uses separate worker processes, pre-warmed connection pools, and ZeroMQ-based IPC to ensure the harness itself never becomes the bottleneck, even when testing rack-scale systems.

Pareto Curves and Step Functions: New Visualisation for New Metrics and Easy Comparison

One of the most compelling innovations is how results are presented. Each benchmark run varies concurrency and captures key metrics, including TTFT (time to first token), throughput (tokens per second), interactivity (tokens per second per user), and response latency. Submitters tune parallelism and batch settings for each operating point, and the visualizer plots these metrics as a Pareto curve (e.g., throughput vs. interactivity)—giving buyers an immediate picture of the real-world trade-offs, such as between serving more users and keeping each user’s experience responsive.

Crucially, MLPerf Endpoints uses step functions rather than interpolated trend lines. GenAI performance is highly non-linear; interpolating between measured points can suggest performance levels that were never actually achieved, masking memory overflows or P99 latency spikes. Step functions show only verified operating points, eliminating what the presentation aptly called “paper performance.” Customers can easily compare these step functions against each other and match these verified points to their own use cases — high concurrency during the day, best possible interactivity at night.

Rolling Submissions: Benchmarking at the Speed of Software Updates

Perhaps the boldest change is operational. MLPerf has historically published results on a fixed bi-annual schedule, depending on the benchmark offering (Training, Inference, Storage, etc). In a market where major model releases land every few weeks, that cadence is too slow for buyers writing RFPs and vendors launching hardware. Starting in Q2 2026, MLPerf Endpoints will move to continuous rolling submissions: submitters can publish peer-reviewed, audited results at any time. Incremental submissions let vendors start with a baseline Pareto curve and iteratively add more operating points as their software stack matures.

The approach is inspired by proven methodologies from other industry-standard bodies such as SPEC and TPC, adapted to the world of AI. Peer review and audit requirements will remain fully intact to deliver the robustness that the industry demands.

What Comes Next

The first MLPerf Endpoints v0.5 demonstration feature results from AMD, Google, Intel, KRAI, and NVIDIA, backed by over 30 supporting organisations, including Argonne National Laboratory, Broadcom, Dell, HPE, Lambda, Lenovo, Oracle, Red Hat, and the University of Florida. The results include models such as DeepSeek-R1, GPT OSS 120B, Llama 3.1 8B, QWEN 3 Coder 480B and more, running on nearly a dozen different systems.

Looking ahead, MLCommons is inviting the broader ecosystem to shape what comes next. Enterprise and IT buyers can join the advisory council. OEMs, CSPs, and ODMs can contribute results to the rolling leaderboard. Model developers and API providers can integrate next-generation SOTA models and build managed roadmaps. Researchers can anchor reproducible baselines using the Endpoints framework. New models — especially popular and commercially relevant ones — are continuously evaluated for inclusion. You can try it out for yourself here.

Get involved: MLPerf Endpoints rolling submissions open in Q2 2026. To participate, contribute, or learn more, visit https://mlcommons.org/benchmarks/endpoints/ or join our working group.

The post Standardizing Generative AI Service Evaluation: An API-Centric Benchmarking Approach appeared first on MLCommons.

The MLPerf Inference benchmark has evolved to an industry standard for measuring the performance of artificial intelligence (AI) infrastructure by creating a fair benchmarking platform and incorporating diverse workloads, including vision, speech, and natural language processing. MLCommons’ effort to stay relevant with the latest AI workloads is evident not only in the introduction of new models in the Generative AI space but also in upgrades to legacy workloads. The YOLO Task Force was formed to upgrade the RetinaNet benchmark in the edge suite to Ultralytics YOLO11, a more modern, state-of-the-art detection model. 

RetinaNet has been a solid, academically sound benchmark for single-shot detection for years, but there are several reasons to upgrade to a more modern YOLO (You Only Look Once) variant for the object detection workload [1]. YOLO has experienced rapid growth across research and real-world applications, driven by accelerated innovation, frequent releases, and strong community adoption – positioning it as one of the most effective and exciting models for modern object detection workloads. YOLO11, released by Ultralytics in September 2024, introduces substantial architectural and training improvements – achieving higher accuracy with fewer parameters and offering model variants ranging from YOLO11n (nano) to YOLO11x (extra large) that support diverse compute-accuracy trade-offs [2]. In contrast, RetinaNet has received fewer major updates in recent years, leading to diminished development momentum and reduced community adoption. Meanwhile, the YOLO family continues to evolve rapidly, reflecting cutting-edge advances and the prevailing trends in the AI industry’s object detection domain. 

Model selection 

Before YOLO, state-of-the-art detectors were “two-stage” systems that first proposed regions of interest and then classified them [1]. The YOLO model was first introduced in 2015 by Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi at the University of Washington. It fundamentally changed computer vision by treating object detection as a single regression problem—predicting bounding boxes and class probabilities simultaneously in a single pass—whereas previous models like R-CNN required thousands of separate passes per image [3]. The “single-shot” approach sacrificed a small amount of accuracy for a massive speedup, enabling real-time detection at 45 frames per second. Community-driven YOLO model improvements from organizations like Ultralytics improved subsequent versions.

Our initial challenge was balancing the proven stability of established versions with the cutting-edge accuracy of the latest releases. While legacy versions like Ultralytics YOLOv8 have solidified their place as the industry standard due to their robust anchor-free design and broad community support, we ultimately focused our evaluation on YOLO11 – and even peeked at the nascent Ultralytics YOLO26 – to ensure the benchmark remains future-proof. 

The technical analysis revealed that YOLO11 offers a significant leap in parameter efficiency and raw accuracy. For our benchmark, we selected the YOLO11l (large) variant, which achieves a 53.4% mAP on the COCO dataset, outperforming the YOLOv8l baseline’s 52.9%. The Mean Average Precision (mAP) score serves as the ultimate arbiter of quality because it balances precision (number of correct detections) and recall (objects found). This is achieved while maintaining a highly competitive footprint of 25.3 million parameters, a refinement achieved by replacing the older C2f modules with more efficient C3k2 blocks and integrating C2PSA (Cross-Stage Partial Spatial Attention), which enhances the model’s focus on salient regions without a proportional increase in computational cost [4]. 

Beyond the YOLO family, we weighed alternative modern architectures to ensure a comprehensive selection process. EfficientDet was noted for its strong accuracy-to-FLOPs ratio via the BiFPN (Bi-directional Feature Pyramid Network), though it often lacks the raw throughput required for high-velocity production environments [5]. Similarly, transformer-based detectors like DETR and Deformable successors are good for their streamlined, NMS-free training pipelines and superior global context [6]. Ultimately, YOLO11 Large was selected for its unparalleled production throughput and its ability to act as a rigorous stress test for hardware interconnects and data-loading pipelines. Choosing YOLO11 allows the MLPerf benchmark to reflect real-world deployment patterns while pushing vendors to optimize for high-efficiency, attention-augmented convolutional neural networks. 

Fig. 1. Schematic diagram of YOLO11 showing the Backbone, Neck, and Head components (adapted from A. T. Khan and S. M. Jensen, LEAF-Net, Dec. 2024). 

Dataset 

Choosing the correct dataset was another fundamental decision in the YOLO Inference benchmark integration task, as it serves as the ground truth for our benchmark’s validity. We selected COCO 2017 (Common Objects in Context) because it was used to train YOLO models and remains the gold standard for object detection. With 80 object categories and over 1.5 million instances, we can ensure that the model won’t just memorize shapes but also truly understand spatial relationships and the varied hidden components in realistic images [7]. The model was not trained by the group; we used a subset of the COCO validation dataset and verified that it maintained its original accuracy.

However, distributing a large-scale dataset for an open benchmark such as MLPerf poses legal and compliance challenges, particularly regarding commercial distribution rights. While the COCO annotations are open, some images from sources are marked “Non-Commercial,” which means they are not compliant with commercial benchmarking. To address this, the Task Force developed a custom filtering pipeline to create a safe subset of the full COCO 2017 dataset. This ensures that the final dataset used by our partners is fully distributable and legally safe for both academia and industry, without compromising the benchmark’s statistical integrity. 

The MLPerf Subset 

Table 1: The COCO MLPerf subset shows the total number of safe images compared to the full dataset.

Loadgen integration

After creating the  COCO dataset, we had to ensure consistent accuracy scores in our YOLO LoadGen integration. MLCommons LoadGen (Load Generator) is a reusable C++ library with Python bindings that effectively and fairly measures the performance of ML inference systems by generating standardized query traffic patterns to ping the model across scenarios such as SingleStream, MultiStream, and Offline. The LoadGen API records all queries and responses for verification and summarizes whether performance latency constraints are met. While remaining model-agnostic and not specifically handling accuracy evaluation, the LoadGen API generates files needed for model-specific accuracy metric calculations [8].

Originally, the YOLO11 implementation generated a standard predictions.json file. While this format is sufficient for general COCO validation, it was incompatible with the COCO MLPerf Accuracy script for several reasons:

• Class mapping inconsistency: YOLO models traditionally use an 80-class index (0-79) derived from the COCO dataset. However, for the MLPerf accuracy evaluation, we wanted to use the 91-category COCO mapping, where class IDs are non-sequential.
• Coordinate normalization: The standard YOLO outputs are often absolute pixel coordinates or in the XYWH (standard x, y coordinates, width, height) format. For our accuracy evaluation, we wanted to use a specific serialized payload: a 7-element float array containing [index, y_min, x_min, y_max, x_max, score, class]. Crucially, these coordinates must be normalized (between 0.0 and 1.0) relative to the original image dimensions.
• Buffer serialization: Unlike a standard JSON dump, the MLPerf LoadGen requires results to be serialized into a byte buffer through the QuerySamplesComplete API.

Once we identified and resolved the issues above, we achieved matching mAP results to the Ultralytics reference using our YOLO LoadGen implementation. We got it to work with the following strategy:

• Class re-indexing: We implemented a robust mapping layer using the official COCO 80-to-91 conversion array. This ensures that every detection reported by YOLO is immediately translated into the category ID space recognized by the accuracy script.

Target ID = COCO_{80_to_91}[Model Class Index]

• Coordinate transformation: The Runner.enque logic was rewritten to handle image geometry dynamically. We first extracted the original height (H) and width (W) from the Ultralytics results object to perform a real-time normalization of the XYXY bounding box:

y_min = box[1] / H

x_min = box[0] / W

y_max = box[3] / H

x_max = box[2] / W

• LoadGen native serialization: We utilized the struct.pack(“7f”, …) method to create a binary payload. This payload is then passed directly into lg.QuerySampleResponse, allowing the MLPerf LoadGen to handle the logging in its native binary format. This change not only fixed the accuracy reporting but also reduced I/O overhead during performance runs by eliminating the need to write to an intermediate file.

By shifting this logic directly into the LoadGen runner, we achieved seamless integration, with mlperf_log_accuracy.json generated to the exact schema required by the accuracy-coco.py script. This ensures that mAP calculations are bit-accurate and directly comparable with other submissions in the MLPerf Inference leaderboard.

Performance evaluation 

For the edge suite, the following performance metrics are measured for the standard MLPerf scenarios: 

1. Offline, where the goal is to measure peak throughput (samples/sec) by processing the entire dataset as a single batch. In this benchmark scenario, LoadGen will send all queries to the System Under Test (SUT) at the start.
2. Single-Stream focuses on the absolute minimum latency for a single image – critical for real-time edge responses. LoadGen sends the queries to the SUT as soon as the previous one completes. The duration of this scenario is set to 1024 queries and 60 seconds.
3. Multi-Stream simulates multiple channels and measures the maximum number of concurrent streams the system under test can support. LoadGen sends queries to the SUT using the same method as Single-Stream, but the run duration is longer, at either 270,336 queries or 600 seconds.

As a single-stage detector with an optimized detection head, YOLO11 eliminates several post-processing and anchor-heavy computation stages present in RetinaNet, ultimately demonstrating a significant improvement in end-to-end inference 

Accuracy metrics 

While RetinaNet provided strong detection accuracy due to its focal loss formulation and multi-scale FPN backbone, YOLO11 achieves superior mAP while also delivering on latency [9]. Our implementation uses the mAP@50-95 standard, a metric that computes the average precision across ten Intersection over Union (IoU) thresholds (0.5-0.95). By doing so, we want to see that the model not only identifies the correct class but also localizes it with high pixel-level accuracy. We found this to be a challenging yet fair set of parameters for the YOLO11l model to maintain high throughput while ensuring that its bounding box predictions are nearly identical to the ground truth. 

We established two distinct target accuracy thresholds: yolo-95 and yolo-99. These represent 95% and 99% of the state-of-the-art reference score, respectively. The yolo-95 threshold serves as the “default” mode, allowing for submissions that prioritize system speed and throughput for standard production needs. In contrast, the yolo-99 threshold represents the “high accuracy” standard for mission-critical applications, requiring the model to converge almost perfectly to the reference accuracy. The mAP scores for the two modes are determined by taking the YOLO11l mAP score from Ultralytics and multiplying it by 0.95 and 0.99 for the default and high accuracy modes respectively. The table below shows a score of 53.4 for YOLO11l from Ultralytics, which translates to 0.534 * 0.95 for the yolo-95 version and 0.534 * 0.99 for the yolo-99 version. By offering these two tiers, the MLPerf benchmark allows hardware vendors to showcase different optimization profiles. 

Table 2: Parameters and accuracy comparison of YOLO11 model variants.

Reference implementation 

The reference implementation for YOLO11 uses the official Ultralytics YOLO11 inference code. For readers interested in running the model in their own environment, we recommend following this MLCommons reference implementation, which includes the code and instructions for running the end-to-end benchmark, starting with the dataset and model downloads. 

Conclusion 

In summary, the transition from RetinaNet to YOLO11 marks a pivotal evolution in the MLPerf Inference benchmark, as the upgrade to a better model reflects industry trends. By adopting the YOLO11l variant, we are challenging hardware vendors to optimize for attention-augmented components like the C2PSA block. This shift ensures that our performance metrics are not just theoretical numbers, but actionable data points that translate directly to the efficiency and responsiveness of real-world AI use cases. 

Furthermore, the introduction of the yolo-95 and yolo-99 accuracy tiers provides a nuanced framework for AI infrastructure evaluation. It allows organizations to showcase the raw speed of aggressive quantization at the 95% threshold while maintaining a path for mission-critical, high-fidelity deployments at the 99% level. This benchmark will continue to serve as a positive step forward for the community, driving innovation and ensuring that the next generation of AI hardware is built to handle the complexity and scale of computer vision. MLCommons will continue working with domain experts and industry members to update the object detection models, ensuring that the MLPerf benchmark remains current and accurately reflects evolving industry trends.

Acknowledge

We would like to express our gratitude to the following individuals for their help and guidance during the development of this benchmark:

• Anandhu Sooraj (MLCommons)
• Arjun Suresh (MLCommons)
• Ashutosh Dhar (NVIDIA)
• Karl Pietri (MLCommons)
• Miro Hodak (AMD)
• Reilly Fairbanks (MLCommons)
• Scott Wasson (MLCommons)
• Zhihan Jiang (NVIDIA)

References

1. J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You Only Look Once: Unified, Real-Time Object Detection,” arXiv preprint arXiv:1506.02640, Jun. 2015. [Online]. Available: https://arxiv.org/abs/1506.02640
2. Ultralytics, YOLO11 Model Documentation. [Online]. Available: https://docs.ultralytics.com/models/yolo11/
3. R. Girshick, “Rich feature hierarchies for accurate object detection and semantic segmentation,” arXiv preprint arXiv:1311.2524, Nov. 2013. [Online]. Available: https://arxiv.org/abs/1311.2524
4. S. Author, T. Author, and L. Author, “YOLOv1 to YOLOv11: A Comprehensive Survey of Real-Time Object Detection Innovations and Challenges,” arXiv, Aug. 2025. [Online]. Available: https://www.arxiv.org/pdf/2508.02067
5. M. Tan, R. Pang, and Q. V. Le, “EfficientDet: Scalable and Efficient Object Detection,” arXiv preprint arXiv:1911.09070, Nov. 2019. [Online]. Available: https://arxiv.org/abs/1911.09070
6. N. Carion et al., “End-to-End Object Detection with Transformers,” arXiv preprint arXiv:2005.12872, May 2020. [Online]. Available: https://arxiv.org/abs/2005.12872
7. T. Lin et al., “Microsoft COCO: Common Objects in Context,” arXiv preprint arXiv:1405.0312, May 2014. [Online]. Available: https://arxiv.org/abs/1405.0312
8. E. Hall et al., “MLPerf Inference Benchmark,” arXiv, Nov. 2019. [Online]. Available: https://arxiv.org/pdf/1911.02549
9. T.-Y. Lin et al., “Focal Loss for Dense Object Detection,” arXiv preprint arXiv:1708.02002, Aug. 2017. [Online]. Available: https://arxiv.org/abs/1708.02002
10. **A. T. Khan and S. M. Jensen, “The schematic diagram of YOLOv11 illustrating its three core components: Backbone, Neck, and Head,” inLEAF-Net: A Unified Framework for Leaf Extraction and Analysis in Multi-Crop Phenotyping Using YOLOv11, preprint, Dec. 2024. Fig. 5. [Online Image]. Available:https://www.researchgate.net/figure/The-schematic-diagram-of-YOLOv11-illustrating-its-three-core-components-Backbone-Neck_fig1_386467184

The post YOLO for the MLPerf Inference v6.0 Edge Suite appeared first on MLCommons.

The MLCommonsⓇ MLPerfⓇ Inference benchmark suite is an industry standard for measuring the performance of machine learning (ML) and artificial intelligence (AI) workloads from diverse domains including vision, speech, and natural language processing. For each of these domains, the suite includes a carefully selected set of workloads that represents the state of the art in the industry across different application segments. These benchmarks not only provide key information to consumers for making application deployment and budget decisions but also enable vendors to deliver critical workload optimizations within certain practical constraints to their customers.

Over the past year, we have seen rapid advancements in the capabilities of video generative models like OpenAI Sora2. Gone are the days of hobbyists generating uncanny clips of Will Smith eating spaghetti; now professional artists create entire workflows around the capabilities of these models. As they now transition from being mere curiosities to becoming core parts of creative workflows, the need for a standardized benchmark has become clear. And thus the MLPerf Text-to-Video Task Force was convened to incorporate a dedicated video generation benchmark into the MLPerf suite.

Model selection

For this benchmark we picked the Wan2.2-T2V-A14B-Diffusers model (released July 2025) by Alibaba, as it was one of the best open weights models on the Text-to-Video leaderboard at the time. The model has been fully open sourced under an Apache 2.0 licence and can be run via Huggingface Diffusers.

The Wan2.2 model is run as a pipeline of 3 models:

1. The UMT5 XXL Text Encoder from Google, used to encode prompts
2. The Wan2.2 A14B Diffusion Transformer, used to generate a latent video representation
3. The Wan2.2 VAE Decoder, used to decode the latent video into a series of frames.

The crucial architectural feature of the Wan2.2-T2V-A14B-Diffusers model is that it is a type of Mixture of Experts model. However, unlike the standard MoE architecture, there is no gating network used to route tokens to experts, instead this model consists of 2 experts which are activated sequentially during the denoising process. The first expert is known as the “High Noise Expert” and is active during the early stages of denoising, after which the model switches to the “Low Noise Expert” which completes the denoising process.

Counterintuitively, most video generation models do not generate videos frame by frame, instead they generate the entire video at once, by denoising a massive video latent. This video latent usually represents both a spatial and temporal segment of the video, for example the Wan2.2 latent represents an area of 32×32 pixels across 4 frames. This means that generating a 5 second video at 720p resolution and 16fps would require a sequence length of 19,320. This leads to the model being heavily compute bound.

Performance metrics

One of the key difficulties we encountered when designing this benchmark was deciding which performance metrics to use. The text-to-video task is computationally expensive with long runtimes, with many systems taking multiple minutes per query.

One consequence of this performance profile is that we had to limit the length, resolution and number of videos generated to ensure that this benchmark remained feasible, whilst also demonstrating frontier capabilities. Therefore we made the following decisions.

• Configuration: We limited the length of the generated videos to 5 seconds, whilst fixing the resolution at 720p. This meant that we would generate 81 720×1280 frames at 16fps.
• Runtime Target: To ensure the benchmark remains accessible to a wide range of submitters, we reduced the dataset size to a practical subset (100 out of 248 samples for performance mode, keep 248 for accuracy mode).

Replacing Server with SingleStream

One of the key changes we’re introducing in this benchmark is the replacement of the Server scenario with a SingleStream scenario for latency measurement. We based this decision on the large amount of compute needed to generate a single video, meaning that videos often take multiple minutes to generate. 

This poses a problem with the Server scenario where the system is assumed to be able to operate in near real-time. In practice this would mean that the System Under Test would quickly become overloaded with requests, meaning that most requests would spend the majority of their time waiting to be processed instead of being processed. This in turn would mean that the final latency measurements wouldn’t accurately reflect the hardware’s performance.

To solve this problem, we replaced the Server scenario with a SingleStream one, where we would only measure the time taken to process requests, ignoring all wait times.

Dataset and Accuracy Metric and task selection

We selected VBench as the official dataset and accuracy framework for this benchmark. We reached this decision after a comparative analysis of available options, prioritizing licensing feasibility, robustness, and ease of adoption.

We evaluated a wide range of datasets at the beginning, including OpenVid-1M, VidGen-1M, WebVid-10M, and ActivityNet. A primary filter for selection was commercial viability, as MLPerf submissions often come from industry partners for commercial hardware validation.

• VidGen-1M and WebVid-10M were disqualified due to restrictive licensing (e.g., Non-Commercial or Research-Only terms), which leads to legal risks for benchmarking.
• OpenVid-1M offered a permissible license (CC BY 4.0) but functioned solely as a dataset and lacked an integrated evaluation framework.

VBench distinguished itself by offering a holistic solution that combined a diverse prompt set with a pre-validated scoring suite. Unlike raw datasets that would require us to independently develop and validate separate accuracy metrics (such as FVD or IS), VBench provided:

• Comprehensive Metrics: A suite of 16 distinct quality dimensions, including Subject Consistency, Motion Smoothness, and Aesthetic Quality.
• Standardized Prompts: A curated list of ~950 prompts designed to stress-test specific generation capabilities.
• Determinism: Experiments confirmed that VBench scores remained stable across different hardware backends (NVIDIA, AMD) with fixed seeds, a critical requirement for cross-vendor fairness.
• Widespread Adoption: VBench is a widely used benchmark by video model builders and has been cited in several technical reports, including that of the Wan model family.

While VBench provided the most robust framework, its default configuration was computationally expensive, requiring over 80 hours for a full inference pass. To align with MLPerf’s accuracy check runtime, we adapted VBench by:

• Subsetting the Dataset: We agreed to reduce to average the 6 key metrics: Subject Consistency, Background Consistency, Motion Smoothness, Dynamic Degree, Appearance Style, Scene.
  This selection focused on the most discriminatory dimensions, such as Dynamic Degree, Multiple Objects, and Scene Quality, while removing metrics that were redundant, static, or computationally trivial for datacenter-class hardware.
• Reducing Dataset Size: By focusing on these 6 metrics, the dataset was reduced to a statistically significant subset (248 samples), striking the necessary balance between rigorous accuracy validation and manageable submission runtimes.

VBench was chosen by the taskforce, because it offered the only commercially viable, legally cleared, and methodologically complete framework that could be adapted to meet the rigorous runtime constraints of the MLPerf Inference benchmark.

Reference implementation

To ensure a fair and reproducible benchmark, the reference implementation for the Text-to-Video task is built on a standardized open-source foundation. 

Here is the reference setup:

• Model Architecture: We utilize the Wan2.2-T2V-A14B-Diffusers model (hosted by Wan-AI). This is a 14-billion parameter Diffusion Transformer designed for high-quality video generation. 
• Precision & Compute: The reference implementation runs in BF16 (BFloat16) precision. This choice reflects modern datacenter standards, balancing numerical stability with efficient memory usage.
  • Reference Accuracy Score: 70.48 (VBench).
  • Minimum Accuracy Threshold: 69.77 (99% of reference).
• Generation Pipeline: The reference pipeline is adapted from the Hugging Face Diffusers library, ensuring broad compatibility and ease of use.
  • Input: Text prompt + Fixed Latent Tensor (to ensure deterministic outputs for debugging and verification).
  • Scheduler: Uses the UniPCMultistepScheduler (aligned with the Wan2.2 default) to optimize step efficiency.
  • Reduced diffusion steps (per TF discussion): 20 steps in diffuse process.
  • Output: 720px1280p resolution at 16 frames per second, 81 frames in a video.
• Containerization: To simplify deployment, the entire reference stack including Python dependencies, CUDA 12.1 libraries, and VBench evaluation tools is provided as a Docker container. This allows submitters to “build and run” the benchmark with a single launch.sh script.

Conclusion

With the introduction of the Text-to-Video task in MLPerf Inference v6.0, we have taken a significant step to generative video workloads. This benchmark provides the industry with a reliable, reproducible approach to measure the rapidly evolving capabilities of both hardware and software.

The architectural decisions behind this inaugural benchmark reflect the the real video generation tasks today:

• Model Selection: We selected Wan2.2-A14B-Diffusers, a powerful open-weights T2V model that represents the state-of-the-art in open-source generation.
• Performance: We adopted SingleStream as the primary metric, and keep the standard Offline scenarios. The high-fidelity video generation task is currently a compute-bound, high-latency task which fits best for the current use cases .
• Accuracy: We integrated VBench as our evaluation framework, ensuring that performance optimizations do not come at the cost of visual fidelity, motion coherence, or prompt adherence.

This release represents a foundational baseline, but the field is moving fast. As generation latencies drop from minutes to seconds, we expect that the benchmark will evolve to include Server Mode scenarios to reflect in real-time use cases.

The post Bringing Text-to-Video to MLPerf Inference v6.0 appeared first on MLCommons.

Scaling compute, guided by neural scaling laws, has dramatically reduced the need for manual feature engineering in natural language processing (NLP) and computer vision by using large-scale attention-based transformer models to automatically learn rich representations from data [1][2]. A similar trend is transforming deep learning-based recommendation systems, which have traditionally relied on architectures built from multi-layer perceptrons (MLPs), graph neural networks (GNNs), and embedding tables [3][4][5]. Recently, large sequential and generative models have been successfully deployed in online content recommendation platforms, delivering substantial improvements in model quality [5][7][8][9][10][11][12][13]. Given the global scale and importance of recommendation systems [6], incorporating such large-scale sequential recommendation models into the MLPerf Inference benchmark suite helps support ongoing infrastructure development.

We introduce DLRMv3, the first sequential recommendation inference benchmark in MLPerf’s DLRM family. DLRMv3 is built around an HSTU-based [5] architecture for ranking, capturing the dominant compute patterns of modern recommendation workloads: long input sequences, attention-heavy computation, and large embedding tables. Compared to the existing DLRM benchmark (DLRMv2 [14]), DLRMv3 increases the model size by 20X from 50GB to 1TB and raises the per-candidate compute by 6500X from 40M FLOP to 260 GFLOP over a span of just three years, aligning the MLPerf Inference suite with contemporary production-scale recommendation workloads, highlighting the increasing demand for compute. This higher compute regime is motivated by reported HSTU scaling behavior where higher model compute yields improved recommendation quality in production, enabling realistic resource burden–accuracy trade-off evaluation.

Task selection

Modern recommendation systems are typically deployed as multi-stage pipelines that separate candidate retrieval from ranking and, in some cases, additional refinement stages such as re-ranking or business-logic post-processing [15][16]. In a common design, a retrieval model first selects a small subset of relevant items from a very large corpus, optimizing for high recall, coverage, and strict latency and memory constraints at scale [15][7]. A downstream ranking model then scores these candidates using richer features and a more expressive architecture, optimizing for fine-grained user engagement metrics (e.g., click-through rate (CTR), watch time, satisfaction) under somewhat looser but still production-critical latency and throughput constraints [15][16][18]. This separation of stages is now standard in large-scale industrial systems, including those used for web, video, and social content recommendation.

DLRMv3 focuses on the ranking stage of this pipeline. Ranking models typically dominate the overall ML compute budget in production recommendation systems and are a focus of innovation in model architecture (e.g., attention-based sequential models and larger embedding tables), making them especially relevant for hardware and systems benchmarking. Focusing on ranking also maintains continuity with prior MLPerf DLRM benchmarks which target click-through-rate prediction.

Formally, given a user’s interaction history (e.g., a sequence of previously viewed or engaged items) and a candidate item, the DLRMv3 model predicts the probability of a desired outcome such as a click, like, or watch. This probabilistic prediction task is directly aligned with earlier DLRM benchmarks, which also model CTR-style binary outcomes.

Model selection

Figure 1. Model Architecture of different DLRM models.

We introduce an HSTU-based architecture as MLPerf’s third-generation deep learning recommendation benchmark (DLRMv3). In the DLRM lineage, DLRMv1 is built from MLPs and embedding tables with simple dot-product feature interaction, and DLRMv2 adds a deep-cross network component for richer explicit feature crossing. DLRMv3 introduces a new sequential feature transformation, interaction, and extraction component based on the hierarchical sequential transduction unit (HSTU) [5] while retaining a single, large embedding table and a top MLP for final prediction (Figure 1(c)).

HSTU-style architectures have been shown in production systems to effectively model long user interaction histories and improve recommendation quality relative to traditional MLP/DCN-based models at comparable or higher compute resources. They also reflect a set of compute characteristics that increasingly dominate modern recommendation inference—long sequences, attention-heavy computation, and large embedding tables—making HSTU a representative and forward-looking choice for a system-level benchmark.

In the table below, we compare model configurations among different generations of DLRM. The 260 GFLOP is computed as 2 * layers * (UIH_length * UIH_length * EmbDim / 2 + UIH_length * EmbDim * EmbDim * 4 + UIH_length * EmbDim * EmbDim * 3), which accounts for attention FLOP as well as the pre- and post-attention GEMMs. Note that this 260 GFLOP “per-candidate” number is an effective normalization: in a typical ranking request, the HSTU encoder processes the shared user-interaction-history (UIH) sequence once, and its output is reused when scoring a candidate set (2K candidates in DLRMv3), so the dominant UIH encoding computational cost is amortized across candidates rather than repeated 2K times. Moreover, because DLRMv3 uses a streaming time-series setup, deployments can reuse UIH-related KV states across consecutive timestamps for the same user, avoiding recomputation of the UIH encoding and reducing redundant dense compute by roughly 80–90% in steady state.

To better align DLRMv3 with MLPerf Inference benchmarking goals and practical constraints, we introduce two intentional deviations from the original HSTU paper setup. These changes are motivated by the need for hardware-friendly, widely implementable benchmarking that still captures the important compute patterns of sequential recommendation models.

Action embedding preprocessing: The original HSTU model uses contextual interleaving of action embeddings, where contextual features and user actions are interleaved in the input sequence. This arrangement provides richer context to the model to learn dependencies between user behavior and item features. However, interleaving doubles the effective sequence length, which in turn significantly increases the computational cost. In DLRMv3, we omit action interleaving in the baseline and instead use a simplified input sequence where action embeddings are directly combined with contextual embeddings without expanding the sequence length. We choose this design because 1) the synthetic benchmark dataset (detailed in the next section) is intended for performance measurement and does not contain a sufficiently rich set of action features to justify the added modeling complexity and doubled sequence length, and 2) the non-interleaved option provides a more balanced accuracy–efficiency tradeoff that is appropriate for a standardized inference benchmark.

Time/position encoding: The original HSTU model uses a relative positional bias in the attention mechanism, implemented as Mask(SiLU(QKT)+bias)V, which helps capture relative temporal relationships between tokens and can improve accuracy. DLRMv3 instead adopts absolute time/position encoding, where a position-dependent bias is added to the query, key, and value vectors, and attention is computed as Mask(SiLU(QKT))V. We use absolute bias because relative bias introduces kernel optimization challenges and can slow down attention computation on many processors, whereas absolute encodings are widely supported, easier to optimize, and yield more predictable performance. 

Dataset selection

Unlike DLRMv1 and DLRMv2, DLRMv3 formulates recommendation as a sequential transduction task over long user interaction histories and a very large item set. To be representative of modern production workloads, a suitable benchmark dataset needs to satisfy several properties simultaneously: (1) reasonably long user-interaction histories per request (thousands of events), so that sequential models and attention layers are meaningfully exercised; (2) a very large item set, consistent with the single large embedding table used in DLRMv3 (hash size on the order of one billion); and (3) a streaming structure, where the items viewed by a given user and the user’s preferences evolve over time and inference requests can be replayed in timestamp order. To the best of our knowledge, no existing public recommendation dataset provides all of these characteristics at the required scale.

For this reason, DLRMv3 uses a synthetic dataset specifically designed to match the model and system characteristics of large-scale sequential recommendation. The generator simulates five million users interacting with a billion items over time. Items are partitioned evenly into 128 categories, and each user is randomly assigned four categories that define their long-term interests. We generate 100 timestamps per user. The first 90 timestamps are used for training, and the remaining 10 timestamps are reserved for inference. For each user and each timestamp, with 70% probability, we generate a user interaction history (UIH) and a candidate sequence, and with 30% probability, the user has no request at that timestamp. The average UIH sequence length per (user, timestamp) is approximately 100, and the candidate sequence length is 2K. The generated sequences respect the streaming setup: at earlier timestamps, only a prefix of item IDs in each category can be sampled, and the maximum reachable ID grows linearly over time, mimicking new items entering the system.

When generating each user’s sequence, item categories for each position are sampled using a Dirichlet-process-style mechanism, following the generation algorithm in Section C of [5]. This setup yields a “rich get richer” dynamic where previously visited categories are more likely to be revisited. For each sampled category, item, and timestamp, we generate a continuous user-item rating (used as an action feature) by combining (1) a category- and time-dependent rating profile that varies smoothly over time via cosine functions with (2) an intrinsic per-item rating drawn once from a global 1-5 distribution representing item quality. The final rating is their average and is then used to derive labels for prediction.

Before converging on the streaming generator described above, we also experimented with the fractal expansion [19] approach for scaling the user-item interaction matrix from a smaller dataset. In brief, fractal expansion applies randomized kronecker products to a small interaction matrix to increase the dataset size while certain static structural properties (e.g., item popularity distribution and user engagement distribution) are preserved. However, fractal expansion is fundamentally designed for static datasets and does not naturally produce the streaming structure required by DLRMv3. In particular, DLRMv3 needs temporally ordered requests with evolving user histories and an expanding reachable item universe (to mimic new items entering the system) so that inference can be replayed in timestamp order. Fractal expansion does not inherently model properties such as per-user sequences over time and preference drift. Due to this fact, we did not select fractal expansion for DLRMv3 and instead designed a generator that simulates the streaming and sequential characteristics of online recommendation workloads.

Checkpoint Training

During the benchmarking creation process, we decided to use a 1TB float16 embedding table to store the one-billion item set in order to reflect the production use cases for such a model. This checkpoint is later trained on a single NVIDIA HGX B200 with eight GPUs and the fully open-sourced Generative Recommenders repo. Because each B200 has 180GB HBM capacity, the 1TB item table can  be fully sharded to eight GPUs with enough capacity remaining in the GPUs’  HBM to store weights, activations, and gradients for the training loop to run. 

Performance and accuracy metrics

The performance metrics chosen for the DLRMv3 benchmark follow the MLPerf Inference conventions and are closely aligned with those used for DLRMv2:

1. Offline scenario: Query processing throughput measured as queries per second (QPS) with no latency constraint beyond completing the full query set as quickly as possible.
   
2. Server scenario: QPS measured under an end-to-end latency constraint on the system under test (SUT) reflecting an online serving environment where requests arrive over time and must be processed under strict tail-latency targets.

To better reflect real-world deployment demands for large-scale recommendation ranking, we set the 99th-percentile SUT latency threshold to 80 ms in the server scenario.

Unlike other MLPerf Inference benchmarks, DLRMv3 uses a streaming setup with explicit timestamped requests. As a result, there is a request ordering constraint: requests associated with earlier timestamps must be served before those with later timestamps. Because MLPerf LoadGen does not natively enforce temporal ordering, this constraint may need to be implemented via SUT-side scheduling or request handling logic to ensure that streaming semantics are preserved while optimizing for throughput and tail latency.

For accuracy verification, we provide a trained reference checkpoint with a model size of 1TB in float16. Accuracy is evaluated on 0.1% subset of the inference dataset, comprising 34,996 requests across 10 inference timestamps. We report normalized entropy (NE), accuracy, and area under the ROC curve (AUC) as the primary quality metrics. For the reference implementation, the observed metrics are:

NE: 86.69%,

Accuracy: 69.65%,

AUC: 78.66%.

These values serve as the target baseline for accuracy compliance in the DLRMv3 benchmark. For accepted submissions, all three metrics (NE, Accuracy, AUC) must be within 99.9% of the reference implementation values.

Reference implementation

We provide a reference implementation of the DLRMv3 benchmark at

https://github.com/mlcommons/inference/tree/master/recommendation/dlrm_v3. The reference implementation covers model implementation, synthetic data generation, and performance/accuracy evaluation under the MLPerf Inference rules, as detailed below:

Model implementation: we provide a PyTorch implementation of the DLRMv3 HSTU-based ranking model, including a large embedding table consistent with the 1B-hash and 512-dimension configuration, a HSTU sequential encoder stack with the DLRMv3 design choices, and customized Triton kernels for HSTU operations.

Synthetic data pipeline: we provide the synthetic data generation script as well as a streaming data loader that constructs per-request model inputs (user interaction history + candidate set) and labels, matching the benchmark specification.

Reference benchmark: the reference benchmark contains three main runtime components:

1. Data producer: this part takes in sample/request ID and performs batching and outputs item IDs of batched samples. This part is executed on the CPU, and the reference implementation supports both single- and multi-threading in Python.
   
2. Sparse forward: this part takes in item ID lists and outputs embedding lookup outputs. This part is executed on the CPU, as datacenter systems typically contain large CPU memory to store the 1TB embedding table. The reference implementation uses PyTorch’s default embedding lookup operation without any quantization (in the float16 datatype). The implementation serves as a baseline for future improvements, e.g. distributed embedding lookup, GPU lookup, etc.
   
3. Dense forward: this part takes in batched embeddings and runs the HSTU encoder forward in the bfloat16 datatype to generate predictions. The implementation employs data parallelism across GPUs on a single host by distributing input batches to eight GPUs using Python multi-processing. On each GPU, the implementation uses customized Triton kernels to improve efficiency. This implementation serves as a baseline for future improvements, including more efficient kernels and low-precision quantization.

To illustrate the performance of the DLRMv3 reference implementation, we report end-to-end results for different numbers of NVIDIA H100 GPUs on a single host with batch size equal to 10. In each configuration, we increase QPS until the system approaches saturation while still maintaining reasonable latency.

A breakdown of query time at 8 GPUs highlights where time is spent in the baseline:

The results show sub-linear scaling in throughput (QPS) with up to eight GPU devices: going from one to eight GPUs increases QPS by 3.3X (300 to 1000), while average latency drops from 78 ms to 60 ms. At eight GPUs, the end-to-end P99 remains slightly above 80 ms due to batching, data movement, and Python/CPU-side overheads. This outcome is expected and consistent with the goals of the reference implementation. The current code is a Python-based baseline, not an aggressively optimized C++/production server, and it is explicitly not targeting the best possible performance. The numbers therefore demonstrate substantial headroom for submitters to optimize request scheduling, embedding placement, kernel optimization, and inference-server integration to satisfy the P99 latency constraint.

Optimization opportunities

DLRMv3 is intentionally designed to expose multiple layers of optimization opportunities across the inference stack–from the serving framework to dense and sparse kernels to KV-cache-aware scheduling. The reference implementation provides a clear but unoptimized baseline in several of these dimensions, leaving room for submitters to explore system and kernel improvements while preserving accuracy.

Inference server library. The current reference pipeline is implemented in Python with a simple CPU/GPU orchestration model. In production, ranking models of this scale are typically deployed behind high-performance inference serving frameworks. There is significant room to improve request batching, thread and stream management, and SUT-side scheduling by integrating with an optimized inference server library. 

KV cache integration. DLRMv3’s sequential HSTU architecture naturally supports KV caching, and there are two main dimensions where caching can yield meaningful gains:

1. In-request cache (M‑FALCON-style microbatching). As discussed in the HSTU paper [5], when serving large candidate sets (e.g., >1K candidates), M‑FALCON applies microbatching over the candidates while reusing the KV cache for the shared user interaction history. This strategy does not change the model’s FLOP count requirements or accuracy, but it can materially improve performance by amortizing candidate processing and overlapping CPU and GPU work within a single request. In a DLRMv3 setting, this arrangement can reduce per-request latency without altering the underlying model.

2. Across-request cache. For users who appear in multiple timestamps (as in the DLRMv3 streaming setup), the KV cache corresponding to their user interaction history can be reused across requests. This avoids recomputing the UIH encoding for each timestamp, reducing redundant FLOP by around 90% and improving both throughput and latency. Careful cache management and invalidation policies are needed to ensure correctness under streaming semantics.
   

Together, in-request KV microbatching (M‑FALCON style) and across-request KV caching offer complementary optimization levers: the former amortizes compute over large candidate sets and enables finer-grained CPU/GPU overlap within a request, while the latter reduces repeated UIH compute across consecutive requests from the same user. The reference implementation does not yet exploit these techniques, leaving them as explicit optimization opportunities for submitters targeting high-performance DLRMv3 deployments.

Dense optimization. The dense path in the reference implementation (HSTU encoder + top MLP) already uses Triton kernels and bf16, but there is still substantial room to improve. Further optimizations could include:

• more efficient attention and feed-forward kernels tailored to DLRMv3’s sequence lengths and dimensions;
• better utilization of vendor-specific libraries and exploration of lower-precision formats (e.g., FP8, INT8) under acceptable accuracy loss.
• optimized multi-GPU scaling within a node via improved load balancing, overlapping of data transfers with compute, and more sophisticated pipeline parallelism.

Sparse optimization. The sparse forward stage currently uses TorchRec embeddings on the CPU, with bf16 and no quantization, to support 1TB size in a simple, portable way. This is a natural baseline, but production systems often employ a combination of GPU-based embedding lookup, distributed sharding, hierarchical caching (e.g., HBM + DRAM), and quantization/compression. Submitters can explore moving embedding tables to accelerators, applying different types of quantization, or introducing model-parallel embeddings, as long as DLRMv3 accuracy targets are met. Improving the embedding pipeline can reduce end-to-end latency and CPU bottlenecks for large-scale deployments.

Conclusion

DLRMv3 extends the MLPerf Inference recommendation suite into the regime of large-scale sequential models, matching the size, structure, and serving patterns of the state-of-the-art production systems. By combining an HSTU-based ranking architecture, a billion-item synthetic streaming dataset, and a reference implementation that cleanly separates sparse and dense computation, DLRMv3 exposes the challenges of modern recommender inference: long user histories, attention-heavy sequence modeling, massive embeddings, and tight tail-latency targets under streaming workloads.

Notably, DLRMv3 increases the model memory footprint by 20X (from 50GB to 1TB) and the per-candidate compute by 6500X (from 40 MFLOP to 260 GFLOP) compared to DLRMv2, reflecting the rapid evolution of recommendation models over just three years. This dramatic scaling in compute and memory enables richer representations and more expressive models, ultimately delivering greater capability and accuracy to the community. By benchmarking these advances, DLRMv3 highlights how performance optimization is driving the frontier of recommender systems.

The benchmark is intentionally designed to leave room for innovation across hardware and software stacks–from inference servers and kernel optimizations to embedding pipelines and KV-cache-aware scheduling, while providing clear accuracy and performance baselines. We hope DLRMv3 will serve as a foundation for the community to evaluate, compare, and advance systems for next-generation recommendation workloads.

Contributors

Lucy Liao (Meta), Yu He (Stanford), Ze Yang (Meta), Zhao Zhu (Meta), Chunxing Yin (Meta), Rafay Khurram (Meta), Yuanjun Yao (Meta), Han Li (Meta), Daisy Shi He (Meta), Shilin Ding (Meta)

References

[1] Scaling Laws for Neural Language Models. https://arxiv.org/abs/2001.08361.

[2] Scaling Vision Transformers. https://arxiv.org/abs/2106.04560. 

[3] Scaling Law for Recommendation Models: Towards General-purpose User Representations.  https://arxiv.org/abs/2111.11294 

[4] Understanding Scaling Laws for Recommendation Models. https://arxiv.org/abs/2208.08489 

[5] Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations. https://arxiv.org/abs/2402.17152v2

[6] Recommender system is the single largest software engine on the planet — Jensen Huang. Q4’23 earning report. https://www.youtube.com/watch?v=txOv_pi-_R4&t=2020s 

[7] LONGER: Scaling Up Long Sequence Modeling in Industrial Recommenders. https://arxiv.org/abs/2505.04421 

[8] Towards Large-scale Generative Ranking. https://arxiv.org/abs/2505.04180 

[9] MTGR: Industrial-Scale Generative Recommendation Framework in Meituan. https://arxiv.org/abs/2505.18654 

[10] From Features to Transformers: Redefining Ranking for Scalable Impact. https://arxiv.org/abs/2502.03417 

[11] Hi-Gen: Generative Retrieval For Large-Scale Personalized E-commerce Search. https://arxiv.org/abs/2404.15675 

[12] HLLM: Enhancing Sequential Recommendations via Hierarchical Large Language Models for Item and User Modeling. https://arxiv.org/abs/2409.12740 

[13] Recommender Systems with Generative Retrieval. https://arxiv.org/abs/2305.05065 

[14] https://github.com/mlcommons/inference/blob/master/recommendation/dlrm_v2/pytorch/README.md 

[15] Deep Neural Networks for YouTube Recommendations, https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45530.pdf 

[16] Deep Learning Based Recommender System: A Survey and New Perspectives, https://arxiv.org/abs/1707.07435 

[17] Sampling-Bias-Corrected Neural Modeling for Large Corpus Item Recommendations, https://research.google/pubs/sampling-bias-corrected-neural-modeling-for-large-corpus-item-recommendations/ 

[18] Recommending What Video to Watch Next: A Multitask Ranking System, https://daiwk.github.io/assets/youtube-multitask.pdf 

[19] Scalable realistic recommendation datasets through fractal expansions, https://arxiv.org/pdf/1901.08910 

The post DLRMv3: Generative recommendation benchmark in MLPerf Inference appeared first on MLCommons.

The MLCommons Inference Working Group is excited to introduce a new Vision-Language Model (VLM) benchmark for the MLPerf Inference v6.0 round. As multimodal AI continues to reshape industries from retail to robotics, measuring the performance of these complex models in realistic production scenarios is more critical than ever.

For this round, we are debuting the Qwen3-VL-235B-A22B-Instruct model (released on Sept 23, 2025), paired with the Shopify Product Catalog dataset (released on Dec 12, 2025) to perform product classification and understanding, a common task in retail and commerce applications. This benchmark represents a significant leap forward in evaluating how AI systems handle the messy, high-stakes reality of hyperscale e-commerce where the infrastructure behind the scenes processes tens of millions of products daily.

The submission deadline is February 13, 2026. We invite all hardware vendors, cloud providers, and AI system experts to submit their results and help raise the bar for multimodal inference performance.

The Rise of Multimodal AI

The shift from unimodal (text-only) to multimodal AI is one of the most transformative trends in the industry. By integrating visual and textual data, businesses can automate complex workflows that were previously impossible, such as visual search, automated product tagging, and complex document understanding.

Market research underscores this urgency. The global multimodal AI market was valued at approximately USD 1.73 billion in 2024 and is projected to reach USD 10.89 billion by 2030, growing at a compound annual growth rate (CAGR) of 36.8% [1]. In the retail and e-commerce sectors specifically, adoption is accelerating even faster, with a projected CAGR of 34.6% through 2030 as merchants deploy personalized styling tools and augmented reality features [2]. This benchmark directly addresses this market demand by simulating a core “product understanding” workload that drives these commercial applications.

Model Selection: The Era of Qwen

This benchmark marks a milestone for the MLPerf Inference Benchmark Suite: it is the first time a model from the Qwen family is being introduced into the suite.

The open-weights landscape has evolved rapidly over the last 18 months, with the Qwen family emerging as a dominant force in the ecosystem. Recent data highlights this ascent:

• Adoption & downloads: Data from Hugging Face confirms that Qwen has cemented its status as a tier-one open-weights model family globally. By late 2025, it achieved record-breaking cumulative download numbers, driven by widespread adoption among developers and researchers [3, 4, 5].
• Performance: Independent benchmarks consistently rank Qwen models at the forefront of the industry. Qwen3-VL 235B A22B, in particular, delivers state-of-the-art results across a wide range of tasks, demonstrating high efficiency by achieving top-tier response quality with a significantly optimized parameter footprint [6, 7].

By selecting Qwen3-VL 235B A22B [8], we ensure MLPerf Inference remains at the bleeding edge, reflecting the models that developers and enterprises are actually deploying in 2026.

Qwen3-VL 235B A22B is a massive-scale mixture-of-experts (MoE) vision-language model designed to unify high-performance reasoning with efficient inference. It features 235 billion total parameters with 22 billion activated parameters per token, allowing it to deliver dense-model quality at a fraction of the computational cost. Built on the Qwen3 backbone [9], it natively supports a 256K token context window for interleaved text, image, and video inputs, and it introduces key architectural innovations like DeepStack (which fuses multi-level vision features) [10] and interleaved-MRoPE (for enhanced spatial-temporal modeling) [11]. This architecture enables state-of-the-art performance in complex tasks ranging from visual agentic workflows to long-context video understanding.

Case Study: Shopify Catalog

To ensure this benchmark reflects real-world production needs, we partnered with Shopify to curate the dataset and define the task. This workload mimics the “hierarchical taxonomy classification” task found in the “Shopify Catalog” product understanding layer, which processes 40 million products daily [12].

In production, the product understanding layer utilizes VLMs to perform a series of tasks that transform the unstructured data (e.g., non-canonical photos or descriptions provided by the merchants) of products into standardized metadata, including hierarchical taxonomy classification, attribute extraction, image understanding, title standardization, description analysis, and review summarization [12].

For the MLPerf Inference v6.0 round, we focus specifically on the hierarchical taxonomy classification task. The system must ingest a product’s title, description, and photo and select the correct category from a dynamic set of potential categories. This task perfectly captures the challenge of modern AI: maintaining high accuracy and throughput on noisy, real-world data at a massive scale.

Example Request and Response

An example request to a VLM inference endpoint might look like the following:

Python
[
  {
    'content': """Please analyze the product from the user prompt
and provide the following fields in a valid JSON object:
- category
- brand
- is_secondhand

You must choose only one, which is the most appropriate, correct, and specifc
category out of the list of possible product categories.

The description of the product sometimes contains various types of source code
(e.g., JavaScript, CSS, HTML, etc.), where useful product information is embedded
somewhere inside the source code. For this task, you should extract the useful
product information from the source code and leverage it, and discard the
programmatic parts of the source code.

Your response should only contain a valid JSON object and nothing more, e.g.,
you should not fence the JSON object inside a ```json code block.
The JSON object should match the followng JSON schema:
```json
{
  "additionalProperties": false,
  "description": "Json format for the expected responses from the VLM.",
  "properties": {
    "category": {
      "description": "The complete category of the product, e.g.,\n\"Clothing & Accessories > Clothing > Shirts > Polo Shirts\".\nEach categorical level is separated by \" > \".",
      "title": "Category",
      "type": "string"
    },
    "brand": {
      "description": "The brand of the product, e.g., \\"giorgio armani\\".",
      "title": "Brand",
      "type": "string"
    },
    "is_secondhand": {
      "description": "True if the product is second-hand, False otherwise.",
      "title": "Is Secondhand",
      "type": "boolean"
    }
  },
  "required": [
    "category",
    "brand",
    "is_secondhand"
  ],
  "title": "ProductMetadata",
  "type": "object"
}
```
""",
    'role': 'system'
  },
  {
    'content': [
      {
        'text': """The title of the product is the following:
```text
Frendorf | Fahrradindikator-Signalweste
```

The description of the product is the following:
```text
Die RADFAHR-INDIKATOR-SIGNALWESTE IST EIN NEUES PRODUKT DERZEIT NICHT IM HANDEL ERHÄLTLICH UND NICHT VERFÜGBARE . VERMEIDEN SIE UNNÖTIGE UND GEFÄHRLICHE RISIKEN FÜR IHR LEBEN! DIE SICHERHEITSWESTE IST DER Einfachste UND SICHERSTE WEG, UM DIE SICHTBARKEIT FÜR FAHRER ZU VERBESSERN. Geeignet für alle Radfahrer Für Kinder, die Fahrrad fahren Für Kinder auf dem Roller Für Jogger, die nachts laufen Für Kinder, die neben einer Straße gehen (die Weste passt ideal über einen Schulranzen) Diese Weste lässt sich hervorragend über einem Rucksack tragen. Eigenschaften: EINFACH FÜR AUTOFARER ZU ERKENNEN FÜR ERWACHSENE UND KINDER Die beiden verstellbaren Riemen machen sie für Kinder und Erwachsene geeignet, was bedeutet, dass eine Weste von mehreren Familienmitgliedern genutzt werden kann. FÜR VERSCHIEDENE KLEIDUNGEN Über einer Jacke oder einem dicken Mantel Über einem Rucksack oder Schulranzen SICHTBARKEIT = SICHERHEIT FERNBEDIENUNG Die Fernbedienung ist äußerst benutzerfreundlich und somit auch für kleine Kinder leicht zu bedienen: rechter Knopf zum Abbiegen nach rechts, linker Knopf zum Abbiegen nach links usw. Das Licht der Fernbedienung und das Blinken der Weste sind synchronisiert, sodass der Nutzer immer informiert ist, was hinter ihm geschieht. IN WENIGEN MINUTEN BEREIT ZUM EINSATZ Die Fernbedienung lässt sich dank der kleinen Kunststoffhalterungen ganz einfach am Lenker anbringen. Lieferumfang: 1 x RADFAHR-INDIKATOR-SIGNALWESTE 1 x Fernbedienung
```

The following are the possible product categories:
```json
['Sporting Goods > Fitness & General Exercise Equipment > Sport Safety Lights & Reflectors > Sport Safety Lights', 'Animals & Pet Supplies > Pet Supplies > Pet Collars & Harnesses > LED Collars', 'Sporting Goods > Outdoor Recreation > Cycling > Bicycle Accessories > Bicycle Computer Accessories > Bicycle Computer Cadence Sensors', 'Vehicles & Parts > Vehicle Parts & Accessories > Motor Vehicle Parts > Motor Vehicle Lighting > Tail Lights', 'Vehicles & Parts > Vehicle Parts & Accessories > Motor Vehicle Parts > Motor Vehicle Lighting > Light Bars', 'Business & Industrial > Work Safety Protective Gear > Work Safety Harnesses > Vest-Style Work Safety Harnesses', 'Sporting Goods > Fitness & General Exercise Equipment > Cardio > Cardio Machine Accessories & Parts > Exercise Bike Accessories & Parts > Exercise Bike Heart Rate Monitors', 'Vehicles & Parts > Vehicle Parts & Accessories > Motor Vehicle Parts > Motor Vehicle Lighting > Turn Signals']
```
""",
        'type': 'text'
      },
      {
        'image_url': {
          'url': 'data:image/JPEG;base64,<ommited base64 encoding of the following image>'
        },
        'type': 'image_url'
      }
    ],
    'role': 'user'
  }
]

Then, the response from the VLM inference endpoint would look like the following:

JSON
{
"category": "Sporting Goods > Fitness & General Exercise Equipment > Sport Safety Lights & Reflectors > Sport Safety Lights",
"brand": "Frendorf",
"is_secondhand": false
}

Performance Metrics

For the “offline” scenario, we measure and compare the overall request throughput (i.e., the number of completed requests per second) when sending all samples from (both the train and test splits of) the dataset to the VLM inference endpoint at least once. This arrangement closely mirrors Shopify’s major production requirements for batch processing: being able to process as many product samples as possible with as few GPU/accelerator resources as possible, beyond the current capacity of tens of millions of product samples daily.

For the “server” scenario, we set the 99th-percentile request latency constraint to be 12 seconds, and then we compare the maximum number of requests per second that we could send to the VLM inference endpoint before the 99th-percentile request latency constraint is violated. This scenario reflects Shopify’s online processing use case of handling demand surges before and during events or holidays (e.g., Black Friday).

Accuracy Metric

The response will be parsed by the following schema:

Python
class ProductMetadata(pydantic.BaseModel):
    """Json format for the expected responses from the VLM."""

    category: str
    """The complete category of the product, e.g.,
    "Clothing & Accessories > Clothing > Shirts > Polo Shirts".
    Each categorical level is separated by " > ".
    """

    brand: str
    """The brand of the product, e.g., "giorgio armani"."""

    is_secondhand: bool
    """True if the product is second-hand, False otherwise."""

We compute the hierarchical F1 score [13, 14] between the predicted categories and the ground-truth categories of all product samples in the dataset. Compared to exact matching, the hierarchical F1 score proportionally accounts for a partial, but not complete, match between a predicted category and the ground-truth product category, such as

Cameras & Optics > Cameras > Video Cameras > Drone Video Cameras

vs.

Cameras & Optics > Cameras > Video Cameras > Cinema Video Cameras

In the event where a response cannot be parsed by the given JSON schema, it will be treated as a complete mismatch between the predicted product category and the ground-truth product category. In this case, the number of category levels, which affects the hierarchical F1 calculation of the (mis-)prediction, will be taken as the same as that of the ground truth. For example, when encountering a JSON parsing error, if the ground truth is

Cameras & Optics > Cameras > Video Cameras > Drone Video Cameras

then the (mis-)prediction will be treated as

|NA| > |NA| > |NA| > |NA|

We set the accuracy target of the category hierarchical F1 score to be 0.7824. This is 99% of the mean category hierarchical F1 score across 10 runs on the original Qwen3-VL-235B-A22B-Instruct, which achieves 0.7903037.

Join the Benchmark

This is your opportunity to demonstrate the capabilities of your hardware and software stacks on one of the most relevant and rapidly growing AI workloads in the industry.

• Model: Qwen/Qwen3-VL-235B-A22B-Instruct
  • Commit SHA: 710c13861be6c466e66de3f484069440b8f31389
• Dataset: Shopify/product-catalogue
  • Commit SHA: d5c517c509f5aca99053897ef1de797d6d7e5aa5
• Submission Deadline: Feb 13, 2026

You can find the reference implementation along with the performance benchmarking code here. The reference implementation is based on vLLM, but its command-line interface and plugin registration enable you to benchmark arbitrary inference systems that expose endpoints via the Chat Completions OpenAI API. We look forward to seeing your submissions!

Acknowledge

We would like to thank the following individuals (sorted alphabetically) for their feedback and help that guided us in the journey of developing this benchmark:

References

1. Grand View Research. (2024). Multimodal AI Market Size And Share | Industry Report, 2030. https://www.grandviewresearch.com/industry-analysis/multimodal-artificial-intelligence-ai-market-report 
2. Mordor Intelligence. (2025). Multimodal AI Market Size & Share Analysis – Industry Reports. https://www.mordorintelligence.com/industry-reports/multimodal-ai-market 
3. Xinhua / China Daily. (Jan 12, 2026). Alibaba’s Qwen leads global open-source AI community with 700 million downloads. https://www.chinadailyhk.com/hk/article/626974 
4. AI World. (2025). Chinese developers account for over 45% of top open-model public downloads. https://www.aiworld.eu/story/chinese-developers-account-for-over-45-of-top-open-model-public-downloads 
5. Malika Aubakirova, Alex Atallah, Chris Clark, Justin Summerville, Anjney Midha. (2025). State of AI: An Empirical 100 Trillion Token Study with OpenRouter. https://openrouter.ai/state-of-ai 
6. LLM-Stats. (2026). Qwen3 VL 235B A22B Instruct. https://llm-stats.com/models/qwen3-vl-235b-a22b-instruct 
7. Artificial Analysis. (2026). Qwen3 VL 235B A22B (Reasoning) Intelligence, Performance & Price Analysis. https://artificialanalysis.ai/models/qwen3-vl-235b-a22b-reasoning  
8. Qwen Team. (2025). Qwen3-VL Technical Report. https://arxiv.org/pdf/2511.21631 
9. Qwen Team. (2025). Qwen3 Technical Report. https://arxiv.org/pdf/2505.09388 
10. Lingchen Meng, Jianwei Yang, Rui Tian, Xiyang Dai, Zuxuan Wu, Jianfeng Gao, and Yu-Gang Jiang. Deepstack: Deeply stacking visual tokens is surprisingly simple and effective for lmms. In Advances in Neural Information Processing Systems, volume 37, pp. 23464–23487, 2024.
11. Jie Huang, Xuejing Liu, Sibo Song, Ruibing Hou, Hong Chang, Junyang Lin, and Shuai Bai. Revisiting multimodal positional encoding in vision-language models, 2025.
12. Audrey-Anne Guindon. (2025). Leveraging multimodal LLMs for Shopify’s global catalogue: Recap of expo talk at ICLR 2025. https://shopify.engineering/leveraging-multimodal-llms 
13. Silla, C. N., & Freitas, A. A. (2011). A survey of hierarchical classification across different application domains. Data Mining and Knowledge Discovery, 22(1), 31-72.
14. Kiritchenko, S., Matwin, S., Nock, R., & Famili, A. F. (2006, June). Learning and evaluation in the presence of class hierarchies: Application to text categorization. In Conference of the Canadian Society for Computational Studies of Intelligence (pp. 395-406). Springer, Berlin, Heidelberg.

The post Call for Submission: Qwen3 VL MoE for MLPerf Inference v6.0 appeared first on MLCommons.

The MLPerf Inference benchmark suite is designed to measure how quickly systems can run AI models across a variety of workloads. The open-source and peer-reviewed suite performs system performance benchmarking in an architecture-neutral, representative, and reproducible manner, creating a level playing field for competition that drives innovation, performance, and energy efficiency for the entire industry. It provides critical technical information for customers who are procuring and tuning AI systems. 

This round of MLPerf Inference results sets a record for the number of participants submitting systems for benchmarking at 27. Those submissions include systems using five newly-available processors and improved versions of AI software frameworks. The v5.1 suite introduces three new benchmarks that further challenge AI systems to perform at their peak against modern workloads.

“The pace of innovation in AI is breathtaking,” said Scott Wasson, Director of Product Management at MLCommons. “The MLPerf Inference working group has aggressively built new benchmarks to keep pace with this progress. As a result, Inference 5.1 features several new benchmark tests, including DeepSeek-R1 with reasoning, and interactive scenarios with tighter latency requirements for some LLM-based tests. Meanwhile, the submitters to MLPerf Inference 5.1 yet again have produced results demonstrating substantial performance gains over prior rounds.” 

Llama 2 70B generative AI test establishes the trendlines

The Llama 2 70B benchmark continues to be the most popular benchmark in the suite, with 24 submitters in this round.

It also gives a clear picture of overall performance improvement in AI systems over time. In some scenarios, the best performing systems improved by as much as 50% over the best system in the 5.0 release just six months ago. This round saw another first: a submission of a heterogeneous system that used software to load-balance an inference workload across different types of accelerators.

In response to demand from the community, this round expands the interactive scenario introduced in the previous version, which tests performance under lower latency constraints as required for agentic and other applications of LLMs. The interactive scenarios, now tested for multiple models, saw robust participation from submitters in version 5.1.

Three new tests introduced

MLPerf Inference v5.1 introduces three new benchmarks to the suite: DeepSeek-R1; Llama 3.1 8B; and Whisper Large V3.

DeepSeek R1 is the first “reasoning model” to be added to the suite. Reasoning models are designed to tackle challenging tasks, using a multi-step process to break down problems into smaller pieces in order to produce higher quality responses.  The workload in the test incorporates prompts from five datasets covering mathematics problem-solving, general question answering, and code generation.

“Reasoning models are an emerging and important area for AI models, with their own unique pattern of processing,” said Miro Hodak, MLPerf Inference working group co-chair. “It’s important to have real data to understand how reasoning models perform on existing and new systems, and MLCommons is stepping up to provide that data. And it’s equally important to thoroughly stress-test the current systems so that we learn their limits; DeepSeek R1 increases the difficulty level of the benchmark suite, giving us new and valuable information.” 

More information on the DeepSeek R1 benchmark can be found here.

Llama 3.1 8B is a smaller LLM useful for tasks such as text summarization in both datacenter and edge scenarios. With the Inference 5.1 release, this model is replacing an older one (GPT-J) but retaining the same dataset, performing the same benchmark task but with a more contemporary model that better reflects the current state of the art. Llama 3.1 8B uses a large context length of 128,000 tokens, whereas GPT-J only used 2048. The test uses the CNN-DailyMail dataset, among the most popular publicly available for text summarization tasks. The Llama 3.1 8B benchmark supports both datacenter and edge systems, with custom workloads for each.

More information on the Llama 3.1 8B benchmark can be found here.

Whisper Large V3 is an open-source speech recognition model built on a transformer-based encoder-decoder architecture. It features high accuracy and multilingual capabilities across a wide range of tasks, including transcription and translation. For the benchmark test it is paired with a modified version of the Librispeech audio dataset. The benchmark supports both datacenter and edge systems.

“MLPerf Inference benchmarks are live and designed to capture the state of AI deployment across the industry,” said Frank Han, co-chair of the MLPerf Inference Working Group. “This round adds a speech-to-text model, reflecting the need to benchmark beyond large language models. Speech recognition combines language modeling with additional stages like acoustic feature extraction and segmentation, broadening the performance profile and stressing system aspects such as memory bandwidth, latency, and throughput. By including such workloads, MLPerf Inference offers a more holistic and realistic view of AI inference challenges.”

More information on the Whisper Large V3 benchmark can be found here.

The momentum builds for AI… and for MLPerf benchmarks

The MLPerf Inference 5.1 benchmark received submissions from a total of 27 participating organizations: AMD, ASUSTek, Azure, Broadcom, Cisco, Coreweave, Dell, GATEOverflow, GigaComputing, Google, Hewlett Packard Enterprise, Intel, KRAI, Lambda, Lenovo, MangoBoost, MiTac, Nebius, NVIDIA, Oracle, Quanta Cloud Technology, Red Hat Inc, Single Submitter: Amitash Nanda, Supermicro, TheStage AI, University of Florida, and Vultr. 

The results included tests for five newly-available accelerators:

• AMD Instinct MI355X
• Intel Arc Pro B60 48GB Turbo
• NVIDIA GB300
• NVIDIA RTX 4000 Ada-PCIe-20GB
• NVIDIA RTX Pro 6000 Blackwell Server Edition
  

“This is such an exciting time to be working in the AI community,” said David Kanter, head of MLPerf at MLCommons. “Between the breathtaking pace of innovation and the robust flow of new entrants, stakeholders who are procuring systems have more choices than ever. Our mission with the MLPerf Inference benchmark is to help them make well-informed choices, using trustworthy, relevant performance data for the workloads they care about the most. The field of AI is certainly a moving target, but that makes our work – and our effort to stay on the cutting edge – even more essential.”

Kanter continued, “We would like to welcome our new submitters for version 5.1:  MiTac, Nebius, Single Submitter: Amitash Nanda, TheStage AI, University of Florida, and Vultr. And I would particularly like to highlight our two participants from academia: Amitash Nanda, and the team from the University of Florida. Both academia and industry have important roles to play in efforts such as ours to advance open, transparent, trustworthy benchmarks. In this round we also received two power submissions, a data center submission from Lenovo and an edge submission from GATEOverflow. MLPerf Power results combine performance results with power measurements to offer a true indication of power-efficient computing. We commend these participants for their submissions and invite broader MLPerf Power participation from the community going forward.”

View the results

To view the results for MLPerf Inference v5.1, please visit the Datacenter and Edge benchmark results pages.

About MLCommons

MLCommons is the world’s leader in AI benchmarking. An open engineering consortium supported by over 125 members and affiliates, MLCommons has a proven record of bringing together academia, industry, and civil society to measure and improve AI. The foundation for MLCommons began with the MLPerf benchmarks in 2018, which rapidly scaled as a set of industry metrics to measure machine learning performance and promote transparency of machine learning techniques. Since then, MLCommons has continued to use collective engineering to build the benchmarks and metrics required for better AI – ultimately helping to evaluate and improve the accuracy, safety, speed, and efficiency of AI technologies.

For additional information on MLCommons and details on becoming a member, please visit MLCommons.org or email participation@mlcommons.org.

Press Inquiries: contact press@mlcommons.org

The post MLCommons Releases New MLPerf Inference v5.1 Benchmark Results appeared first on MLCommons.

The Reasoning LLM Task Force was convened to ensure that the MLPerf® Inference v5.1 benchmarks address the evolving capabilities of advanced reasoning language models in real-world applications. Reasoning LLMs stand out for their mathematical, logical, and step-by-step problem-solving capacities. They are being deployed on the hardest use cases such as knowledge and reasoning, code generation and debugging, and math & science problem solving.

These models generate Chains of Thoughts as part of their responses; their output consists of reasoning tokens and output tokens. They benefit from large output lengths where the model is allowed to “think” longer by emitting more reasoning tokens, which typically constitute the bulk of the output generation. Performance evaluation considers the token generations from both reasoning and output tokens. However, only output tokens are used for the accuracy evaluation.

With the addition of DeepSeek-R1 671B (DS-R1) to the MLPerf inference suite, we focus on two key aspects:
Creating a first industry-standard benchmark for reasoning models

• Creating a first industry-standard benchmark for reasoning models

• Evaluate AI performance for long output sequence lengths required by reasoning LLMs

Model selection

DeepSeek-R1 671B (released in January 2025) is the flagship reasoning LLM from DeepSeek, fully open-sourced on HuggingFace. Published under an MIT license, it is a significant step forward for open-source reasoning LLMs, with scale and capabilities competing with state-of-the-art commercial offerings at the time.

Some key aspects of this model include:

• 671B total parameters, with 37B activated per-token and a maximum output length of 20K (the model may support higher lengths)
• The model is trained with a combination of cold-start supervised data and large-scale reinforcement learning, resulting in performance on par with the state of the art in math, code, and reasoning tasks

Popular use cases include:

• Step-by-step planning: DeepSeek-R1 excels at breaking down complex problems into actionable sub-steps, making it suited to workflow automation where clarity and logical progression are crucial for robust outcomes

• Knowledge & reasoning: Providing expert answers across diverse domains, synthesizing logic and concepts across fields like math, science, programming, and professional knowledge
• Generating & debugging code: Supporting technically demanding workflows that benefit from logical clarity and error resolution

Dataset and task selection

The dataset for benchmarking DeepSeek-R1 was curated to highlight advantages of a large reasoning model over traditional LLMs. With the chosen maximum output sequence length of 20K (the highest yet in the MLPerf Inference benchmark suite), DS-R1 is able to conclude its answer from long chains of thoughts and reasoning traces on the most challenging problems.

The MLCommons® task force integrated samples from a diverse mix of open datasets:

• AIME (1983–2024): link
  These advanced high school mathematics problems trace their roots to the storied American Invitational Mathematics Examination, serving as a gold standard for mathematical reasoning assessment. Each problem was selected for its challenge level and capacity to test logical breakdowns, multi-step solutions, and mathematical creativity. The result is an expression or value and evaluation is via Exact-Match.
• MATH500: link
  A curated set of 500 multilingual math problems, designed to probe both computational accuracy and linguistic flexibility. The format ensures models are evaluated on diverse language inputs—crucial for international deployments—alongside pure reasoning skill. The result is a mathematical expression or a numerical value and evaluation is via Exact-Match.
• GPQA-Diamond: link
  This unique dataset features graduate-level multiple-choice questions in biology, physics, and chemistry, intentionally crafted to be immune to trivial lookup strategies. The content rigorously tests a model’s ability to synthesize scientific logic and demonstrate oversight rather than regurgitation. The result is a textual word in response to the question in the input sequence. Evaluation is via Exact-Match.
• MMLU-Pro: link
  Building on the original MMLU benchmark, MMLU-Pro ramps up difficulty by expanding answer choices and emphasizing reasoning-intensive queries across broader domains. It ensures models aren’t just strong in niche areas but can generalize across challenging, expert-level knowledge. The result is a specific choice among the multiple-choices presented to the model. Evaluation is via Exact-Match.
• LiveCodeBench: link
  Over 500 live-updating code competition tasks push models to parse, generate, and debug complex software solutions in real time. This dataset highlights logical planning, error identification, and the code reasoning demanded by modern technical workflows. The result is a code that solved the problem stated in the input. Evaluation is via code execution, passing test-cases and runtime constraints.

With mean input and output sequence lengths of 800 and 3,880 tokens, respectively, the DS-R1 dataset highlights the model’s ability in parsing proficiency, contextual linking, and synthesizing insights from complex and lengthy inputs.

Performance metrics

The performance metrics chosen for the DeepSeek-R1 benchmark are the same as those used for the Llama 2 70B benchmark in MLPerf Inference v4.0: (1) token generation throughput for the offline scenario and (2) token generation throughput while constrained by Time To First Token (TTFT) and Time Per Output Token (TPOT) for the server scenario. For more information, please refer to the “Measuring performance of the LLM” section of “Llama 2 70B: An MLPerf Inference Benchmark for Large Language Models” on the MLCommons website.

To reflect the demands of real-world deployment, we set the 99th percentile TTFT threshold to two seconds and the 99th percentile TPOT threshold to 80 ms. These thresholds reflect the computational challenges of deploying large reasoning models with a large thinking budget while maintaining reasonable responsiveness.

Accuracy metric

The DS-R1 benchmark employs task-specific accuracy metrics to evaluate performance across the dataset. Exact match to ground truth is used for multiple-choice style questions (GPQA-Diamond, MMLU-Pro) and math/numerical questions (AIME, MATH500). For code generation (LiveCodeBench), we evaluate the generated code using a code-evaluation framework and test cases. The accuracy threshold for closed division submissions is set to 99% of the FP8 reference.

( See https://github.com/mlcommons/inference/tree/master/language/deepseek-r1 )

The reference implementation for DeepSeek-R1 leverages PyTorch and the official DeepSeek-R1 inference code. Three backends are provided (vLLM, SGLang and PyTorch) to support a wide range of AI processors.

Reference implementation

The reference implementation for DeepSeek-R1 leverages PyTorch and the official DeepSeek-R1 inference code. Three backends are provided (vLLM, SGLang and PyTorch) to support a wide range of AI processors.

For readers interested in running the model themselves, we encourage them to follow the reference implementation, which contains the code and the instructions on how to run the entire end-to-end benchmark.

Conclusion

The usage of LLMs is expanding rapidly across a diversity of application domains, with a demand for LLMs at different scales and response latencies with large sequence lengths. MLPerf is keeping pace with these trends by introducing a very large scale 671B-parameter reasoning benchmark in the new MLPerf Inference v5.1 release. 

In this post, we shared the motivation and the complexities involved in the choice of model, task, dataset, accuracy, performance, and verification of such LLM benchmarks. We believe the design choices of this benchmark objectively address most of the critical deployment decisions faced by practitioners while delivering important performance insights to customers. With the addition of this benchmark to MLPerf Inference, we now offer a comprehensive suite of language benchmarks at all scales (7B to 671B), diversity of architectures (including mixture of experts (MoE) models), types (reasoning and non-reasoning), and deployment scenarios (datacenter, edge, or low latency).

Details of the MLPerf Inference DeepSeek-R1 benchmark and reference implementation can be found here.

The post DeepSeek Reasoning for MLPerf Inference v5.1 appeared first on MLCommons.