MLPerf was originally designed to help a relatively small number of cloud providers and downstream system builders inform their AI hardware purchasing decisions. Today, procuring inference compute is a salient business decision for companies of all sizes – and it means evaluating options across neoclouds, cloud providers, and managed services simultaneously. These buyers need reliable, comparable, and independent performance benchmarks to inform their purchasing decisions, and these benchmarks must be dynamic enough to keep pace with an industry that launches new models weekly. 

MLPerf Endpoints is designed to meet this more diverse and complex demand for benchmarks to inform AI system procurement decisions. Four key principles are guiding our development of enterprise-buyer-centric benchmarks:  

1. Current: Results must keep pace with the market. Buyers can’t wait months for a benchmark round to include new hardware or models. 
2. Comprehensive: Buyers need benchmarks that cover the many competing inference providers, systems, and workloads available for purchase. 
3. Comparable: Buyers need to see apples-to-apples results across vendors, normalized for cost or power, to inform procurement decisions.
4. Commentary: To support buyers in these decisions, Endpoints provides additional context for benchmark results through visualizations, data filtering, and analysis. 

MLPerf Endpoints v0.7 is a foundation release with initial results from Coreweave, Google, Intel, KRAI, and Nvidia. We want to congratulate these members on their excellent results, spanning several orders of magnitude in performance across 3 benchmarks. This is the infrastructure on which we are building a more dynamic, comprehensive, and comparable inference benchmark suite for the data center. Endpoints currently supports automated submission pipelines, continuous review tooling, and dynamic visualization of results that you can see at mlcommons.endpoints.com. We are also evolving our benchmarking rules towards more buyer-centric benchmarks.

Later this year, we will deliver MLPerf Endpoints v1.0 with more buyer-centric rules, normalization, and an expanded set of benchmarks including agentic workloads and then open the rolling submission process to our broader membership. The rolling submission process ensures Endpoints will provide current and up-to-date results that move at the pace of the market. 

We’re grateful to our 30+ supporters who have helped guide the development of MLPerf Endpoints, including AMD, Argonne National Laboratory, Broadcom, Core 42, Dell, HPE, Lambda, Oracle,  and Red Hat. Their partnership has been invaluable in stress-testing our rules, processes, and improving our infrastructure. With this release, we are just getting started. 

If you’re a system or service provider, now is the time to get involved, help shape the rules, and plan your submission at a time of your choosing. Complete this form to let us know you’re interested in submitting, and what hardware you want to submit on. The community-developed rules are actively being refined for the 1.0 release later this year, and early participants in this effort will shape Endpoints’ direction. If you’re an enterprise buyer, Endpoints is being built for you, and we want to know how we can best support you and your organization. Let us know by emailing alejandro@mlcommons.org.

We deeply value the trust our members place in us as the benchmark of record for inference. We are excited to build this future of MLPerf with all of you. 

[1] A. Tschand, A. T. R. Rajan, S. Idgunji, et al., “MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from µWatts to MWatts for Sustainable AI,” in 2025 IEEE International Symposium on High-Performance Computer Architecture (HPCA), 2025. arXiv:2410.12032.

[2] D. Kanter, M. Ahmad, H. Kassa, and S. Rishab, “MLPerf Training v4.1 Results — Press Briefing,” MLCommons, Nov. 13, 2024. [Online]. Available: https://docs.google.com/presentation/d/1KSIJBvIV9OcswF1mVbhGRN0nUWbSgbYSoxu6dHCwajM/

The post MLPerf Endpoints v0.7: A Foundation Release appeared first on MLCommons.

The MLPerf Inference benchmark suite must evolve alongside AI deployment patterns. Early inference benchmarks focused on image classification, object detection, speech recognition, recommendation, and single-turn language generation. Those workloads remain important, but they no longer cover one of the fastest-growing ways large language models are used in production: multi-turn agentic inference.

For example, a coding assistant is far more complex than a single query. The agent will read an issue, inspect files, run commands, observe failures, edit code, and iterate potentially many times. Similarly, a workflow agent gathers customer information, calls tools, interprets results, asks follow-up questions, and continues until the task is resolved. In both cases, the workload is a trajectory: a sequence of dependent turns where each request includes the conversation history up to that point.

This changes the serving problem in four concrete ways:

• Context grows across the trajectory, so prefill and KV-cache pressure increase over time.
• KV-cache reuse is now a critical serving optimization for performance and efficiency.
• Output lengths vary widely, from compact tool calls to long reasoning traces.
• Turn dependencies make throughput a measure of closed-loop progress, not independent request rate.

Figure 1. Illustration of an agentic inference scenario involving multiple turns.

The Agentic Inference benchmark adds this class of workload to the MLPerf Endpoints framework. It keeps the general MLPerf measurement principles intact while defining the workload-specific pieces: model choice, dataset composition, multi-turn load generation, output validation, and constraints for optimizations such as prefix caching and speculative decoding.

Terminology

• Turn: one client-issued user or tool request and the model response generated for that request.
• Trajectory: the ordered sequence of turns for one task or simulated user; later turns include the accumulated conversation history.

Model selection

The benchmark needs long-context, thinking-capable LLMs that stress the serving behavior of agentic applications: growing context, KV-cache reuse, variable output lengths, and strict turn dependencies. For this benchmark, we chose Kimi K2.6 and Qwen3.6-35B-A3B. Kimi K2.6 brings state-of-the-art coding capability and a large model profile representative of leading agentic systems; Qwen3.6-35B-A3B is more compact and brings a new Gated DeltaNet (GDN) architecture and exceptional coding performance for its size. This gives the benchmark coverage across distinct serving behaviors, architectural choices, and speculative-decoding paths. Both models are evaluated with the same methodology and dataset. To avoid ambiguity, each model produces its own result; the two models are not run together or combined into a single score.

Table 1. Model metadata and benchmark settings

Dataset and task selection

The benchmark dataset combines two agentic domains that exercise different infrastructure bottlenecks. Together, they contain 613 multi-turn trajectories: 113 agentic coding trajectories and 500 agentic workflow trajectories. Across the reference dataset, these trajectories contain 30,335 client-issued turns and 30,328 assistant turns. By using real collected traces for benchmarking, the workload simulates real-world behavior in the serving stack, including how speculative decoding and expert-rank balancing behave under realistic multi-turn traffic.

Table 2. Dataset composition and mean token statistics

The agentic coding traces come from the DeepSWE dataset by DataCurve (datacurve.ai), which contains software engineering tasks built around repository-level bugs and feature requests. A typical trace begins with a user request describing an issue, followed by an assistant that investigates the repository through a series of bash commands. The agent searches files, reads code, runs tests, observes errors, edits the implementation, and iterates. These traces are deep: the median trajectory contains dozens of turns, and the conversation history grows steadily as command outputs, file contents, and test logs accumulate.

The Workato agentic workflow traces come from enterprise customer-support and orchestration scenarios. They represent interactions where a simulated customer asks for help and the agent uses tools to retrieve orders, track shipments, check policies, escalate cases, or resolve account questions. These trajectories are usually shallower than the coding traces, but they include a much larger shared system prompt with many tool definitions and business rules. The agentic workflow traces were provided by Workato, the agentic control and execution plane for the enterprise. The data are synthetic traces modeled on Workato’s production experience orchestrating customer support agents for enterprise customers.

The two domains are intentionally different:

• Coding traces grow aggressively over many turns.
• Workflow traces begin with a large common prefix and grow more slowly.
• Coding stresses KV-cache capacity and long-context scheduling.
• Workflow stresses shared-prefix reuse and routing locality.
• The combined workload prevents systems from optimizing for only one agentic traffic shape and stresses context-aware routing.

Client Design

We introduced multi-turn support to MLPerf Endpoints so the benchmark can drive full end-to-end agentic workloads against standard serving stacks. Submitters simply need to spin up an OpenAI-compatible endpoint using vLLM, SGLang, TensorRT-LLM, or another serving framework, then point the client at that endpoint for full end-to-end benchmarking.

The client handles the multi-turn behavior:

• Closed-loop replay: one active conversation issues one turn at a time and waits for the complete model response before the next turn.
• Target concurrency: the load generator controls the number of active users or conversations without breaking turn dependencies.
• Inter-turn delay: dataset-provided waits before tool or user turns preserve realistic pacing without counting delay as model serving latency.
• Conversation-aware routing: a stable X-Session-ID header is sent for each trajectory so routers can preserve KV-cache locality.
• Cache salting: deterministic salt markers around the system prompt allow valid within-trajectory reuse while blocking invalid cross-trajectory reuse to ensure the benchmark is representative of production workloads.
• Deterministic prompt reconstruction: future prompts are built from the pre-recorded dataset rather than from live model output, which keeps performance runs reproducible while still measuring generated outputs.
• Generated token cache clearing: In order to enable fair benchmarking of all platforms, the generated tokens are cleared out of the KV cache by introducing a whitespace character – ensuring that the performance is independent of the system that was used for generating the traces.

Performance metrics

The primary performance chart is a Pareto curve where each point is a fixed-concurrency benchmark result. Submitters need to submit a full Pareto curve as defined in the MLPerf Endpoints rules.

• Y-axis: output tokens per second per system, measuring aggregate serving throughput.
• X-axis: output tokens per second per user, computed as output tokens per trajectory / (E2E time – total delays).

The two axes represent complementary views of the serving system. The x-axis, output tokens per second per user, captures how quickly an individual agent progresses through its task; moving right means faster task completion. The y-axis, output tokens per second per system, captures the total work completed across all active agents; moving up means greater aggregate throughput and higher system utilization. As concurrency increases, the system may complete more total work while each individual agent progresses more slowly. The Pareto curve makes this tradeoff visible.

Figure 2. Chart illustrating agentic inference performance

Accuracy metrics

Accuracy is enforced as a three-level hierarchy so the Pareto curve measures usable agentic progress rather than responses that are shorter, lower quality, or easier to serve.

• OSL mean value: prevents systems from gaining throughput by truncating responses or shifting the answer-length distribution. The check requires generated outputs to preserve the expected mean output sequence length for the workload.
• Inline accuracy: catches behavior drift during the exact performance configuration being reported. It operates on the outputs produced by the performance run itself and compares them against ground-truth values preserved in the dataset, including coding tool-call action checks and workflow intent-code checks, so no separate inference pass is required.
• Standalone accuracy: verifies task-level capability beyond turn-level similarity. The standalone evaluation will use 200 tasks from the SWE-bench Verified dataset to check whether the submitted model configuration can solve representative tasks end to end.

All three levels must be run for every point on the Pareto curve, and each score must meet a pre-defined threshold. Thresholds are defined separately for each level and for both Kimi K2.6 and Qwen3.6-35B-A3B.

Table 3. Tentative accuracy values, subject to change.

Reference implementation

Submitters only need to spin up an OpenAI-compatible server using vLLM, SGLang, TensorRT-LLM, or another serving framework, then point the MLPerf Endpoints client at that endpoint to run the full benchmark. The MLPerf Endpoints Agentic Inference README includes the example configs, command lines, and setup steps.

Conclusion

Agentic inference is one of the fastest growing applications of AI today and it’s far more complicated than existing single-turn text generation. Agentic workflows have serving patterns with unique constraints: growing context, strict turn dependencies, tool-mediated delays, shared prefixes, long-tail trajectories, and output quality checks that must run on the same configuration used for performance.

The Agentic Inference benchmark brings those constraints into MLPerf in a reproducible form. By combining deep coding trajectories with shared-prefix workflow trajectories, it measures whether serving systems can make progress for real multi-turn users while still using hardware efficiently.

As agentic applications become a larger share of production LLM traffic, the industry needs benchmarks that measure the systems work that actually matters: preserving cache locality, scheduling long contexts, maintaining output quality, and balancing aggregate throughput with per-user progress. This benchmark is a step toward that measurement standard. If your team is building serving infrastructure for multi-turn agents, join MLCommons to access the reference implementation and submit results

References

• MLPerf Endpoints repository.
• MLPerf Endpoints Inference Rules and Pareto collection methodology.
• MLPerf Endpoints Agentic Inference reference implementation.
• SWE-bench Verified benchmark.
• Workato agentic workflow and enterprise task definitions.
• DeepSWE Dataset by Datacurve.

The post Agentic Inference for MLPerf Inference appeared first on MLCommons.

At GTC, MLCommons^® co-founder David Kanter unveiled the answer: MLPerf^® Endpoints, a ground-up rethinking of how the industry’s benchmark of record measures generative AI performance. With over 125 member organizations, more than 90,000 reproducible results to date, and recognition by IEEE and ISO/IEC SC42, MLPerf already underpins critical procurement decisions across government, industry, and academia. Endpoints is designed to keep that trust intact while adapting to a landscape that looks fundamentally different from what it was just two years ago. You can try it out here.

Why Traditional Approaches Need To Change

Traditional MLPerf inference benchmarks used a tightly coupled architecture: the load generator and model server ran as a single local process with shared dependencies. That worked well for classical ML, but generative AI deployments are API-first — whether on-prem, in the cloud, or via managed cloud endpoints. 

Meanwhile, measuring GenAI performance is harder than it looks. Real serving combines accuracy, latency, throughput, and sequence length into a non-linear, multi-dimensional surface. Long-tail queries, variable arrival patterns, and tight SLAs interact in ways that simple scenarios miss entirely.

An API-Centric Architecture

MLPerf Endpoints replaces the monolithic design with a decoupled client that communicates with any model-serving API endpoint via standard interfaces such as HTTP or gRPC. The benchmark client is lightweight and production-ready; the system under test is simply a URL. This means zero-effort integration for submitters — point the client at your endpoint and run. The architecture also enables benchmarking of managed cloud services alongside bare-metal deployments on an equal footing, something the previous framework could not easily support.

Under the hood, a new scalable load generator uses separate worker processes, pre-warmed connection pools, and ZeroMQ-based IPC to ensure the harness itself never becomes the bottleneck, even when testing rack-scale systems.

Pareto Curves and Step Functions: New Visualisation for New Metrics and Easy Comparison

One of the most compelling innovations is how results are presented. Each benchmark run varies concurrency and captures key metrics, including TTFT (time to first token), throughput (tokens per second), interactivity (tokens per second per user), and response latency. Submitters tune parallelism and batch settings for each operating point, and the visualizer plots these metrics as a Pareto curve (e.g., throughput vs. interactivity)—giving buyers an immediate picture of the real-world trade-offs, such as between serving more users and keeping each user’s experience responsive.

Crucially, MLPerf Endpoints uses step functions rather than interpolated trend lines. GenAI performance is highly non-linear; interpolating between measured points can suggest performance levels that were never actually achieved, masking memory overflows or P99 latency spikes. Step functions show only verified operating points, eliminating what the presentation aptly called “paper performance.” Customers can easily compare these step functions against each other and match these verified points to their own use cases — high concurrency during the day, best possible interactivity at night.

Rolling Submissions: Benchmarking at the Speed of Software Updates

Perhaps the boldest change is operational. MLPerf has historically published results on a fixed bi-annual schedule, depending on the benchmark offering (Training, Inference, Storage, etc). In a market where major model releases land every few weeks, that cadence is too slow for buyers writing RFPs and vendors launching hardware. Starting in Q2 2026, MLPerf Endpoints will move to continuous rolling submissions: submitters can publish peer-reviewed, audited results at any time. Incremental submissions let vendors start with a baseline Pareto curve and iteratively add more operating points as their software stack matures.

The approach is inspired by proven methodologies from other industry-standard bodies such as SPEC and TPC, adapted to the world of AI. Peer review and audit requirements will remain fully intact to deliver the robustness that the industry demands.

What Comes Next

The first MLPerf Endpoints v0.5 demonstration feature results from AMD, Google, Intel, KRAI, and NVIDIA, backed by over 30 supporting organisations, including Argonne National Laboratory, Broadcom, Dell, HPE, Lambda, Lenovo, Oracle, Red Hat, and the University of Florida. The results include models such as DeepSeek-R1, GPT OSS 120B, Llama 3.1 8B, QWEN 3 Coder 480B and more, running on nearly a dozen different systems.

Looking ahead, MLCommons is inviting the broader ecosystem to shape what comes next. Enterprise and IT buyers can join the advisory council. OEMs, CSPs, and ODMs can contribute results to the rolling leaderboard. Model developers and API providers can integrate next-generation SOTA models and build managed roadmaps. Researchers can anchor reproducible baselines using the Endpoints framework. New models — especially popular and commercially relevant ones — are continuously evaluated for inclusion. You can try it out for yourself here.

Get involved: MLPerf Endpoints rolling submissions open in Q2 2026. To participate, contribute, or learn more, visit https://mlcommons.org/benchmarks/endpoints/ or join our working group.

The post Standardizing Generative AI Service Evaluation: An API-Centric Benchmarking Approach appeared first on MLCommons.