A Collaborative Effort by Industry Leaders

MLPerf Client represents a collaboration among technology leaders, including AMD, Intel, Microsoft, NVIDIA, Qualcomm Technologies, Inc., and top PC OEMs. These stakeholders have pooled resources and expertise to create a standardized benchmark, offering new insight into performance on key consumer AI workloads.

“MLPerf Client is a pivotal step forward in measuring consumer AI PC performance, bringing together industry heavyweights to set a new standard for evaluating generative AI applications on personal computers,” said David Kanter, Head of MLPerf at MLCommons.

Key Features of MLPerf Client v0.5

• AI model: The benchmark’s tests are based on Meta’s Llama 2 7B large language model, optimized for reduced memory and computational requirements via 4-bit integer quantization.
• Tests and metrics: Includes four AI tasks—content generation, creative writing, and text summarization of two different document lengths—evaluated using familiar metrics like time-to-first-token (TTFT) and tokens-per-second (TPS).
• Hardware optimization: Supports hardware-accelerated execution on integrated and discrete GPUs via two distinct paths: ONNX Runtime GenAI and Intel OpenVINO.
• Platform support: This initial release supports Windows 11 on x86-64 systems, with future updates planned for Windows on Arm and macOS.
• Freely accessible: The benchmark is freely downloadable from MLCommons.org, empowering anyone to measure AI performance on supported systems.

Future Development

While version 0.5 marks the benchmark’s debut, MLCommons plans to expand its capabilities in future releases, including support for additional hardware acceleration paths and a broader set of test scenarios incorporating a range of AI models.

Availability

MLPerf Client v0.5 is available for download now from MLCommons.org. See the website for additional details on the benchmark’s hardware and software support requirements. 

For more information and details on becoming a member, please visit MLCommons.org or contact participation@mlcommons.org.

The post MLCommons Introduces MLPerf Client v0.5 appeared first on MLCommons.

As the impact of AI grows and offers new capabilities to everyone, it is increasingly an integral part of the computing experience. Silicon for client systems incorporates AI-specific hardware acceleration capabilities of various types, and OS and application vendors are adding AI-driven features into software to boost productivity and to unleash the creativity of millions of end users. As these hardware and software capabilities proliferate, many ML models will execute locally on client systems. The industry will require reliable, standard ways to measure the performance and efficiency of AI acceleration solutions on client systems.

The MLPerf Client benchmarks will be scenario-driven focusing on real end-user use cases and grounded in feedback from the community. The first benchmark will focus on a large language model, specifically, the Llama 2 LLM. The MLCommons community has already navigated many of the challenges LLMs present in client systems, such as balancing performance against output quality, licensing issues involving datasets and models, and safety concerns through the incorporation of Llama 2-based workloads in the MLCommons training and inference benchmark suites. This learning will help jump-start this new client work.

Initial MLPerf Client working group participants include representatives from AMD, Arm, ASUSTeK, Dell Technologies, Intel, Lenovo, Microsoft, NVIDIA, and Qualcomm Technologies, Inc. among others. 

“The time is ripe to bring MLPerf to client systems, as AI is becoming an expected part of computing everywhere,” said David Kanter, Executive Director at MLCommons®. “Large language models are a natural and exciting starting point for our MLPerf Client working group. We look forward to teaming up with our members to bring the excellence of MLPerf into client systems and drive new capabilities for the broader community.”

We’re happy to announce that Ramesh Jaladi, Senior Director of Engineering in the IP Performance group at Intel; Yannis Minadakis, Partner GM, Software Development at Microsoft; and Jani Joki, Director of Performance Benchmarking at NVIDIA have agreed to serve as co-chairs of the MLPerf Client working group. Additionally, Vinesh Sukumar, Senior Director, AI/ML Product Management at Qualcomm, has agreed to lead a benchmark development task force within the working group.

“Good measurements are the key to advancing AI acceleration,” said Jaladi. “They allow us to set targets, track progress, and deliver improved end-user experiences in successive product generations. The whole industry benefits when benchmarks are well aligned with customer needs, and that’s the role we expect the MLPerf Client suite to play in consumer computing.”

“Microsoft recognizes the need for quality benchmarking tools tailored to the AI acceleration capabilities of Windows client systems, and we welcome the opportunity to collaborate with the MLCommons community to tackle this challenge,” said Minadakis.

“The MLPerf benchmarks have served as a measuring stick for substantial advances in machine learning performance and efficiency in data center solutions,” said Joki. “We look forward to contributing to the creation of benchmarks that will serve a similar role in client systems.”

“Qualcomm is proud to advance the client ecosystem and looks forward to the innovative benchmarks that this MLPerf Working Group will establish for machine learning,” said Sukumar. “Benchmarks remain an important tool in the development and fine tuning of silicon, and MLCommons’ focus on end-user use cases will be key to on-device AI testing.”

We encourage all interested parties to participate in our effort. For more information on the MLPerf Client working group, including information on how to join and contribute to the benchmarks, please visit the working group page or contact the chairs via email at client-chairs@mlcommons.org.

The post Announcing the New MLPerf Client Working Group appeared first on MLCommons.