As that footprint expands, a hard question follows. How do you fairly measure their performance and efficiency when no two of these devices look alike? A single release can include anything from a 60 MHz microcontroller to a vector-enabled RISC-V core to a dedicated neural processing unit.

MLPerf® Tiny is the consortium-built answer. Developed by the MLCommons® Tiny Working Group with EEMBC, it provides an architecture-neutral way to compare ultra-low-power systems using the same workloads, models, and measurement methodology. 

This post covers what MLPerf Tiny measures, why energy is treated as a first-class metric, and what the v1.4 submission round reveals about the direction of edge AI.

The rise of TinyML and why it matters

TinyML refers to ML models that are small enough, typically under 2M weights, to run on microcontrollers and other constrained devices that draw sub-milliwatt to low-milliwatt power. That footprint matters because it opens up places that cloud or smartphone inference cannot go, unlocking new applications and capabilities. 

In 2026, that includes:

• Industrial vibration sensors flagging bearing wear before a machine fails
• Agricultural sensors lasting months on a single charge
• Hearing aids suppressing background noise on-device
• Always-on keyword spotters in earbuds
• Patch wearables monitoring ECG without uploading raw signals

The appeal is practical. Inference on the device keeps data where it’s created instead of sending it to the cloud. That means lower latency, lower cost, better privacy, and, for many battery-powered systems, much longer operating life.

The software ecosystem has matured just as quickly. TensorFlow Lite for Microcontrollers, ExecuTorch, Edge Impulse, STEdgeAI-Core, NXP eIQ, and vendor-specific compilers such as AndesAIRE and RUHMI have made it much easier to deploy models on embedded hardware. But they’ve also exposed the challenge of comparing results across substantially different platforms in a robust and trusted manner. Vendors benchmark different models across different datasets and under varying conditions, so performance numbers rarely tell the whole story.

The benchmarking gap no one had filled

Building a fair benchmark for ultra-low-power ML is hard in itself. The 2021 paper introducing MLPerf Tiny identified four challenges:

• Measuring energy fairly. Power consumption varies from one device to another. On top of that, vendors don’t always measure the same things. Some include peripherals, firmware startup, or I/O activity, while others measure only the inference itself. These differences can change the reported numbers.
• Fitting within tight memory budgets. TinyML devices typically work in kilobytes rather than gigabytes, roughly 6 orders of magnitude tighter than smartphone ML. Reference models, harness overhead, and multiple quantization paths all have to fit in that resource-constrained envelope, which lacks many of the conveniences of cloud systems.
• Software & Hardware heterogeneity. Devices range from general-purpose microcontrollers to neural processors, event-based architectures, and in-memory compute. Also, each vendor typically ships its own tightly coupled toolchain. That makes a portable benchmark non-trivial since that may require over-constraining the stack, which can erase the very performance the benchmark is meant to measure. 

Figure 1: Summary of the Tiny Machine Learning Stack. There is diversity at every level, which makes standardization for benchmarking challenging. 

For years, embedded developers had benchmarks, just not ones that answered the questions they actually cared about.

CoreMark became the standard for measuring Microcontroller Unit (MCU) performance, but it was never designed for ML workloads. MLMark moved closer by benchmarking ML inference, yet its reference models, including ResNet-50, MobileNet, and SSD-MobileNet, target edge AI processors rather than resource-constrained microcontrollers. It also stops short of measuring energy, which is often the limiting factor in embedded deployments.

That leaves a gap. Speed and energy efficiency aren’t the same thing. A model can run fast and still drain the battery. So in TinyML, both have to be measured. 

MLPerf Tiny was created to fill that gap. More than 50 organizations from industry and academia, including Harvard, Google, STMicroelectronics, Qualcomm, Syntiant, Renesas, Infineon, CERN, and Silicon Labs, spent roughly 18 months building the benchmark suite. It is based on compact reference models, representative workloads, and a standardized methodology that treats energy as a first-class metric. The first public results were released as version 0.5 in June 2021.

What MLPerf Tiny actually measures

MLPerf Tiny is designed to answer a simple question. If everyone runs the same ML task, which hardware executes it most efficiently?

For that, it evaluates three metrics, including accuracy, latency, and energy per inference. Each task carries a fixed quality target, so submissions are compared on speed and energy at equivalent accuracy rather than trading one against the other.

Energy is the differentiator. In TinyML, a device that runs an inference in 10 ms but at twice the joules of a competitor is often the wrong choice, because battery life is the binding constraint. MLPerf Tiny uses EEMBC’s EnergyRunner harness to perform calibrated power measurement on the device under test, making energy directly comparable across silicon.

MLPerf Tiny is built to be modular. Improvements can come from anywhere in the stack, whether that’s the chip, the compiler, the runtime, or the model itself. A fixed reference keeps every submission measured against the same bar. 

To make that comparison fair, each task includes a predefined model, dataset, and a minimum accuracy target. Vendors can’t swap in an easier model or lower the accuracy to improve their numbers. Instead, they focus on optimizing how the model runs on their hardware. Once the implementation is ready, MLPerf measures three things, including whether it meets the required accuracy, how long each inference takes, and how much energy each inference consumes.

Visual Wake Words shows how this works in practice. Every submission must determine whether a 96×96 image contains a person, achieving at least 80% accuracy on the official test set. Because every vendor solves the same problem under the same rules, the results are directly comparable. For example, ASYGN’s ColibriNPU completed each inference using just 22.2 microjoules. It’s low enough that a CR2032 coin-cell battery could perform one inference every second for more than three years.

That’s the key idea behind MLPerf Tiny. The workload, evaluation, and accuracy targets are all fixed. What changes is how efficiently each hardware and software stack delivers the result.

The five benchmarks and the real-world problems they represent

MLPerf Tiny started with four tasks in v0.5 (2021) and expanded to five in v1.3 (2025). Each task corresponds to a recurring TinyML deployment pattern:

• Keyword Spotting (KWS): A small DS-CNN trained on the Google Speech Commands dataset, representative of wake-word and voice-command detection in smart speakers, earbuds, and hearables.
• Visual Wake Words (VWW): A MobileNetV1 binary classifier on 96×96 images, mirroring person-presence detection in low-power vision sensors for doorbells, security cameras, and occupancy sensing.
• Image Classification (IC): A ResNet-style network on CIFAR-10, representative of general low-resolution vision workloads on edge devices.
• Anomaly Detection (AD): An autoencoder trained on industrial machine sounds (ToyADMOS), representative of predictive maintenance and condition monitoring.
• Streaming Wake Word: Added in v1.3, a 1D depthwise-separable CNN that detects a target word in a continuous audio stream. Unlike other benchmarks, this one also measures the device while it’s idle and listening, which is how wake-word systems actually run most of the time. These are the conditions under which most production wake-word systems actually run.

No benchmark can cover every embedded AI workload. These five capture the patterns that appear repeatedly in commercial TinyML products, making them a practical baseline for comparison. 

A benchmark built for everyone in the stack

MLPerf Tiny has two divisions because not everyone wants to measure the same thing.

1. The Closed Division fixes the reference model, dataset, and processing pipeline. That leaves hardware and low-level software optimizations as the only variables, making results directly comparable across platforms.
2. The Open Division is more flexible. Participants can use their own models, training methods, and optimizations, as long as they solve the same task and meet the required accuracy target. That makes it a place to evaluate new architectures, compiler techniques, and research ideas without losing the ability to compare against established benchmarks.

Figure 2: MLPerf Tiny’s modular design allows direct comparisons and shows improvement over the reference. Each reference implementation is swappable: green components can change in either division, orange only in the open division. 

MLPerf Tiny serves three different audiences:

• Hardware vendors use it to prove their silicon
• Software teams to evaluate their compilers and runtimes
• Researchers to validate novel architectures, and potential customers to evaluate candidate hardware and software platforms against a neutral standard.

The v1.4 round includes examples of all three. 

What the results are already telling us

The v1.4 round closed with submissions from nine organizations, including Andes, ASYGN, DeepGate, Kai Jiang, Qualcomm, Renesas, STMicroelectronics, Syntiant, and the University of Leeds. Together, they submitted 25 system configurations across the Closed and Open Divisions. Five of the participating organizations were first-time submitters, more than doubling the participant count from v1.3.

These submissions highlight the real value of MLPerf Tiny. Every result is measured against the same task, model, accuracy target, and energy methodology, making improvements directly comparable instead of relying on vendor claims. 

A few signals stand out:

• Dedicated accelerators are moving into MCU-class parts. STMicroelectronics reports that enabling the hardware signal processor on the STM32U3 reduces image-classification inference time by up to 76.0% and 23.3% lower power on ic2 versus the same Cortex-M33 configuration. Additionally, the STM32H7P preview, with its Neural-ART NPU, improves inference time by up to 96.0% on the same workload compared to its CM7-only baseline.
• Sensing-hub architectures are advancing rapidly. Qualcomm’s submission on the Snapdragon 8 Elite Gen 5 Sensing Hub reports legacy workload latencies below 0.30 ms for keyword spotting, visual wake words, and image classification.
• Power efficiency is winning on streaming workloads. Syntiant’s NDP120 runs the streaming wake-word benchmark at a 3.3% duty cycle, meaning the device is idle ~97% of the time, with the remaining capacity available for concurrent tasks like noise cancellation or beamforming. 
• Open Division innovation is healthy. The University of Leeds reports a median throughput of 1,849.7 inferences per second for image classification on its Versal-based heterogeneous platform, with a custom on-fabric batch-normalization engine running at 313 MHz.

Together, the round shows that the field is maturing beyond CPU-only baselines. Hardware acceleration, streaming workloads, and heterogeneous compute have become the defining characteristics of competitive TinyML systems.

Conclusion

Standardization is what makes engineering progress measurable. By fixing the models, datasets, accuracy targets, and energy measurement methodology, MLPerf Tiny allows the entire stack, including silicon, compilers, runtimes, and models, to improve against a shared reference. That shared foundation is how responsible and efficient edge AI gets built. It depends not on the claims of any single vendor, but on a community-maintained benchmark that any submitter can reproduce and any practitioner can trust.

To explore the full v1.4 results, visit the MLPerf Tiny results page and read the supplemental containing details of all our submitters’ results. To contribute to future rounds, join the MLPerf Tiny Working Group.

The post The Benchmark Behind the Next Wave of Ultra-Low-Power AI appeared first on MLCommons.

MLPerf Tiny, introduced in August 2021, is the first industry-standard benchmark suite testing performance at inference across ultra-low-power tiny machine learning systems. Developed from the collaborative effort of over 50 organizations, the MLPerf Tiny benchmark suite arose from the need to evaluate devices that process these “tiny” neural networks in a fair, standardized, and reproducible manner. These tiny networks, typically with fewer than 1 million parameters, handle sensory inputs such as audio and vision to bring AI capabilities to power-constrained form factors. With TinyML, inference occurs directly on-device and near sensors, resulting in faster responses and enhanced privacy. This approach sidesteps the high energy costs of wireless data transmission, often a much larger drain than the computation itself. 

The goal of MLPerf Tiny is to provide a representative set of deep neural nets and benchmarking code to compare performance between embedded devices. Embedded devices include microcontrollers, Digital Signal Processors (DSPs), and tiny neural network accelerators. These devices typically run at between 10MHz and 250MHz, and can perform inference with less than 50mW of power.

MLPerf Tiny Benchmark Suite

The MLCommons Tiny Working Group spearheads the development and ongoing management of MLPerf Tiny. To ensure fairness, comparability, and to foster innovation, MLPerf Tiny operates with two distinct divisions:

Closed Division: This division focuses on “apples-to-apples” comparisons. Submissions must use the same specified model, dataset, and metrics, allowing for direct hardware comparisons where only the underlying hardware and optimized software stack are variables.

Open Division: This division encourages broader innovation by allowing submitters to use different or custom models, datasets, or training methods.

Three key metrics – accuracy, latency and energy –  are used to weigh the tradeoffs between systems across tasks such as keyword spotting, visual wake words, image classification, and anomaly detection. A dataset and quality target define each of these task-specific benchmarks.

Keyword Spotting

Wake word detection, a type of keyword spotting mainly used for voice applications, is a popular application of a TinyML use case. Low power consumption, cost, and latency are critical for wake word detection to continuously monitor for a specific word or phrase. This Keyword Spotting benchmark leverages the Speech commands v2 dataset, which contains utterances from speakers with a diverse range of accents. A small depth-wise convolutional neural network (CNN) is used in the closed division of this benchmark. At 38.6K parameters, it fits the memory requirements for most resource-constrained devices. An accuracy requirement of 90% is set in this Keyword Spotting benchmark to account for slight variations from quantization strategies. 

Visual Wake Words

The Visual Wake Words Challenge resembles a common microcontroller vision use-case of identifying whether a person is present in an image or not (e.g., smart doorbell applications).  

The MSCOCO 2014 dataset is used for training, validation, and testing, with images resized to 96×96 pixels. The model used is a MobileNetV1 with two output classes: “person” and “no person”. The 80% accuracy quality target metric is to account for the changes in accuracy that occur from quantizations and rounding differences between platforms.

Image Classification

As advancements in machine vision enable compact, efficient, and high-performance embedded systems, they are revolutionizing manufacturing, IoT, and autonomous technology. Consequently, there is a rising demand for standardized evaluation frameworks to classify images using embedded devices. 

The  Image Classification benchmark of MLPerf Tiny uses the CIFAR-10 dataset, which contains 60,000 32x32x3 RGB images across 10 classes (e.g., airplanes, cars, cats, dogs). A custom ResNetv1 model, which is a modified version of the official ResNet, takes these 32x32x3 images as input and produces a probability vector of size 10. The quality target for this benchmark is 85% top-1 accuracy to accommodate minor differences in quantization and various other optimizations.

Anomaly Detection

Anomaly detection separates normal from anomalous samples using unsupervised learning. This is crucial for industrial applications like machine failure detection, where failure data is scarce and training relies mainly on data of standard operation. The Anomaly Detection benchmark of MLPerf Tiny utilizes unsupervised learning with an autoencoder (AE) model, serving as the reference implementation in the DCASE2020 competition, alongside the toy-car machine subset of its dataset. The threshold for this benchmark was set to AUC 0.85.

How MLPerf Tiny Overcame Traditional Challenges with Benchmarking TinyML

Overcoming Memory Limitations

Memory limitations pose a challenge for TinyML systems, as even a benchmark’s overhead can exceed a device’s capacity. To overcome this, the MLPerf Tiny suite is designed with compactness in mind. It accommodates diverse hardware by allowing multiple quantization levels and precision variants. A lightweight communication protocol is used over a standard UART (Universal Asynchronous Receiver/Transmitter) interface to minimize the memory footprint for data transfer. Additionally, the benchmarks use efficient models specifically selected to test common computational patterns while remaining within the tight memory and storage constraints of most embedded devices.

Overcoming Hardware and Software Heterogeneity

The TinyML landscape is highly fragmented, with a vast array of microcontrollers, custom accelerators, and specialized processors. This hardware heterogeneity poses a significant challenge for traditional benchmarks, which were often designed for more uniform computing environments.

To address this, the MLPerf Tiny benchmarks take a unique approach by focusing on the model itself, rather than the underlying hardware or software implementation.

Even in the closed division, there are no instruction-level requirements. The only rule is that the submitted model must be mathematically equivalent to the reference model. This design enables a wide range of hardware platforms to implement and optimize the benchmark, accommodating various architectures and specialized accelerators.

Similarly, the diversity of software stacks in the TinyML space presents a challenge for achieving consistent and reproducible measurements. Software optimizations are often tightly coupled with specific hardware platforms, making a one-size-fits-all approach impractical. The MLPerf Tiny benchmarks acknowledge this software heterogeneity by avoiding constraints on specific machine learning runtimes, libraries, or other software components. Any system submitted to the closed division only needs to implement a model mathematically equivalent to the reference and support a minimal communication protocol.

This flexibility is demonstrated with the benchmark suite’s modular design, which has enabled a wide variety of systems that use both open-source and proprietary runtimes to participate in the benchmarks.

Modular Design

The modular design of MLPerf Tiny allows for flexible benchmarking and direct comparisons between different solutions. Each benchmark has a reference implementation that includes everything from the training data and script to a reference hardware platform. This provides a baseline result, allowing submitters to modify specific components and demonstrate their competitive advantage. 

The components of the benchmark are divided into two categories in the figure below.

• Green components (e.g., Quantization, Compiler, Inference Framework, Kernels, SoC, CPU, and Accelerator) can be modified in both the Open and Closed divisions.
• Orange components (e.g., Training Data, Training Script, Model Architecture, Pruning) can only be modified in the Open division.

Limitations of MLPerf Tiny

One limitation of this benchmark suite is that it doesn’t account for continuous streaming inputs, where information from previous time steps stays relevant for future inference cycles. The prior energy measurements didn’t accurately reflect duty-cycled deployment scenarios, which are common in real-world applications. Additionally, the earlier benchmark’s feature extraction was performed on the host, which is not representative of typical on-device processing. Finally, the use of static inference created ambiguity regarding the data loading process, making it difficult to get an accurate picture of performance. To address these limitations, MLCommons introduces a new streaming benchmark in the current iteration (V1.3).

A New Streaming Benchmark

The streaming benchmark evaluates the real-time detection of a wake word, such as “Hey Siri” or “Alexa,” in a noisy audio stream. The specific wakeword used for this test is “Marvin,” selected because it has the most recorded samples among the multi-syllabic and non-numeric words in the Speech Commands dataset.

The test uses a synthesized 20-minute audio recording that includes 50 instances of the “Marvin” wake word, along with various background noises like music, speech, and traffic. This scenario mimics real-world applications where a device needs to identify a specific sound pattern consistently.

A 1D time-separable model, similar to Nvidia’s Matchbox model, was chosen for its balance of performance and efficiency. The model was trained on the wakeword itself, other words from the Speech Commands dataset, and a variety of background noises. This approach isn’t limited to wakeword detection; it can be applied to other acoustic events like gunshots or baby cries, or even non-acoustic events like vibrational patterns in industrial monitoring.

A key challenge in this streaming benchmark is the real-time synchronous delivery and collection of input and output. To ensure accuracy, a detection is considered correct only if it occurs within one second of the end of the actual wakeword.

A microcontroller unit (MCU)-based interface board was used to synchronize the system. It delivered audio to the device under test (DUT) and recorded a timestamp whenever the DUT pulsed a GPIO (General Purpose Input/Output) pin to signal a detection.

The interface board also measured two other critical metrics:

Duty Cycle: A second GPIO signal from the DUT indicates when it was actively processing. The interface board captures this “Active” signal to measure the duty cycle, which represents the percentage of time the processor is busy. A low duty cycle is desirable, as it means the device has spare processing cycles for other tasks.

Power Consumption: A power monitor measures the DUT’s energy usage. An efficient system must not only run the neural model with low energy but also quickly transition to a low-power idle state when not processing audio.

MLPerf Tiny v1.3 Benchmark Overview

Top-1 accuracy is a metric used in multi-class classification to evaluate the performance of a model. For example, an 85% top-1 accuracy for image classification means that for 85% of the images tested, the model’s highest-confidence prediction was the correct one.

Conclusion

MLPerf Tiny provides an industry-standard benchmark suite for evaluating ultra-low-power tiny machine learning systems. It addresses challenges like memory limitations and hardware/software heterogeneity through its compact and flexible design, offering both a Closed Division for direct comparisons and an Open Division for innovation. The suite utilizes metrics such as accuracy, latency, and energy across various tasks, including keyword spotting, visual wake words, image classification, and anomaly detection. Notably, a new streaming benchmark has been introduced to address previous limitations, focusing on real-time detection and incorporating measurements of duty cycle and power consumption.

Call to Action

We invite you to learn more about MLPerf Tiny and get involved. Explore the benchmarks, review the results, and consider joining the MLCommons Tiny Working Group. Your contributions will help shape the future of TinyML. Additionally, we are seeking interviewees to gather community input and explore possibilities for enriching new benchmarks.

The post A New TinyML Streaming Benchmark for MLPerf Tiny v1.3 appeared first on MLCommons.

Version 1.3 adds a new test: a one-dimensional depthwise separable convolutional neural network (1D DS-CNN). A 1D DS-CNN is trained on sequential data, such as sensor readings or audio waveforms, and is often used to identify signals, triggers, or threshold events in continuous, real-time data streams. The new test measures performance in recognizing “wake words” in a continuous audio stream. Tiny ML deployments often monitor a continuous stream of data from a sensor, such as a microphone, accelerometer, or camera. The streaming scenario in the new test exercises capabilities important to these deployments, such as low-power idle, rapid wake-up, data ingestion, and feature extraction. The wake-word detection is just one example of a streaming task.  Others include speech enhancement, real-time translation, or industrial monitoring for preventive maintenance. The capability to evaluate streaming scenarios opens up many new opportunities.

This version also introduces a new, open-source test harness for running the benchmark suite. This simplifies the process for submitters of gaining access to and executing the test harness, as well as troubleshooting any issues that occur while running the tests.

MLPerf Tiny v1.3 participation and submitters

This release includes 70 results across five benchmark tests, submitted by four participants: Kai Jiang, Qualcomm, ST Microelectronics, and Syntiant. This includes 27 power results, an increase from the previous release. Five hardware platforms were benchmarked for the first time in this release.

View the Results

To view the results for MLPerf Tiny v1.3, please visit the Tiny benchmark results.

We invite stakeholders to join the MLPerf Tiny working group and help us continue to evolve the benchmark suite.

About MLCommons

MLCommons is the world’s leader in AI benchmarking. An open engineering consortium supported by over 125 members and affiliates, MLCommons has a proven record of bringing together academia, industry, and civil society to measure and improve AI. The foundation for MLCommons began with the MLPerf benchmarks in 2018, which rapidly scaled as a set of industry metrics to measure machine learning performance and promote transparency of machine learning techniques. Since then, MLCommons has continued using collective engineering to build the benchmarks and metrics required for better AI – ultimately helping to evaluate and improve AI technologies’ accuracy, safety, speed, and efficiency.

For additional information on MLCommons and details on becoming a member, please visit MLCommons.org or email participation@mlcommons.org.

Press Inquiries: contact press@mlcommons.org

The post MLCommons New MLPerf Tiny 1.3 Benchmark Results Released appeared first on MLCommons.

Machine learning inference on the edge is fast becoming a popular way to add intelligence to different devices and increase energy efficiency, privacy, responsiveness, and autonomy. The MLPerf Tiny benchmark suite captures inference use cases that involve “tiny” neural networks and tests them in a fair and reproducible manner. These networks are typically under 100 kB and process data from sensors including audio and vision to provide endpoint intelligence for low-power devices in the smallest form factors.

“We are pleased by the continued adoption of the MLPerf Tiny benchmark suite throughout the industry,” said David Kanter, Executive Director of MLCommons. “The diversity of submissions shows us that the industry is embracing AI through increased software support, which makes our benchmarking work all the more important.”

This latest round of MLPerf Tiny results includes submissions from Bosch, Kai Jiang (individual), Qualcomm Technologies, Inc., Renesas, STMicroelectronics, Skymizer, and Syntiant, with 91 overall performance results including 18 energy measurements. The results included a range of new, capable hardware systems designed to take advantage of AI-powered processes and the latest software stacks that increase performance and efficiency.

“We are pleased to see the MLPerf Tiny benchmark being used to characterize a wide range of low-power systems, including a variety of microprocessor architectures and AI-enabled low-power sensing hubs,” said Csaba Kiraly, MLPerf Tiny working group co-chair. “Congratulations to all the submitters.”

“The Tiny ML community is continuing to push the envelope with multiple new systems incorporating AI-specific features as well as new software stacks,” said Jeremy Holleman, co-chair of the MLPerf Tiny working group.

View the Results
View the MLPerf Tiny v1.2 benchmark results.

The post MLPerf Tiny v1.2 Results appeared first on MLCommons.

Faster training paves the way for more capable intelligent systems

Training models faster empowers researchers to unlock new capabilities, such as the latest advances in generative AI. The latest MLPerf Training round demonstrates broad industry participation and highlights performance gains of up to 1.54x compared to just six months ago and 33-49x over the first round, reflecting the tremendous rate of innovation in systems for machine learning.

The MLPerf Training benchmark suite comprises full system tests that stress machine learning models, software, and hardware for a broad range of applications. The open-source and peer-reviewed benchmark suite provides a level playing field for competition that drives innovation, performance, and energy-efficiency for the entire industry.

In this round, MLPerf Training added two new benchmarks to the suite. The first is a large language model (LLM) using the GPT-3 reference model that reflects the rapid adoption of generative AI. The second is an updated recommender, modified to be more representative of industry practices, using the DLRM-DCNv2 reference model. These new tests help advance AI by ensuring that industry-standard benchmarks are representative of the latest trends in adoption and can help guide customers, vendors, and researchers alike.

“I’m excited to see the debut of GPT-3 and DLRM-DCNv2, which were built based on extensive feedback from the community and leading customers and demonstrate our commitment to keep the MLPerf benchmarks representative of modern machine learning,” said David Kanter, executive director of MLCommons®.

The MLPerf Training v3.0 round includes over 250 performance results, an increase of 62% over the last round, from 16 different submitters: ASUSTek, Azure, Dell, Fujitsu, GIGABYTE, H3C, IEI, Intel & Habana Labs, Krai, Lenovo, NVIDIA, NVIDIA + CoreWeave, Quanta Cloud Technology, Supermicro, and xFusion. In particular, MLCommons would like to congratulate first time MLPerf Training submitters CoreWeave, IEI, and Quanta Cloud Technology.

“It is truly remarkable to witness system engineers continuously pushing the boundaries of performance on workloads that hold utmost value for users via MLPerf,” said Ritika Borkar, co-chair of the MLPerf Training Working Group. “We are particularly thrilled to incorporate an LLM benchmark in this round, as it will inspire system innovation for a workload that has the potential of revolutionizing countless applications.”

MLPerf Tiny Results Reflect the Rapid Pace of Embedded Devices Innovation

Tiny compute devices are a pervasive part of everyone’s everyday life, from tire sensors in your vehicles to your appliances and even your fitness tracker. Tiny devices bring intelligence to life at very little cost.

ML inference on the edge is increasingly attractive to increase energy efficiency, privacy, responsiveness, and autonomy of edge devices. Tiny ML breaks the traditional paradigm of energy and compute hungry ML by eliminating networking overhead, allowing for greater overall efficiency and security relative to a cloud-centric approach. The MLPerf Tiny benchmark suite captures a variety of inference use cases that involve “tiny” neural networks, typically 100 kB and below, that process sensor data, such as audio and vision, to provide endpoint intelligence for low-power devices in the smallest form factors. MLPerf Tiny tests these capabilities in a fair and reproducible manner, in addition to offering optional power measurement.

In this round, the Tiny ML v1.1 benchmarks include 10 submissions from academic, industry organizations, and national labs, producing 159 peer-reviewed results. Submitters include: Bosch, cTuning, fpgaConvNet, Kai Jiang, Krai, Nuvoton, Plumerai, Skymizer, STMicroelectronics, and Syntiant. This round includes 41 power measurements, as well. MLCommons congratulates Bosch, cTuning, fpgaConvNet, Kai Jiang, Krai, Nuvoton, and Skymizer on their first submissions to MLPerf Tiny.

“I’m particularly excited to see so many companies embrace the Tiny ML benchmark suite,” said David Kanter, Executive Director of MLCommons. “We had 7 new submitters this round which demonstrates the value and importance of a standard benchmark to enable device makers and researchers to choose the best solution for their use case.”

“With so many new companies adopting the benchmark suite it’s really extended the range of hardware solutions and innovative software frameworks covered. The v1.1 release includes submissions ranging from tiny and inexpensive microcontrollers to larger FPGAs, showing a large variety of design choices,” said Dr. Csaba Kiraly, co-chair of the MLPerf Tiny Working Group. “And the combined effect of software and hardware performance improvements are 1000-fold in some areas compared to our initial reference benchmark results, which shows the pace that innovation is happening in the field.”

View the Results

To view the results for MLPerf Training v3.0 and MLPerf Tiny v1.1, and to find additional information about the benchmarks please visit:
Training v3.0 and Tiny v1.1.

About MLCommons

MLCommons is an open engineering consortium with a mission to make machine learning better for everyone through benchmarks and data. The foundation for MLCommons began with the MLPerf benchmark in 2018, which rapidly scaled as a set of industry metrics to measure machine learning performance and promote transparency of machine learning techniques. In collaboration with its 50+ members – global technology providers, academics, and researchers, MLCommons is focused on collaborative engineering work that builds tools for the entire machine learning industry through benchmarks and metrics, public datasets, and best practices.

For additional information on MLCommons and details on becoming a Member or Affiliate, please visit MLCommmons or contact participation@mlcommons.org.

The post MLPerf Results Show Rapid AI Performance Gains appeared first on MLCommons.

The MLPerf benchmark suites are comprehensive system tests that stress machine learning models including underlying software and hardware and in some cases, optionally measuring energy usage. The open-source and peer-reviewed benchmark suites create a level playing ground for competition, which fosters innovation and benefits society at large through better performance and energy efficiency for AI and ML applications.

The MLPerf Training benchmark suite measures the performance for training machine learning models that are used in commercial applications such as recommending movies, speech-to-text, autonomous vehicles, and medical imaging. MLPerf Training v2.1 includes nearly 200 results from 18 different submitters spanning all the way from small workstations up to large scale data center systems with thousands of processors.

The MLPerf HPC benchmark suite is targeted at supercomputers and measures the time it takes to train machine learning models for scientific applications and also incorporates an optional throughput metric for large systems that commonly support multiple users. The scientific workloads include weather modeling, cosmological simulation, and predicting chemical reactions based on quantum mechanics. MLPerf HPC 2.0 includes over 20 results from 5 organizations with time-to-train and throughput for all models and submissions from some of the world’s largest supercomputers.

The MLPerf Tiny benchmark suite is intended for the lowest power devices and smallest form factors, such as deeply embedded, intelligent sensing, and internet-of-things applications. It measures inference performance – how quickly a trained neural network can process new data and includes an optional energy measurement component. MLPerf Tiny 1.0 encompasses submissions from 8 different organizations including 59 performance results with 39 energy measurements or just over 66% – an all-time record.

“We are pleased to see the growth in the machine learning community and especially excited to see the first submissions from xFusion for MLPerf Training, Dell in MLPerf HPC and GreenWaves Technologies, OctoML, and Qualcomm in MLPerf Tiny,” said MLCommons Executive Director David Kanter. “The increasing adoption of energy measurement is particularly exciting, as a demonstration of the industry’s outstanding commitment to efficiency.”

To view the results and find additional information about the benchmarks please visit: https://mlcommons.org/en/training-normal-21/, https://mlcommons.org/en/training-hpc-20/, and https://www.mlcommons.org/en/inference-tiny-10/

About MLCommons

MLCommons is an open engineering consortium with a mission to benefit society by accelerating innovation in machine learning. The foundation for MLCommons began with the MLPerf benchmark in 2018, which rapidly scaled as a set of industry metrics to measure machine learning performance and promote transparency of machine learning techniques. In collaboration with its 50+ founding partners – global technology providers, academics and researchers, MLCommons is focused on collaborative engineering work that builds tools for the entire machine learning industry through benchmarks and metrics, public datasets and best practices.

For additional information on MLCommons and details on becoming a Member or Affiliate of the organization, please visit http://mlcommons.org/ and contact participation@mlcommons.org.

Press Contact:
David Kanter
press@mlcommons.org

The post Latest MLPerf Results Display Gains for All appeared first on MLCommons.

The MLPerf benchmarks are full system tests that stress machine learning models, software, and hardware and optionally measure power usage. The open-source and peer-reviewed benchmark suites provide a level playing field for competition that drives innovation, performance, and energy-efficiency for the entire industry.

“This was an outstanding effort by the ML community with so many new participants and the tremendous increase in the number and diversity of submissions.” said David Kanter, Executive Director of MLCommons. “I’m especially excited to see greater adoption of power and energy measurements, highlighting the industry’s focus on efficient AI.”

The MLPerf Inference benchmarks primarily focus on datacenter and edge systems and submitters include Alibaba, ASUSTeK, Azure, Deci.ai, Dell, Fujitsu, FuriosaAI, Gigabyte, H3C, Inspur, Intel, Krai, Lenovo, Nettrix, Neuchips, NVIDIA, Qualcomm Technologies, Inc., Supermicro, and ZhejiangLab. This round set new records with over 3,900 performance results and 2,200 power measurements, respectively 2X and 6X more than the prior round, demonstrating the momentum of the community.

The MLPerf Mobile benchmark suite targets smartphones, tablets, notebooks, and other client systems with the latest submissions highlighting an average 2X performance gain over the previous round. MLPerf Mobile v2.0 includes a new image segmentation model, MOSAIC, that was developed by Google Research with feedback from MLCommons. The MLPerf Mobile application and the corresponding source code, which incorporates the latest updates and submitting vendors’ backends, are expected to be available in the second quarter of 2022.

The MLPerf Tiny benchmark suite is intended for the lowest power devices and smallest form factors, such as deeply embedded, intelligent sensing, and internet-of-things applications. The second round of MLPerf Tiny results showed tremendous growth in collaboration with submissions from Alibaba, Andes, hls4ml-FINN team, Plumerai, Renesas, Silicon Labs, STMicroelectronics, and Syntiant. Collectively, these organizations submitted 19 different systems with 3X more results than the first round and over half the results incorporating energy measurements, an impressive achievement for the first benchmarking round with energy measurement.

MLCommons would like to congratulate first time MLPerf Inference submitters ASUSTeK, Azure, H3C, and ZhejiangLab and also Gigabyte and Fujitsu for their first power measurements along with first time MLPerf Tiny submitters Alibaba, Andes, Plumerai, Renesas, Silicon Labs, and STMicroelectronics and also the hls4ml-FINN team and Syntiant on their first energy measurements.

To view the results and find additional information about the benchmarks please visit https://mlcommons.org/en/inference-datacenter-20/,
https://mlcommons.org/en/inference-edge-20/,
https://mlcommons.org/en/inference-mobile-20/, and https://www.mlcommons.org/en/inference-tiny-07/

About MLCommons

MLCommons is an open engineering consortium with a mission to benefit society by accelerating innovation in machine learning. The foundation for MLCommons began with the MLPerf benchmark in 2018, which rapidly scaled as a set of industry metrics to measure machine learning performance and promote transparency of machine learning techniques. In collaboration with its 50+ founding partners – global technology providers, academics and researchers, MLCommons is focused on collaborative engineering work that builds tools for the entire machine learning industry through benchmarks and metrics, public datasets and best practices.

For additional information on MLCommons and details on becoming a Member or Affiliate of the organization, please visit http://mlcommons.org/ and contact participation@mlcommons.org.

Press Contact:
David Kanter
press@mlcommons.org

The post MLPerf Results Show Advances in Machine Learning Inference Performance and Efficiency appeared first on MLCommons.

Embedded machine learning is a burgeoning field where AI-driven sensor data analytics is performed in real-time, close to where the data resides. The new MLPerf Tiny Inference benchmark suite captures a variety of use cases that involve “tiny” neural networks, typically 100 kB and below, that process sensor data such as audio and vision to provide endpoint intelligence.

The first v0.5 round included five submissions from academic, industry organizations, and national labs, producing 17 peer-reviewed results. Submissions this round included software and hardware innovations from Latent AI, Syntiant, PengCheng Labs, Columbia, UCSD, CERN, and Fermilab. To view the results, please visit https://www.mlcommons.org/en/inference-tiny-05/.

MLPerf Tiny Inference: A New Measurement to Advance Intelligence in Everyday Devices

As a new benchmark, MLPerf Tiny Inference enables reporting and comparison of embedded ML devices, systems, and software. Developed in partnership with EEMBC, the Embedded Microprocessor Benchmark Consortium, the benchmark consists of four machine learning tasks that encompass the use of microphone and camera sensors with embedded devices:

• Keyword Spotting (KWS), which uses a neural network that detects keywords from a spectrogram;
• Visual Wake Words (VWW), a binary image classification task for determining the presence of a person in an image;
• Tiny Image Classification (IC), a small image classification benchmark with 10 classes; and
• Anomaly Detection (AD), which uses a neural network to identify abnormalities in machine operating sounds.

KWS has several use cases in endpoint consumer devices, such as earbuds and virtual assistants. VWW has application use cases, for instance, with in-home security monitoring. IC has myriad use cases for smart video recognition applications. AD has several applications in industrial manufacturing for tasks such as predictive maintenance, asset tracking and monitoring.

“To understand progress and advance innovation, particularly in edge computing, the ML industry needs benchmarks,” said Peter Torelli, President of EEMBC. “Creating new metrics and measurement across neural networks and a variety of form factors is challenging, and we’re thrilled to partner with MLCommons to make MLPerf Tiny a reality.”

“The goal of MLPerf is to measure performance for machine learning across the full spectrum of systems – from microwatts to megawatts,” said Professor Vijay Janapa Reddi of Harvard University and MLPerf Tiny Inference working group chair. “This new benchmark will bring intelligence to devices like wearables, thermostats, and cameras, and further MLCommons’ mission to accelerate machine learning innovation to benefit everyone.”

“Tiny machine learning is a fast-growing field and will help to infuse ‘intelligence’ in the small everyday items that surround us,” said MLPerf Tiny Inference working group chair, Colby Banbury, of Harvard University. “By bringing MLPerf benchmarks to these devices, we can help to measure performance and drive efficiency improvements over time.”

MLPerf Tiny v0.5 marks a major milestone in MLCommons’ line-up of MLPerf inference benchmark suites. With the addition of MLPerf Tiny, MLCommons covers the full range of machine learning inference benchmarks, ranging from cloud and datacenter benchmarks that consume kiloWatts of power down to tiny IoT devices that consume only a few milliWatts of power, and everything in between. MLPerf Tiny benchmarks will stimulate tinyML innovation in the academic and research communities and push the state-of-the-art forward in embedded machine learning.

The MLPerf Tiny v0.5 inference benchmarks were created thanks to the contributions and leadership of our working members over the last 18 months, including representatives from: Harvard University, EEMBC, CERN, Columbia, Digital Catapult, Fermilab, Google, Infineon, Latent AI, ON Semiconductor, Peng Cheng Laboratories, Qualcomm, Renesas, SambaNova Systems, Silicon Labs, STMicroelectronics, Synopsys, Syntiant, UCSD, and VoiceMed.

The MLPerf Tiny working group recently submitted a paper to the NeurIPS benchmarks and datasets track that provides in-depth information about the design and implementation of the benchmark suite (https://openreview.net/pdf?id=8RxxwAut1BI). Additional information about the MLPerf Tiny Inference benchmarks is available at the github repository.

About MLCommons

MLCommons is an open engineering consortium with a mission to accelerate machine learning innovation, raise all boats and increase its positive impact on society. The foundation for MLCommons began with the MLPerf benchmark in 2018, which rapidly scaled as a set of industry metrics to measure machine learning performance and promote transparency of machine learning techniques. In collaboration with its 50+ founding member partners – global technology providers, academics and researchers, MLCommons is focused on collaborative engineering work that builds tools for the entire machine learning industry through benchmarks and metrics, public datasets and best practices.

For additional information on MLCommons and details on becoming a member of the organization, please visit http://mlcommons.org/ or contact membership@mlcommons.org.

Press Contact:
mlcommons@strangebrewstrategies.com

The post MLPerf Tiny Inference Benchmark appeared first on MLCommons.