Machine learning inference on the edge is fast becoming a popular way to add intelligence to different devices and increase energy efficiency, privacy, responsiveness, and autonomy. The MLPerf Tiny benchmark suite captures inference use cases that involve “tiny” neural networks and tests them in a fair and reproducible manner. These networks are typically under 100 kB and process data from sensors including audio and vision to provide endpoint intelligence for low-power devices in the smallest form factors.

“We are pleased by the continued adoption of the MLPerf Tiny benchmark suite throughout the industry,” said David Kanter, Executive Director of MLCommons. “The diversity of submissions shows us that the industry is embracing AI through increased software support, which makes our benchmarking work all the more important.”

This latest round of MLPerf Tiny results includes submissions from Bosch, Kai Jiang (individual), Qualcomm Technologies, Inc., Renesas, STMicroelectronics, Skymizer, and Syntiant, with 91 overall performance results including 18 energy measurements. The results included a range of new, capable hardware systems designed to take advantage of AI-powered processes and the latest software stacks that increase performance and efficiency.

“We are pleased to see the MLPerf Tiny benchmark being used to characterize a wide range of low-power systems, including a variety of microprocessor architectures and AI-enabled low-power sensing hubs,” said Csaba Kiraly, MLPerf Tiny working group co-chair. “Congratulations to all the submitters.”

“The Tiny ML community is continuing to push the envelope with multiple new systems incorporating AI-specific features as well as new software stacks,” said Jeremy Holleman, co-chair of the MLPerf Tiny working group.

View the Results
View the MLPerf Tiny v1.2 benchmark results.

The post MLPerf Tiny v1.2 Results appeared first on MLCommons.

Faster training paves the way for more capable intelligent systems

Training models faster empowers researchers to unlock new capabilities, such as the latest advances in generative AI. The latest MLPerf Training round demonstrates broad industry participation and highlights performance gains of up to 1.54x compared to just six months ago and 33-49x over the first round, reflecting the tremendous rate of innovation in systems for machine learning.

The MLPerf Training benchmark suite comprises full system tests that stress machine learning models, software, and hardware for a broad range of applications. The open-source and peer-reviewed benchmark suite provides a level playing field for competition that drives innovation, performance, and energy-efficiency for the entire industry.

In this round, MLPerf Training added two new benchmarks to the suite. The first is a large language model (LLM) using the GPT-3 reference model that reflects the rapid adoption of generative AI. The second is an updated recommender, modified to be more representative of industry practices, using the DLRM-DCNv2 reference model. These new tests help advance AI by ensuring that industry-standard benchmarks are representative of the latest trends in adoption and can help guide customers, vendors, and researchers alike.

“I’m excited to see the debut of GPT-3 and DLRM-DCNv2, which were built based on extensive feedback from the community and leading customers and demonstrate our commitment to keep the MLPerf benchmarks representative of modern machine learning,” said David Kanter, executive director of MLCommons®.

The MLPerf Training v3.0 round includes over 250 performance results, an increase of 62% over the last round, from 16 different submitters: ASUSTek, Azure, Dell, Fujitsu, GIGABYTE, H3C, IEI, Intel & Habana Labs, Krai, Lenovo, NVIDIA, NVIDIA + CoreWeave, Quanta Cloud Technology, Supermicro, and xFusion. In particular, MLCommons would like to congratulate first time MLPerf Training submitters CoreWeave, IEI, and Quanta Cloud Technology.

“It is truly remarkable to witness system engineers continuously pushing the boundaries of performance on workloads that hold utmost value for users via MLPerf,” said Ritika Borkar, co-chair of the MLPerf Training Working Group. “We are particularly thrilled to incorporate an LLM benchmark in this round, as it will inspire system innovation for a workload that has the potential of revolutionizing countless applications.”

MLPerf Tiny Results Reflect the Rapid Pace of Embedded Devices Innovation

Tiny compute devices are a pervasive part of everyone’s everyday life, from tire sensors in your vehicles to your appliances and even your fitness tracker. Tiny devices bring intelligence to life at very little cost.

ML inference on the edge is increasingly attractive to increase energy efficiency, privacy, responsiveness, and autonomy of edge devices. Tiny ML breaks the traditional paradigm of energy and compute hungry ML by eliminating networking overhead, allowing for greater overall efficiency and security relative to a cloud-centric approach. The MLPerf Tiny benchmark suite captures a variety of inference use cases that involve “tiny” neural networks, typically 100 kB and below, that process sensor data, such as audio and vision, to provide endpoint intelligence for low-power devices in the smallest form factors. MLPerf Tiny tests these capabilities in a fair and reproducible manner, in addition to offering optional power measurement.

In this round, the Tiny ML v1.1 benchmarks include 10 submissions from academic, industry organizations, and national labs, producing 159 peer-reviewed results. Submitters include: Bosch, cTuning, fpgaConvNet, Kai Jiang, Krai, Nuvoton, Plumerai, Skymizer, STMicroelectronics, and Syntiant. This round includes 41 power measurements, as well. MLCommons congratulates Bosch, cTuning, fpgaConvNet, Kai Jiang, Krai, Nuvoton, and Skymizer on their first submissions to MLPerf Tiny.

“I’m particularly excited to see so many companies embrace the Tiny ML benchmark suite,” said David Kanter, Executive Director of MLCommons. “We had 7 new submitters this round which demonstrates the value and importance of a standard benchmark to enable device makers and researchers to choose the best solution for their use case.”

“With so many new companies adopting the benchmark suite it’s really extended the range of hardware solutions and innovative software frameworks covered. The v1.1 release includes submissions ranging from tiny and inexpensive microcontrollers to larger FPGAs, showing a large variety of design choices,” said Dr. Csaba Kiraly, co-chair of the MLPerf Tiny Working Group. “And the combined effect of software and hardware performance improvements are 1000-fold in some areas compared to our initial reference benchmark results, which shows the pace that innovation is happening in the field.”

View the Results

To view the results for MLPerf Training v3.0 and MLPerf Tiny v1.1, and to find additional information about the benchmarks please visit:
Training v3.0 and Tiny v1.1.

About MLCommons

MLCommons is an open engineering consortium with a mission to make machine learning better for everyone through benchmarks and data. The foundation for MLCommons began with the MLPerf benchmark in 2018, which rapidly scaled as a set of industry metrics to measure machine learning performance and promote transparency of machine learning techniques. In collaboration with its 50+ members – global technology providers, academics, and researchers, MLCommons is focused on collaborative engineering work that builds tools for the entire machine learning industry through benchmarks and metrics, public datasets, and best practices.

For additional information on MLCommons and details on becoming a Member or Affiliate, please visit MLCommmons or contact participation@mlcommons.org.

The post MLPerf Results Show Rapid AI Performance Gains appeared first on MLCommons.

The MLPerf benchmark suites are comprehensive system tests that stress machine learning models including underlying software and hardware and in some cases, optionally measuring energy usage. The open-source and peer-reviewed benchmark suites create a level playing ground for competition, which fosters innovation and benefits society at large through better performance and energy efficiency for AI and ML applications.

The MLPerf Training benchmark suite measures the performance for training machine learning models that are used in commercial applications such as recommending movies, speech-to-text, autonomous vehicles, and medical imaging. MLPerf Training v2.1 includes nearly 200 results from 18 different submitters spanning all the way from small workstations up to large scale data center systems with thousands of processors.

The MLPerf HPC benchmark suite is targeted at supercomputers and measures the time it takes to train machine learning models for scientific applications and also incorporates an optional throughput metric for large systems that commonly support multiple users. The scientific workloads include weather modeling, cosmological simulation, and predicting chemical reactions based on quantum mechanics. MLPerf HPC 2.0 includes over 20 results from 5 organizations with time-to-train and throughput for all models and submissions from some of the world’s largest supercomputers.

The MLPerf Tiny benchmark suite is intended for the lowest power devices and smallest form factors, such as deeply embedded, intelligent sensing, and internet-of-things applications. It measures inference performance – how quickly a trained neural network can process new data and includes an optional energy measurement component. MLPerf Tiny 1.0 encompasses submissions from 8 different organizations including 59 performance results with 39 energy measurements or just over 66% – an all-time record.

“We are pleased to see the growth in the machine learning community and especially excited to see the first submissions from xFusion for MLPerf Training, Dell in MLPerf HPC and GreenWaves Technologies, OctoML, and Qualcomm in MLPerf Tiny,” said MLCommons Executive Director David Kanter. “The increasing adoption of energy measurement is particularly exciting, as a demonstration of the industry’s outstanding commitment to efficiency.”

To view the results and find additional information about the benchmarks please visit: https://mlcommons.org/en/training-normal-21/, https://mlcommons.org/en/training-hpc-20/, and https://www.mlcommons.org/en/inference-tiny-10/

About MLCommons

MLCommons is an open engineering consortium with a mission to benefit society by accelerating innovation in machine learning. The foundation for MLCommons began with the MLPerf benchmark in 2018, which rapidly scaled as a set of industry metrics to measure machine learning performance and promote transparency of machine learning techniques. In collaboration with its 50+ founding partners – global technology providers, academics and researchers, MLCommons is focused on collaborative engineering work that builds tools for the entire machine learning industry through benchmarks and metrics, public datasets and best practices.

For additional information on MLCommons and details on becoming a Member or Affiliate of the organization, please visit http://mlcommons.org/ and contact participation@mlcommons.org.

Press Contact:
David Kanter
press@mlcommons.org

The post Latest MLPerf Results Display Gains for All appeared first on MLCommons.

The MLPerf benchmarks are full system tests that stress machine learning models, software, and hardware and optionally measure power usage. The open-source and peer-reviewed benchmark suites provide a level playing field for competition that drives innovation, performance, and energy-efficiency for the entire industry.

“This was an outstanding effort by the ML community with so many new participants and the tremendous increase in the number and diversity of submissions.” said David Kanter, Executive Director of MLCommons. “I’m especially excited to see greater adoption of power and energy measurements, highlighting the industry’s focus on efficient AI.”

The MLPerf Inference benchmarks primarily focus on datacenter and edge systems and submitters include Alibaba, ASUSTeK, Azure, Deci.ai, Dell, Fujitsu, FuriosaAI, Gigabyte, H3C, Inspur, Intel, Krai, Lenovo, Nettrix, Neuchips, NVIDIA, Qualcomm Technologies, Inc., Supermicro, and ZhejiangLab. This round set new records with over 3,900 performance results and 2,200 power measurements, respectively 2X and 6X more than the prior round, demonstrating the momentum of the community.

The MLPerf Mobile benchmark suite targets smartphones, tablets, notebooks, and other client systems with the latest submissions highlighting an average 2X performance gain over the previous round. MLPerf Mobile v2.0 includes a new image segmentation model, MOSAIC, that was developed by Google Research with feedback from MLCommons. The MLPerf Mobile application and the corresponding source code, which incorporates the latest updates and submitting vendors’ backends, are expected to be available in the second quarter of 2022.

The MLPerf Tiny benchmark suite is intended for the lowest power devices and smallest form factors, such as deeply embedded, intelligent sensing, and internet-of-things applications. The second round of MLPerf Tiny results showed tremendous growth in collaboration with submissions from Alibaba, Andes, hls4ml-FINN team, Plumerai, Renesas, Silicon Labs, STMicroelectronics, and Syntiant. Collectively, these organizations submitted 19 different systems with 3X more results than the first round and over half the results incorporating energy measurements, an impressive achievement for the first benchmarking round with energy measurement.

MLCommons would like to congratulate first time MLPerf Inference submitters ASUSTeK, Azure, H3C, and ZhejiangLab and also Gigabyte and Fujitsu for their first power measurements along with first time MLPerf Tiny submitters Alibaba, Andes, Plumerai, Renesas, Silicon Labs, and STMicroelectronics and also the hls4ml-FINN team and Syntiant on their first energy measurements.

To view the results and find additional information about the benchmarks please visit https://mlcommons.org/en/inference-datacenter-20/,
https://mlcommons.org/en/inference-edge-20/,
https://mlcommons.org/en/inference-mobile-20/, and https://www.mlcommons.org/en/inference-tiny-07/

About MLCommons

MLCommons is an open engineering consortium with a mission to benefit society by accelerating innovation in machine learning. The foundation for MLCommons began with the MLPerf benchmark in 2018, which rapidly scaled as a set of industry metrics to measure machine learning performance and promote transparency of machine learning techniques. In collaboration with its 50+ founding partners – global technology providers, academics and researchers, MLCommons is focused on collaborative engineering work that builds tools for the entire machine learning industry through benchmarks and metrics, public datasets and best practices.

For additional information on MLCommons and details on becoming a Member or Affiliate of the organization, please visit http://mlcommons.org/ and contact participation@mlcommons.org.

Press Contact:
David Kanter
press@mlcommons.org

The post MLPerf Results Show Advances in Machine Learning Inference Performance and Efficiency appeared first on MLCommons.

Embedded machine learning is a burgeoning field where AI-driven sensor data analytics is performed in real-time, close to where the data resides. The new MLPerf Tiny Inference benchmark suite captures a variety of use cases that involve “tiny” neural networks, typically 100 kB and below, that process sensor data such as audio and vision to provide endpoint intelligence.

The first v0.5 round included five submissions from academic, industry organizations, and national labs, producing 17 peer-reviewed results. Submissions this round included software and hardware innovations from Latent AI, Syntiant, PengCheng Labs, Columbia, UCSD, CERN, and Fermilab. To view the results, please visit https://www.mlcommons.org/en/inference-tiny-05/.

MLPerf Tiny Inference: A New Measurement to Advance Intelligence in Everyday Devices

As a new benchmark, MLPerf Tiny Inference enables reporting and comparison of embedded ML devices, systems, and software. Developed in partnership with EEMBC, the Embedded Microprocessor Benchmark Consortium, the benchmark consists of four machine learning tasks that encompass the use of microphone and camera sensors with embedded devices:

• Keyword Spotting (KWS), which uses a neural network that detects keywords from a spectrogram;
• Visual Wake Words (VWW), a binary image classification task for determining the presence of a person in an image;
• Tiny Image Classification (IC), a small image classification benchmark with 10 classes; and
• Anomaly Detection (AD), which uses a neural network to identify abnormalities in machine operating sounds.

KWS has several use cases in endpoint consumer devices, such as earbuds and virtual assistants. VWW has application use cases, for instance, with in-home security monitoring. IC has myriad use cases for smart video recognition applications. AD has several applications in industrial manufacturing for tasks such as predictive maintenance, asset tracking and monitoring.

“To understand progress and advance innovation, particularly in edge computing, the ML industry needs benchmarks,” said Peter Torelli, President of EEMBC. “Creating new metrics and measurement across neural networks and a variety of form factors is challenging, and we’re thrilled to partner with MLCommons to make MLPerf Tiny a reality.”

“The goal of MLPerf is to measure performance for machine learning across the full spectrum of systems – from microwatts to megawatts,” said Professor Vijay Janapa Reddi of Harvard University and MLPerf Tiny Inference working group chair. “This new benchmark will bring intelligence to devices like wearables, thermostats, and cameras, and further MLCommons’ mission to accelerate machine learning innovation to benefit everyone.”

“Tiny machine learning is a fast-growing field and will help to infuse ‘intelligence’ in the small everyday items that surround us,” said MLPerf Tiny Inference working group chair, Colby Banbury, of Harvard University. “By bringing MLPerf benchmarks to these devices, we can help to measure performance and drive efficiency improvements over time.”

MLPerf Tiny v0.5 marks a major milestone in MLCommons’ line-up of MLPerf inference benchmark suites. With the addition of MLPerf Tiny, MLCommons covers the full range of machine learning inference benchmarks, ranging from cloud and datacenter benchmarks that consume kiloWatts of power down to tiny IoT devices that consume only a few milliWatts of power, and everything in between. MLPerf Tiny benchmarks will stimulate tinyML innovation in the academic and research communities and push the state-of-the-art forward in embedded machine learning.

The MLPerf Tiny v0.5 inference benchmarks were created thanks to the contributions and leadership of our working members over the last 18 months, including representatives from: Harvard University, EEMBC, CERN, Columbia, Digital Catapult, Fermilab, Google, Infineon, Latent AI, ON Semiconductor, Peng Cheng Laboratories, Qualcomm, Renesas, SambaNova Systems, Silicon Labs, STMicroelectronics, Synopsys, Syntiant, UCSD, and VoiceMed.

The MLPerf Tiny working group recently submitted a paper to the NeurIPS benchmarks and datasets track that provides in-depth information about the design and implementation of the benchmark suite (https://openreview.net/pdf?id=8RxxwAut1BI). Additional information about the MLPerf Tiny Inference benchmarks is available at the github repository.

About MLCommons

MLCommons is an open engineering consortium with a mission to accelerate machine learning innovation, raise all boats and increase its positive impact on society. The foundation for MLCommons began with the MLPerf benchmark in 2018, which rapidly scaled as a set of industry metrics to measure machine learning performance and promote transparency of machine learning techniques. In collaboration with its 50+ founding member partners – global technology providers, academics and researchers, MLCommons is focused on collaborative engineering work that builds tools for the entire machine learning industry through benchmarks and metrics, public datasets and best practices.

For additional information on MLCommons and details on becoming a member of the organization, please visit http://mlcommons.org/ or contact membership@mlcommons.org.

Press Contact:
mlcommons@strangebrewstrategies.com

The post MLPerf Tiny Inference Benchmark appeared first on MLCommons.