MLPerf Mobile v4.0

The latest revision of the MLPerf Mobile app features several notable improvements.

First, MLPerf Mobile v4.0 includes a new test based on MobileNetV4, a state-of-the-art image classification model from Google Research. The MobileNetV4-Conv-L model boasts an impressive 83% accuracy with the ImageNet dataset, versus 76% accuracy for the prior standard, MobileNetEdgeTPU.

MobileNetV4-Conv-L is designed to perform well across a range of mobile processor types, from CPUs and GPUs to neural accelerators. The MLPerf Mobile working group worked closely with the MobileNetV4 team in order to ensure optimized performance. This combination of an improved model architecture and collaborative optimization has proven quite potent. Although MobileNetV4-Conv-L executes six times the number of mathematical operations of its predecessor, MobileNetEdgeTPU, benchmark execution times have only increased by a factor of roughly 4.6. More details about the model can be found in the MobileNetV4 paper.

MLPerf Mobile v4.0 incorporates support for accelerated machine learning (ML) inference across a range of popular mobile systems on a chip (SoCs) and the devices based on them. New in this release is support for independent hardware vendor (IHV)-provided hardware acceleration paths on the MediaTek Dimensity 9300 and 9300+ SoCs, Qualcomm Snapdragon 7/8/8s Gen 3 chips, and Samsung Exynos 2400 SoCs. These new additions join a number of already-supported SoCs. Furthermore, MLPerf Mobile v.4.0 now runs on even more Android-based devices via its TensorFlow Lite fallback path.

In addition to expanded device support, MLPerf Mobile v4.0 features an improved user experience courtesy of a refreshed and friendlier user interface and new configuration options that offer more control over exactly how the benchmarks run.

“Today, we celebrate four years and eight submissions since the inception of the MLPerf Mobile benchmark,” said Mostafa El-Khamy, co-chair of the MLPerf Mobile working group. “The MLPerf Mobile benchmark suite is continuously evolving. Each of the vision tasks–classification, object detection, and segmentation–has been updated since the first version of the benchmark application, and a new super-resolution task was added last year. Looking ahead, the MLPerf Mobile group is working to add generative AI tasks to a future version of the benchmark suite.”

The MLPerf Mobile app is available for download on GitHub for device and chip manufacturers and others interested in using the benchmark. Please see the release notes for a complete list of supported devices, download links, and a full accounting of the updates and new features in v4.0.

MLPerf Mobile v4.0 results

MLCommons has published a new set of benchmark results from Qualcomm Technologies, Inc. and Samsung based on the MLPerf Mobile v4.0 application. To view the results, visit the MLPerf Mobile results page.

We encourage additional participation to continue to help shape the MLPerf Mobile benchmark suite. To contribute, please join the MLPerf Mobile working group.

About MLCommons

MLCommons is the world leader in building benchmarks for AI. It is an open engineering consortium with a mission to make AI better for everyone through benchmarks and data. The foundation for MLCommons began with the MLPerf benchmarks in 2018, which rapidly scaled as a set of industry metrics to measure machine learning performance and promote transparency of machine learning techniques. In collaboration with its 125+ members, global technology providers, academics, and researchers, MLCommons is focused on collaborative engineering work that builds tools for the entire machine learning industry through benchmarks and metrics, public datasets, and best practices.

For additional information on MLCommons and details on becoming a member or affiliate, please visit MLCommons.org or email participation@mlcommons.org.

The post MLPerf Mobile v4.0 application adds new benchmark, expands hardware support appeared first on MLCommons.

Inference is the critical operational step in machine learning, where a trained model is deployed for actual use, bringing intelligence into a vast array of applications and systems. Machine learning inference is behind everything from the latest generative AI chatbots to safety features in vehicles such as automatic lane-keeping, and speech-to-text interfaces. Improving performance and power efficiency will lead the way for deploying more capable AI systems that benefit society.

The MLPerf benchmark suites are comprehensive system tests that stress machine learning models including the underlying software and hardware and in some cases, optionally measuring power efficiency. The open-source and peer-reviewed benchmark suites create a level playing ground for competition, which fosters innovation and benefits society at large through better performance and power efficiency for AI and ML applications.

The MLPerf Inference benchmarks primarily focus on datacenter and edge systems. This round featured even greater participation across the community with a record-breaking 25 submitting organizations, over 6,700 performance results, and more than 2,400 performance and power efficiency measurements. The submitters include Alibaba, ASUSTeK, Azure, cTuning, Deci.ai, Dell, Gigabyte, H3C, HPE, Inspur, Intel, Krai, Lenovo, Moffett, Nettrix, NEUCHIPS, Neural Magic, NVIDIA, Qualcomm Technologies, Inc., Quanta Cloud Technology, rebellions, SiMa, Supermicro, VMware, and xFusion, with nearly half of the submitters also measuring power efficiency.

MLCommons congratulates our many first time MLPerf Inference submitters on their outstanding results and accomplishments. cTuning, Quanta Cloud Technology, rebellions, SiMa, and xFusion all debuted their first performance results. cTuning, NEUCHIPS, and SiMa also weighed in with their first power efficiency measurements. Lastly, HPE, NVIDIA, and Qualcomm all submitted their first results for inference over the network.

The MLPerf Mobile benchmark suite is tailored for smartphones, tablets, notebooks, and other client systems. The MLPerf Mobile application for Android and iOS is expected to be available shortly.

To view the results and find additional information about the benchmarks please visit https://mlcommons.org/en/inference-datacenter-30, https://mlcommons.org/en/inference-edge-30 and https://mlcommons.org/en/inference-mobile-30.

About MLCommons

MLCommons is an open engineering consortium with a mission to make machine learning better for everyone through benchmarks and data. The foundation for MLCommons began with the MLPerf benchmark in 2018, which rapidly scaled as a set of industry metrics to measure machine learning performance and promote transparency of machine learning techniques. In collaboration with its 50+ founding partners – global technology providers, academics and researchers, MLCommons is focused on collaborative engineering work that builds tools for the entire machine learning industry through benchmarks and metrics, public datasets and best practices.

For additional information on MLCommons and details on becoming a Member or Affiliate of the organization, please visit http://mlcommons.org and contact participation@mlcommons.org.

Press Contact:
Kelly Berschauer
kelly@mlcommons.org

The post MLPerf Inference Delivers Power Efficiency and Performance Gains appeared first on MLCommons.

The MLPerf benchmarks are full system tests that stress machine learning models, software, and hardware and optionally measure power usage. The open-source and peer-reviewed benchmark suites provide a level playing field for competition that drives innovation, performance, and energy-efficiency for the entire industry.

“This was an outstanding effort by the ML community with so many new participants and the tremendous increase in the number and diversity of submissions.” said David Kanter, Executive Director of MLCommons. “I’m especially excited to see greater adoption of power and energy measurements, highlighting the industry’s focus on efficient AI.”

The MLPerf Inference benchmarks primarily focus on datacenter and edge systems and submitters include Alibaba, ASUSTeK, Azure, Deci.ai, Dell, Fujitsu, FuriosaAI, Gigabyte, H3C, Inspur, Intel, Krai, Lenovo, Nettrix, Neuchips, NVIDIA, Qualcomm Technologies, Inc., Supermicro, and ZhejiangLab. This round set new records with over 3,900 performance results and 2,200 power measurements, respectively 2X and 6X more than the prior round, demonstrating the momentum of the community.

The MLPerf Mobile benchmark suite targets smartphones, tablets, notebooks, and other client systems with the latest submissions highlighting an average 2X performance gain over the previous round. MLPerf Mobile v2.0 includes a new image segmentation model, MOSAIC, that was developed by Google Research with feedback from MLCommons. The MLPerf Mobile application and the corresponding source code, which incorporates the latest updates and submitting vendors’ backends, are expected to be available in the second quarter of 2022.

The MLPerf Tiny benchmark suite is intended for the lowest power devices and smallest form factors, such as deeply embedded, intelligent sensing, and internet-of-things applications. The second round of MLPerf Tiny results showed tremendous growth in collaboration with submissions from Alibaba, Andes, hls4ml-FINN team, Plumerai, Renesas, Silicon Labs, STMicroelectronics, and Syntiant. Collectively, these organizations submitted 19 different systems with 3X more results than the first round and over half the results incorporating energy measurements, an impressive achievement for the first benchmarking round with energy measurement.

MLCommons would like to congratulate first time MLPerf Inference submitters ASUSTeK, Azure, H3C, and ZhejiangLab and also Gigabyte and Fujitsu for their first power measurements along with first time MLPerf Tiny submitters Alibaba, Andes, Plumerai, Renesas, Silicon Labs, and STMicroelectronics and also the hls4ml-FINN team and Syntiant on their first energy measurements.

To view the results and find additional information about the benchmarks please visit https://mlcommons.org/en/inference-datacenter-20/,
https://mlcommons.org/en/inference-edge-20/,
https://mlcommons.org/en/inference-mobile-20/, and https://www.mlcommons.org/en/inference-tiny-07/

About MLCommons

MLCommons is an open engineering consortium with a mission to benefit society by accelerating innovation in machine learning. The foundation for MLCommons began with the MLPerf benchmark in 2018, which rapidly scaled as a set of industry metrics to measure machine learning performance and promote transparency of machine learning techniques. In collaboration with its 50+ founding partners – global technology providers, academics and researchers, MLCommons is focused on collaborative engineering work that builds tools for the entire machine learning industry through benchmarks and metrics, public datasets and best practices.

For additional information on MLCommons and details on becoming a Member or Affiliate of the organization, please visit http://mlcommons.org/ and contact participation@mlcommons.org.

Press Contact:
David Kanter
press@mlcommons.org

The post MLPerf Results Show Advances in Machine Learning Inference Performance and Efficiency appeared first on MLCommons.