“It’s an exciting moment for the community,” said Shriya Rishab, MLPerf Training Working Group co-chair. “We’re seeing strong convergence on a set of best practices for training AI models, but at the same time there is increasing technical diversity in the underlying frameworks and systems that are being used to host and run them.”

MLPerf Training v6.0 adds two new benchmarks, emphasizing sparse computation

The MLPerf Training benchmark suite comprises full system tests that stress models, software, and hardware for a range of machine learning (ML) applications. The open-source and peer-reviewed benchmark suite provides a level playing field for competition, driving innovation, performance, and energy efficiency across the industry. The suite’s benchmark collection is curated by a panel of experts from the AI community.

Version 6.0 adds two new benchmarks: DeepSeek V3 and GPT-OSS 20B, both highlighting the industry-wide shift to sparse computation as exemplified by a Mixture-of-Experts (MoE) architecture. Mixture-of-Experts is a model architecture that uses a smart “router” to send different tokens to specialized sub-networks (“experts”). This enables using a high-parameter-count model that is very efficient because training and inference only activate a fraction of the experts for any given token, reducing the computational cost. 

DeepSeek V3 is a large-scale pretraining model, utilizing an MoE architecture. It uses 671 billion total parameters, of which 37 billion are activated per token. It provides a standardized platform for evaluating the training efficiency of a leading open-weights MoE model at production scale.

GPT-OSS 20B, also an MoE model, uses a much smaller footprint: 21 billion total parameters, of which 3.6 billion are activated per token. This allows organizations to evaluate the complex routing logic and sparse computation patterns common to MoE architecture on hardware configurations as small as a single 8-GPU node.

“Sparse computation is a dominant trend in AI right now,” said Rishab. “Over the past two years, all of the major new generative AI models have utilized a sparse computation architecture, frequently MoE. We have introduced our new DeepSeek V3 benchmark to test large-scale sparse computation training systems, and in fact it is now the largest benchmark in our suite with 671 billion parameters. It also exercises the performance of critical innovations that are now standard in the industry, including Multi-head Latent Attention (MLA) and auxiliary-loss-free load balancing.

“On the opposite end of the spectrum, we’ve introduced the GPT-OSS 20B benchmark as an entry point for organizations that may not have the resources to train the largest-scale models, but want to build advanced capabilities. We’ve carefully designed the benchmark for this scenario, including training from randomized weights to avoid the overhead of multi-gigabyte checkpoint downloads; using the same dataset as existing benchmarks in the suite such as Llama 3.1 8B; and choosing a representative sliver of end-to-end training to reduce the cost of generating benchmark results without compromising on the quality of the benchmark.

“Both of these new benchmarks saw quick uptake, drawing many results. Stakeholders clearly see the importance of performance benchmarking for MoE architectures.”

Increasing diversity of submissions highlights new paths to AI training

Version 6.0 set new records for diversity of the systems submitted. Participants in this round of the benchmark submitted 95 unique systems, utilizing thirteen different hardware accelerators, 19 different host processors and a couple of different software frameworks. 60% of the systems were multi-node.

Notably, there are more than double the number of cloud systems submitted compared to the version 5.1 results six months ago, reflecting the emerging market for hosting AI training in the cloud.

“There are more ways of getting your AI training than ever before,” said Pavan Yalamanchili, MLPerf Working Group co-chair. “Several companies now offer training systems in the cloud, complementing the on-premises systems that continue to be built out at a furious pace. And we are excited to see so many competitive submissions from a variety of on-premises and cloud providers.”

At the same time, the submissions illustrate growing technical diversity, reflecting a robust, rapidly advancing ecosystem. For example, submitters used multiple different FP4-precision recipes, reflecting the current diversity and exploration across the industry. 

“The diversity of FP4 implementations we see in the submissions is not surprising,” said Yalamanchili. “Some implementations are more flexible than others, which allow them to be used in unique training scenarios. But here is where MLPerf’s benchmarking delivers critical insight and value: it allows stakeholders to understand which implementations deliver the best performance for their specific needs. In particular, because MLPerf benchmarks require submissions to meet an accuracy threshold, we shine a spotlight on the differences in performance that these kinds of hardware and implementation design choices can lead to.”

Record industry participation points to broad ecosystem, driven by generative AI

The MLPerf Training v6.0 round includes performance results from 24 submitting organizations: AMD, ASUSTeK, Azure, Cisco, CoreWeave, Dell, Fujitsu, GigaComputing, Google, HPE, Inventec, Krai, Lambda, MITAC, Nebius, Netweb Technologies India LTD, NVIDIA, Oracle, Quanta Cloud Technologies, SCITIX, Supermicro, tinycorp, TTA and Vultr.  “We would especially like to welcome first-time MLPerf Training submitters, Inventec, Netweb Technologies India LTD, TTA and Vultr ” said David Kanter, Head of MLPerf at MLCommons.

Robust participation by a broad set of industry stakeholders strengthens the AI ecosystem as a whole and helps to ensure that the benchmark is serving the community’s needs. We invite submitters and other stakeholders to join the MLPerf Training working group and help us continue to evolve the benchmark.

View the results

Please visit the Training benchmark page to view the full results for MLPerf Training v6.0 and find additional information about the benchmarks. To learn about each submitter’s results, read the supplemental.

About ML Commons

MLCommons is the world’s leader in AI benchmarking. An open engineering consortium supported by over 125 members and affiliates, MLCommons has a proven record of bringing together academia, industry, and civil society to measure and improve AI. The foundation for MLCommons began with the MLPerf benchmarks in 2018, which rapidly scaled as a set of industry metrics to measure machine learning performance and promote transparency of machine learning techniques. Since then, MLCommons has continued using collective engineering to build the benchmarks and metrics required for better AI – ultimately helping to evaluate and improve AI technologies’ accuracy, safety, speed, and efficiency.

For additional information on MLCommons and details on becoming a member, please visit MLCommons.org or email participation@mlcommons.org.

Press Inquiries: contact press@mlcommons.org

The post MLCommons Releases MLPerf Training v6.0 Results appeared first on MLCommons.

Pretraining large language models (LLMs) requires massive computational resources, and the MLPerf Training benchmark suite has reflected that reality. Pretraining tests like Llama 3.1 405B and Llama 3.1 8B focus on dense models that can require substantial multi-node infrastructure. This can create a barrier to entry for many organizations looking to participate in benchmarking. 

To address this, the MLPerf Training Working Group—led by a task force from AMD, NVIDIA, and NIT University—is introducing a new pretraining benchmark: GPT-OSS 20B. A modern, Mixture-of-Experts (MoE) alternative, GPT-OSS 20B allows researchers and organizations to evaluate the complex routing logic and sparse computation patterns common to MoE architectures—on hardware configurations as small as a single 8-GPU node.

Model Selection and Architecture

The task force identified GPT-OSS 20B as the ideal small-scale candidate with a modern, sparse architecture for three reasons:

• Sparse Efficiency: The model features 21B total parameters, but utilizes an MoE design that activates only 3.6B parameters per token. This allows it to maintain a massive knowledge base with a computational footprint similar to less dense models.
• From-Scratch Training: To keep the benchmark simple and avoid the overhead of multi-gigabyte checkpoint downloads, GPT-OSS 20B is trained from randomized weights. This makes it a pure test of a system’s ability to optimize a sparse model from its initial state.
• Reference Implementation: The reference code is built on AMD’s Primus framework, a versatile new training library supporting both AMD and NVIDIA backends. Primary validation was conducted on AMD Instinct MI355X and NVIDIA B200 systems.

Dataset and Tokenization

GPT-OSS 20B leverages the C4 (Colossal Cleaned Common Crawl) dataset, specifically using the same pre-tokenized subset and Llama-3 compatible tokenizer as the Llama 3.1 8B benchmark. This makes it easy for submitters who already have the dataset setup for other MLPerf Training benchmarks to run GPT-OSS 20B as well. The training data consists of approximately 80 GB of pre-shuffled C4 shards hosted on MLCommons storage. To maintain stability, the benchmark evaluates against the first 1,024 samples of the validation set every 12,288 samples (768 iterations at GBS=16).

The Challenge of Statistical Variance (CV)

A primary objective in benchmarking is ensuring fairness. In large-scale training, this is measured by the Coefficient of Variation (CV), or the ratio of the standard deviation to the mean ($CV = \frac{\sigma}{\mu} \times 100\%$).

Why High CV Undermines Fairness

When a benchmark has a high CV, it suffers from “statistical noise.” If one run takes 170k samples and another takes 250k samples due to random chance, the results no longer reflect hardware or software superiority. For a benchmark to be a trusted industry standard, a user must be able to reproduce the results. High variance makes it impossible to distinguish between an engineering breakthrough and luck.

Engineering Stability: The Path to 5% CV

One common approach to reducing CV is starting from a pretrained checkpoint that has already passed through the unstable early stages of training. The task force wanted to keep the benchmark simple, so requiring submitters to download a multi-gigabyte checkpoint would add overhead and complexity. Instead, the task force implemented three critical technical interventions to reduce CV from approximately 15% to less than 5%.

1. Eliminating Validation Noise

Early tests showed massive, non-representative spikes in evaluation loss.

• The Discovery: The validation dataset was being shuffled at every evaluation interval. In a sparse MoE model, where routing is highly sensitive to input distribution, this shuffling introduced artificial “jitter.”
• The Fix: The benchmark now mandates evaluation on a static, unshuffled set of the first 1024 samples of the C4 validation set. This ensures the test remains identical across every step of every run.

2. Stabilizing the Optimizer 

While many modern models use an Adam epsilon ($\epsilon$) of $10^{-8}$, the task force found this caused excessive divergence in 20B-scale MoE training from scratch. By aligning to $\epsilon = 10^{-5}$ (the standard used in Llama 3.1 8B), the team provided the numerical stability necessary to prevent “unlucky” gradient updates from derailing the sparse experts.

3. Standardizing Initialization

Technical Configuration and Quality Metrics

To ensure all participants start with the same “statistical energy,” the task force strictly defined the weight initialization standard (init_method_std = 0.008). This prevents variance born from different starting points in the high-dimensional loss landscape.

The target accuracy for the benchmark is a validation loss (log perplexity) of 3.34. This target was chosen based on extensive sweeps on AMD MI355X and NVIDIA B200 hardware, representing a point of stable convergence that balances thoroughness with a reasonable runtime (~6.5 hours to convergence) using BFloat16 precision.

To optimize for their specific hardware, submitters are permitted to tune three hyperparameters: global batch size, learning rate, and learning rate warmup. 

Conclusion

GPT-OSS 20B brings MoE pretraining into the MLPerf Training benchmark suite. By identifying and neutralizing the sources of training variance—specifically validation shuffling, optimizer instability, and initialization inconsistency—the task force has delivered a stable, high-fidelity benchmark that remains accessible to all submitters. This ensures that MLPerf Training v6.0 scores reflect genuine hardware and software efficiency.

GPT-OSS 20B provides the community with a standardized way to evaluate sparse pretraining performance alongside the suite’s existing dense workloads. The reference implementation is available on the MLCommons GitHub repository.

For additional information on MLCommons and details on becoming a member, please visit MLCommons.org or email participation@mlcommons.org.

The post GPT-OSS 20B: A Sparse MoE Pretraining Benchmark for MLPerf Training v6.0 appeared first on MLCommons.

As Large Language Model (LLM) development increasingly adopts sparse computation, the benchmarks used to evaluate training performance need to keep pace. MLPerf Training v6.0 adds a large-scale pretraining benchmark built on DeepSeek-V3, a Mixture-of-Experts (MoE) architecture with 671B total parameters, of which 37B are activated per token. 

This benchmark captures the performance of critical innovations now standard in the industry, including Multi-head Latent Attention (MLA) and auxiliary-loss–free load balancing.

Technical Architecture & Implementation

DeepSeek-V3 introduces specific computational patterns that differentiate it from the dense models (e.g., Llama 3.1) currently in the suite:

• Multi-head Latent Attention (MLA): Unlike standard Multi-Head Attention (MHA), MLA uses low-rank joint compression for Key-Value (KV) caches to reduce memory bandwidth bottlenecks during training and inference.
• Fine-Grained Expert Segmentation: Each expert feed-forward network (FFN) is segmented into $m$ smaller experts. While a typical MoE might use a top-2 routing over 16 experts, DeepSeek-V3 expands this to 160 routed experts plus shared experts to capture common knowledge.
• Multi-Token Prediction (MTP): The model is trained with a 2-token prediction objective. This requires a shared trunk and two dedicated output heads, increasing the compute-to-memory ratio during the backward pass.
• Auxiliary-Loss-Free Load Balancing: To prevent routing collapse without the performance degradation of heavy auxiliary losses, a bias term is dynamically adjusted for each expert based on real-time load.

Benchmark Definition & Reference Setup

The task is defined as LLM Pretraining using a Mixture-of-Experts objective.

Dataset and Tokenization

• Dataset: C4 (Colossal Clean Crawled Corpus)
• Tokenizer: Llama-3 compatible tokenizer (128k vocabulary)
• Sequence Length: 4,096 tokens

Convergence and Checkpointing

The task force identified that MoE models spend early training time in a state of token imbalance. Since the benchmark captures only a small slice of the full end-to-end training, this token imbalance state lasted for approximately 50% of the benchmarking time, which is not representative of steady-state MoE training. To ensure the benchmark measures steady-state hardware efficiency, the task force adopted a warm-start approach.

Since the benchmark uses a Llama 8B tokenizer instead of the original DeepSeek tokenizer, initializing from a Hugging Face checkpoint leads to significant token imbalance. To address this, the task force fine-tuned the checkpoint for 50 steps, bringing the token-per-expert distribution close to that obtained with the original DeepSeek tokenizer (Fig 1). The resulting checkpoint is in HuggingFace format and is hosted by MLCommons. This methodology ensures more than 98% of the benchmark run occurs in a balanced expert state, reflecting long-term training dynamics.

(Fig 1)

Global Batch Size (GBS) Selection

This benchmark requires a GBS of 15,360 or greater.While internal testing showed that the model maintains computational efficiency at lower batch sizes (e.g., 512, 1k, or 2k), the task force mandated a higher floor for three reasons:

• Representativeness: The original DeepSeek-V3 pretraining used a batch size scheduling strategy that peaked at 15,360. To remain representative of the real-world, large-scale pretraining described in the paper, the benchmark targets the 15k-18k range.
• Fairness in Benchmarking: Setting a minimum GBS of 15k ensures a fair and representative playing field for all submitters, preventing “hero runs” on tiny batch sizes that don’t reflect production-scale MoE training.
• Convergence Scaling: The task force established a square-root scaling rule for the learning rate to maintain stability across these larger batch sizes:
  $LR(GBS) = 2.4 \times 10^{-5} \times \sqrt{\frac{GBS}{16384}}$

Engineering Challenges and Validations

Development of the reference implementation (using NVIDIA NeMo Megatron-bridge) highlighted several critical requirements for convergence:

• Expert Parallelism (EP): For a single layer, routed experts are uniformly deployed across 64 GPUs (8 nodes). Node-limited routing is enforced where each token is sent to a maximum of M=4 nodes.
• Memory Management: Expert capacity must not be restricted to avoid out-of-memory errors (OOMs) caused by token imbalance during the initial steps after checkpoint loading, ensuring the benchmark remains representative of real-world training conditions. 
• Target Metric: The benchmark targets a cross-entropy validation loss of 3.6 with a Coefficient of Variation (CV) of 1.5% (Fig 2).

(Fig 2)

Conclusion

This benchmark provides a standardized platform for evaluating the training efficiency of a leading open-weights MoE model at production scale. By defining clear requirements for convergence, batch size, and expert parallelism, the DeepSeek-V3 benchmark ensures that MLPerf Training continues to reflect the contemporary state of AI infrastructure.

The reference implementation is available on the MLCommons GitHub repository, and the task force welcomes submissions and feedback from the community. For additional information on MLCommons and details on becoming a member, please visit MLCommons.org.

The post DeepSeek-V3: A Large-Scale MoE Pretraining Benchmark for MLPerf Training v6.0 appeared first on MLCommons.

The scale of this challenge is significant. The Llama 3.1 405B benchmark added in MLPerf Training v5.0, for example, required a minimum of 256 GPUs per submission. This mirrors leading AI development but creates barriers for organizations looking to benchmark their systems without massive GPU clusters. 

Since v0.7 in 2020, MLPerf Training has used BERT as one of the pretraining benchmarks. Its versatility allows it to run on a single node and scale to hundreds of GPUs. However, the BERT model, introduced by Google in 2018, is now significantly outdated and cannot address the requirements of modern LLMs. Finding a suitable replacement required balancing accessibility with architectural relevance.

Model Selection

The MLPerf Training Working Group spun out a task force to find the best alternative. One of the primary requirements was a model small enough to run on a single node for easy execution.

The task force evaluated several candidates, and Llama 3.1 8B quickly became the front-runner. With Llama3.1 405B already added to the Training suite in v5.0, the 8B variant allowed submitters to leverage the existing implementation while working at a much smaller scale. Its architecture also makes it a good proxy for evaluating the performance of even larger models.

Training Data 

Like the Llama3.1 405B benchmark, the 8B references the C4 (Colossal Cleaned Common Crawl) dataset. However, the 8B model’s smaller parameter count and shorter training time require only a subset of the full dataset. This reduces effort involved for submitters and keeps benchmark run times reasonable. 

The benchmark uses the default C4 dataset split between training and validation. For training, the c4-train.<x>-of-01024.json.gz files (768 <= x <= 1023) are used, where the last 256 of 1,024 shards are randomly shuffled. For evaluation, a customized subset c4-validation-91205-samples.en.json.gz is used, which contains the first 91,205 samples from the unshuffled C4 validation dataset. This is the smallest number of samples needed to yield over 47,185,920 tokens. During each evaluation run, the first 1,024 sequences (roughly 8.4 million tokens) from this set are tested. The training data is shuffled to introduce variability, while the validation data remains unshuffled to ensure consistent assessment across runs.

To simplify setup, MLPerf hosts this preprocessed dataset subset. Submitters can download it by following these instructions.  

Implementation Details

The reference code is implemented in the NVIDIA NeMo Framework, an open-source, scalable AI framework built for large language models. By offering modular components and pre-trained model recipes, NeMo enables developers to build performant models tailored to specific use cases. The reference code functionality is tested on NVIDIA B200 GPUs and AMD MI325X. 

Unlike many MLPerf Training benchmarks, including Llama 3.1 405B, this one starts from randomized weights rather than a checkpoint. This simplifies running and porting across different systems. 

The benchmark also uses the native Llama 3.1 8B tokenizer, whereas the 405B benchmark deliberately uses a different tokenizer (32k vocab size Mixtral 8x22B) to force the model checkpoint to adapt to a new token distribution. Without a checkpoint, the 8B benchmark can use its native tokenizer, making setup even easier. The tokenizer can be downloaded directly from the MLPerf website following these instructions. 

To optimize convergence to the target accuracy criterion, submitters can adjust three hyperparameters: batch size, learning rate, and number of warmup samples. However, submitters must match the behaviors defined by RCPs (Reference Convergence Points). The full rules governing RCPs are available here. 

The task force continued using the validation loss perplexity as the convergence criterion, maintaining consistency with the 405B benchmark. Extensive experiments on reference hardware established a target value of 3.3. 

Conclusion

This new pretraining benchmark based on Meta’s Llama 3.1 8B model brings modern LLM pretraining evaluation within reach of more organizations. It was intentionally designed to be easy to set up and run on small to moderately sized computational resources. By requiring only a single node, using a subset of the C4 dataset, and starting from random weights, it lowers the barrier to entry while maintaining relevance to current AI development practices. Organizations with experience running the Llama 3.1 405B benchmark can leverage that existing expertise, while those new to MLPerf Training gain an accessible entry point.

With this addition to MLPerf Training v5.1, the suite now offers LLM pretraining benchmarks from single-node systems to massive multi-cluster workloads, enabling standardized evaluation across different system scales.

For technical specifications and submission guidelines, visit our MLPerf Training benchmark page. Our submission rules and reference code are available on GitHub

About MLCommons

MLCommons is the world’s leader in AI benchmarking. An open engineering consortium supported by over 125 members and affiliates, MLCommons has a proven record of bringing together academia, industry, and civil society to measure and improve AI. The foundation for MLCommons began with the MLPerf benchmarks in 2018, which rapidly scaled as a set of industry metrics to measure machine learning performance and promote transparency of machine learning techniques. Since then, MLCommons has continued using collective engineering to build the benchmarks and metrics required for better AI – ultimately helping to evaluate and improve AI technologies’ accuracy, safety, speed, and efficiency.

For additional information on MLCommons and details on becoming a member, please visit MLCommons.org or email participation@mlcommons.org.

The post MLPerf Training Adds Llama 3.1 8B Benchmark appeared first on MLCommons.

Text-to-image models create images from text prompts. One widely used technique is diffusion, where models learn to iteratively transform random noise into complete images. Since Stable Diffusion (SDv2) was added to the MLCommons MLPerf Training benchmark suite in October 2023, the importance of these models has increased dramatically. 

Model sizes and architectures have also evolved since then, with parameter counts growing from ~800M to 3.5B in Stable Diffusion XL. Diffusion Transformer (DiT) models successfully integrated transformer architecture into the diffusion process. Leading models, including Stable Diffusion 3.5, have adopted this innovation, marking a shift the SDv2 benchmark no longer reflects. 

Recognizing this gap, the MLPerf Training Working Group formed a task force to evaluate a refresh of the text-to-image benchmark. MLPerf Training v5.1 introduces a new text-to-image benchmark based on Black Forest Labs’ Flux.1, a 11.9B-parameter transformer-based model that reflects the current state of generative AI.

Model Selection

The task force evaluated candidates on four primary criteria: performance, architecture, size, and availability.

• Image Quality: SDv2 can no longer compete with more modern models, such as Imagen3, Flux.1 and SD3.5, in image quality and prompt adherence
• Architecture: A representative benchmark must mirror the architectural approaches used in state-of-the-art models and represent real-world scenarios. For text-to-image generation, transformer-based architectures have become dominant, making this a critical requirement. 
• Size: Modern text-to-image models have parameter counts in the billions, while SDv2 has fewer than 900M parameters. These larger parameter counts introduce new memory and computation challenges, such as different parallelization requirements, that smaller models don’t face. 
• Availability: MLPerf Training requires open-source architectures, significantly limiting the possible choices.

Black Forest Lab’s Flux.1 met all four criteria. This transformer-based latent diffusion model has 11.9B parameters and a fully open-source license. It can compete with closed-source models across multiple quality benchmarks, making it a strong representative of current text-to-image generation approaches.

Dataset Selection

MLPerf Training requires publicly available, open-source datasets, a constraint that presents some challenges since even open-sourced models are typically trained on closed, proprietary data. The SDv2 benchmark used a subset of the LAION dataset for training, making it an early option for this new benchmark as well. The task force also evaluated CC12M, another popular open-source dataset for captioned images. For validation, both benchmarks use a subset of COCO-2014, which remains the standard for evaluating text-to-image models. 

The two training datasets differ significantly in composition. The LAION subset contains 6.5M samples drawn from a 400M collection, prioritizing web-scale coverage. CC12M provides nearly 12M samples but emphasizes clean, high-precision captions over raw scale.

To make an evidence-based decision, the task force trained an initial Flux.1 implementation on both datasets for short and long training periods, then evaluated them on COCO-2014 using FID and CLIP as metrics. The number of training steps was chosen as an approximated upper bound of the desired training duration for the benchmark. As shown in the charts below, CC12M consistently outperformed the LAION subset across both training durations, making it the clear choice for the benchmark. 

Further experimentation revealed that, for the chosen benchmarking region (detailed in the following section), a 90% reduction in dataset size had minimal impact on image quality. As such, the final training dataset consists of 1,099,776 samples from CC12M.

Implementation Details

Flux.1 is a latent diffusion model built on DiT blocks, with architectural details and training procedures following the report: Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. The model uses multimodal transformer blocks to concurrently process text and image data. From a modelling perspective, a key architectural difference from SDv2 is the shift to rectified flows, which encourage linear denoising trajectories for more efficient decoding.

The reference code is implemented with torchtitan, an open-source framework for large-scale AI training built on native Pytorch. Its focus on clean, minimal implementations and enabled fast prototyping and development in collaboration with Meta’s torchtitan team. Because it builds directly on native Pytorch, the code is straightforward to read and understand. The implementation has been tested on NVIDIA B200 GPUs.

Since the original Flux.1 training code is not open-source, the reference builds on the torchtitan implementation of latent diffusion. The training procedure works as follows:

1. Given an image-text pair, encode the text using T5-XXL and CLIP-ViT-H encoders, and encode the image using a VAE (Variational Autoencoder).
2. Sample Gaussian noise (φ) matching the shape of the image encoding (i). Select a random point along the line between the image encoding and this noise to obtain a noisy latent. The difference (φ – i) becomes the ground truth.
3. Feed the noisy latent and text encodings to the transformer model, which tries to regress the ground truth using mean squared error loss.

Intuitively, the network learns to identify the noise that should be removed to obtain a clean latent. During inference, this process is applied iteratively, starting from Gaussian noise, producing a latent that the VAE decodes into an image.

As with the previous text-to-image benchmark, the encoders are frozen during training. This allows all encodings for the dataset to be pre-computed once and made available to submitters, eliminating redundant computation during benchmark runs.

Evaluation Approach

A major difference from the SDv2 benchmark is the evaluation metric, a change that significantly streamlines the benchmarking process. 

The previous benchmark used FID and CLIP scores, which currently remain  industry-standard metrics for text-to-image models. However, these metrics require generating complete images and running them through separate models for evaluation, which makes them time-consuming and intensive to compute during a training submission. 

Following the Scaling Rectified Flow Transformers for High-Resolution Image Synthesis research, the task force found that validation loss computed over equally-sampled noise levels correlates highly with FID and CLIP scores. This correlation allows submitters to leverage a much simpler metric that requires only a single forward pass per sample, rather than full image generation. This alternative metric significantly speeds up evaluation while maintaining meaningful quality assessment.

Benchmark Parameters

A full training run of Flux.1 would take prohibitively long for benchmarking purposes, so the task force selected a suitable benchmarking region. Training the model from scratch until a target validation loss of 0.586 provided a good balance between run-to-run variance and room for scaling compute. Using 64 NVIDIA B200 GPUs with torch.compile and bf16 training, the reference implementation completes a run in approximately 95 minutes, a baseline submitters are expected to improve upon significantly. Due to run-to-run variance, each submission requires 10 runs.

Conclusion

The Flux.1 benchmark brings MLPerf Training’s text-to-image evaluation in line with current generative AI practices. With 12x more parameters than SDv2 and a transformer-based architecture, it better reflects the current landscape of text-to-image models. 

This update ensures MLPerf Training continues to provide relevant performance insights as text-to-image generation rapidly evolves, offering submitters a benchmark that reflects the systems and techniques used in production environments today.

Our submission rules and reference code are available on GitHub. 

About MLCommons

MLCommons is the world’s leader in AI benchmarking. An open engineering consortium supported by over 125 members and affiliates, MLCommons has a proven record of bringing together academia, industry, and civil society to measure and improve AI. The foundation for MLCommons began with the MLPerf benchmarks in 2018, which rapidly scaled as a set of industry metrics to measure machine learning performance and promote transparency of machine learning techniques. Since then, MLCommons has continued using collective engineering to build the benchmarks and metrics required for better AI – ultimately helping to evaluate and improve AI technologies’ accuracy, safety, speed, and efficiency.

For additional information on MLCommons and details on becoming a member, please visit MLCommons.org or email participation@mlcommons.org.

The post MLPerf Training Introduces Flux.1 Text-to-Image Benchmark appeared first on MLCommons.

MLPerf Training v5.0 introduces new Llama 3.1 405B benchmark

The MLPerf Training benchmark suite comprises full system tests that stress models, software, and hardware for a range of machine learning (ML) applications. The open-source and peer-reviewed benchmark suite provides a level playing field for competition that drives innovation, performance, and energy efficiency for the entire industry.

Version 5.0 introduces a new large language model pretraining benchmark based on the Llama 3.1 405B generative AI system, which is the largest model to be introduced in the training benchmark suite. It replaces the gpt3-based benchmark included in previous versions of the MLPerf Training benchmark suite. An MLPerf Training task force selected the new benchmark because it is a competitive model representative of the current state-of-the-art LLMs, including recent algorithmic updates and training on more tokens. More information on the new benchmark can be found here. Despite just being introduced, the Llama 3.1 405B benchmark is already receiving more submissions than the gpt3-based predecessor saw in previous rounds – demonstrating the popularity and importance of large-scale training.

Rapid performance improvements for key training scenarios

The MLPerf Training working group regularly adds emerging training workloads to the benchmark suite to ensure that it reflects industry trends. The Training 5.0 benchmark results show notable performance improvements for newer benchmarks, indicating that the industry is prioritizing emerging training workloads over older ones. The Stable Diffusion benchmark saw a 2.28x speed increase for 8-processor systems compared to the 4.1 version six months ago, and the Llama 2.0 70B LoRA benchmark increased its speed 2.10x versus version 4.1; both outpacing historical expectations for computing performance improvements over time as per Moore’s Law. Older benchmarks in the suite saw more modest performance improvements.

On multi-node, 64-processor systems, the RetinaNet benchmark saw a 1.43x speedup compared to the prior v3.1 benchmark round (the most recent to include comparable scale systems), while the Stable Diffusion benchmark had a dramatic 3.68x increase.

“This is the sign of a robust technology innovation cycle and co-design: AI takes advantage of new systems, but the systems are also evolving to support high-priority scenarios,” said Shriya Rishab, MLPerf Training working group co-chair. 

Increasing diversity of processors, increasing scale of systems, broadening ecosystem

Submissions to MLPerf Training 5.0 utilized 12 unique processors, all in the available (production) category. Five of the processors have become publicly available since the last version of the benchmark suite.

• AMD Instinct MI300X 192GB HBM3
• AMD Instinct MI325X 256GB HBM3e
• NVIDIA Blackwell GPU (GB200)
• NVIDIA Blackwell GPU (B200-SXM-180GB)
• TPU-trillium

Submissions also included three new processor families: 

• 5th Generation AMD Epyc Processor (“Turin”)
• Intel Xeon 6 Processor (“Granite Rapids”)
• Neoverse V2 as part of NVIDIA GB200

In addition, the number of multi-node systems submitted increased more than 1.8x when compared to version 4.1.

“The picture is clear: AI workloads are scaling up, systems are scaling up to run them, and hardware innovation continues to boost performance for key scenarios,” said Hiwot Kassa, MLPerf Training working group co-chair. “In-house large scale systems were built by few companies, but the increased proliferation – and competition – in AI-optimized systems is enabling the broader community to scale up their own infrastructure. Most notably, we see an increasing cadre of cloud service providers offering access to large-scale systems, democratizing access to training large models. 

“The industry is not standing still, and neither can we. MLCommons is committed to continuing to evolve our benchmark suite so that we can capture and report on the innovation that is happening in the field of AI.”

Record industry participation

The MLPerf Training v5.0 round includes 201 performance results from 20 submitting organizations: AMD, ASUSTeK, Cisco Systems Inc., CoreWeave, Dell Technologies, GigaComputing, Google Cloud, Hewlett Packard Enterprise, IBM, Krai, Lambda, Lenovo, MangoBoost, Nebius, NVIDIA, Oracle, Quanta Cloud Technology, SCITIX, Supermicro, and TinyCorp.

“We would especially like to welcome first-time MLPerf Training submitters AMD, IBM, MangoBoost, Nebius, and SCITIX,” said David Kanter, Head of MLPerf at MLCommons. ”I would also like to highlight Lenovo’s first set of power benchmark submissions in this round – energy efficiency in AI training systems is an increasingly critical issue in need of accurate measurement.”

MLPerf Training v5.0 set a new high-water mark for the >200 submissions. The vast majority of the individual benchmark tests that carried over from the previous round saw an increase in submissions.

Robust participation by a broad set of industry stakeholders strengthens the AI/ML ecosystem as a whole and helps to ensure that the benchmark is serving the community’s needs. We invite submitters and other stakeholders to join the MLPerf Training working group and help us continue to evolve the benchmark.

View the results

Please visit the Training benchmark page to view the full results for MLPerf Training v5.0 and find additional information about the benchmarks.

About ML Commons

MLCommons is the world’s leader in AI benchmarking. An open engineering consortium supported by over 125 members and affiliates, MLCommons has a proven record of bringing together academia, industry, and civil society to measure and improve AI. The foundation for MLCommons began with the MLPerf benchmarks in 2018, which rapidly scaled as a set of industry metrics to measure machine learning performance and promote transparency of machine learning techniques. Since then, MLCommons has continued using collective engineering to build the benchmarks and metrics required for better AI – ultimately helping to evaluate and improve AI technologies’ accuracy, safety, speed, and efficiency.

For additional information on MLCommons and details on becoming a member, please visit MLCommons.org or email participation@mlcommons.org.

Press Inquiries: contact press@mlcommons.org

The post New MLCommons MLPerf Training v5.0 Benchmark Results Reflect Rapid Growth and Evolution of the Field of AI appeared first on MLCommons.

Pretraining Large Language Models (LLMs) is the initial phase of LLM training where the model learns to understand and generate human-like text by training the model on vast amounts of unstructured textual data. A pre-trained model can then be fine-tuned on domain-specific datasets, which gives the model knowledge of specific tasks. Reasoning approaches have become popular recently, helping models better explain their thinking process. Because pre-training uses large amounts of data, it is generally the most compute-intensive phase of LLM training. 

MLCommons^® added a GPT3 Pretraining benchmark to MLPerf^® Training in 2022 to help academics and industry experts optimize LLM pretraining. It has been a huge success, as we have seen over 3x speedups on the results of this benchmark for large systems (over 10,000 accelerators).

Over this same period, newer state-of-the-art models have demonstrated greater scale and many architectural advancements. For example, Google’s PaLM model, released in 2022, contains 540B parameters, three times as many as GPT3’s 175B parameter count. Meta’s Llama 2 and Llama 3 series, released in 2023, utilize new algorithms such as Root Mean Square Layer Normalization (RMSNorm, Zhang et al., https://arxiv.org/abs/1910.07467), Rotary Position Embedding (RoPE, Su et al., 2024), and Grouped Query Attention (GQA, Ainslie et al., 2023). 

A MLPerf Training working group task force was created to investigate a July 2024 proposal to  replace GPT3 with Llama 3.1 405B. Task force members evaluated the proposal and industry options to recommend the best new pretraining reference. They agreed that Llama 3.1 405B would best represent the current state-of-the-art pretraining and adopted it as a new benchmark. Details on the review and selection process are included in this blog.

Model selection

The most important factor the task force considered when constructing this new pretraining benchmark was the model architecture. As mentioned above, the current GPT3 architecture does not include some of the recent algorithm updates and therefore is not competitive with other state-of-the-art models such as Nemotron-4, GPT4.5 and Claude 3.7 Sonnet. However, Meta’s Llama 3.1 405B model, with an architecture and scale similar to these top-tier models, demonstrates on-par performance with these models across multiple quality benchmarks, thus positioning itself as a competitive model representative of the current state-of-the-art models. 

Choosing a model with high community engagement was also an important consideration, and Llama 3.1 405B is the perfect candidate on that basis. With more than 300 million total downloads of all Llama versions to date, the Llama family of models is very widely adopted. The MLPerf community, for instance, has already included Llama 2 in the LoRA benchmark. Additionally, Meta’s released technical paper details the training process, model architecture, dataset and hyperparameters, which allowed us to easily create a high-quality benchmark based on their work.

One other requirement for the new MLPerf Training benchmark is that the model architecture must be publicly accessible so that all submitters can download it and reproduce the results on their end. This typically requires a permissive license. Thanks to Meta’s support, the current Llama 3.1 405B model is now available to all MLCommons members for benchmarking. 

Dataset selection

To help submitters transition easily from GPT3 to Llama 3.1 405B, we use the same AllenAI C4-en dataset (Colossal Cleaned Common Crawl dataset) in both benchmarks. This dataset contains 365M English-language paragraphs scraped from the internet, cleaned and preprocessed to convenient JSON formats for training the model. 

One key difference is that Llama 3.1 405B uses a context length of 8,192, which is 4x larger than the 2,048-token context of GPT3. The context length represents the amount of text that a model can process at once, thus a larger context length leads to better understanding. 

Benchmark technical details

The reference code is implemented in the NVIDIA NeMo Framework, an open-sourced and scalable AI framework built for large language models. By offering easy-to-use modular components and pre-trained model recipes, NeMo enables developers to efficiently build performant models tailored to specific use cases. The reference code functionality is tested on NVIDIA H100 GPUs. 

To ensure that the new benchmark model setup is easily understandable without complex knobs and hyperparameters scattered around the training script, it starts with NeMo’s integrated Llama 3.1 405B training recipe, which closely followed the original paper’s hyperparameters and introduced only the minimum necessary changes to our script, such as adjusting the optimizer’s learning rates. 

When researching how to construct this benchmark, we first explored the GPT3-style setup. For the GPT3 pretraining benchmark, the task force pretrained from random weights for about 12.6T tokens to generate a stable customized checkpoint. Submitters were required to continue pretraining from this checkpoint until they reached a predetermined quality metric. This methodology was used to ensure that the GPT3 pretraining benchmark showcased a reproducible slice of the actual GPT3 pretraining process.

Applying the same methodology to the Llama 3.1 405B pretraining benchmark, the task force generated a customized checkpoint trained on 32B tokens. But experiment results showed that resuming from this checkpoint had more variation in the convergence curves than we could tolerate. This meant that resuming from a custom-trained checkpoint in the benchmark was not feasible. The interpretation that we were still in very early phases of training after the 32B tokens, meant that the checkpoint generated was still quite unstable.

So the taskforce decided to train from the fully trained and stable HuggingFace checkpoint by giving it some new information, which led to a much more stable convergence trend. The benchmark now begins from the checkpoint and stops when the evaluation log perplexity reaches our defined target.

One noticeable difference between the task force’s implementation and Meta’s implementation is that we replaced the original Tiktokenizer, which used a vocab size of 128k, with the 32k vocab size Mixtral 8x22b tokenizer. The motivation behind this change was that Meta’s checkpoint is already well-trained, so it will not learn anything new on the current C4 dataset. Replacing the tokenizer forces the model checkpoint to adapt to a new, different token distribution, therefore continuing to pretrain on the current dataset. This difference implies that, when we resume from the pre-trained checkpoint, only the first 32,000 rows of the 128,256-row word embedding layer weights will be loaded to the model.

As mentioned before, the Llama 3.1 405B pretraining benchmark is a replacement for the GPT3 pretraining benchmark, and the following components will be unchanged between the two:

• Validation dataset: To reduce the cost of validation, we chose to create a customized subset of the full C4 validation dataset so that the evaluation time would not be significant compared to training time. Specifically, we found out that using only 5,760 sequences is sufficient to measure model convergence, and since only 91,205 validation samples are needed to yield the required 5,760 validation sequences, the first 91,205 unshuffled rows of the full C4 validation dataset were selected as the customized validation dataset for this benchmark.

• Loss and target accuracy metric: As with GPT3, log perplexity is used, computed on the customized validation dataset to evaluate the model’s convergence behavior. This metric is not costly to compute and is widely adopted as a versatile indicator of evaluating LLMs.
• Limit number of submission logs: We recognize that the Llama 3.1 405B pretraining benchmark is very expensive to run, so to ensure fairness while reducing submitters costs, each submitter is required to only run the benchmark 3 times, the same as the GPT3 pretraining benchmark. The median result is then reported.
• Restriction on hyperparameter searching: Because running the benchmark is expensive, we believe it is reasonable to disallow hyperparameter searches and borrowing. Most hyperparameters have been given static values. For a very few others, such as the learning rate, formulas are provided to compute them based on the global batch size.

Conclusion

In this blog post, we have provided an overview of a new LLM pretraining benchmark based on Meta’s Llama 3.1 405B model with twice as many parameters and four times as many training tokens as GPT3. This benchmark is massively more computationally intensive and contains its own new challenges. It is the task force’s belief that this new benchmark will encourage more innovation in the area of LLM pretraining.

About MLCommons

MLCommons is the world’s leader in AI benchmarking. An open engineering consortium supported by over 125 members and affiliates, MLCommons has a proven record of bringing together academia, industry, and civil society to measure and improve AI. The foundation for MLCommons began with the MLPerf benchmarks in 2018, which rapidly scaled as a set of industry metrics to measure machine learning performance and promote transparency of machine learning techniques. Since then, MLCommons has continued using collective engineering to build the benchmarks and metrics required for better AI – ultimately helping to evaluate and improve AI technologies’ accuracy, safety, speed, and efficiency.

For additional information on MLCommons and details on becoming a member, please visit MLCommons.org or email participation@mlcommons.org.

The post MLCommons MLPerf Training Expands with Llama 3.1 405B appeared first on MLCommons.

New generations of hardware accelerators

MLPerf Training v4.1 includes preview category submissions using new hardware accelerators that will be generally available in the next round:

• Google “Trillium” TPUv6 accelerator (preview)
• NVIDIA “Blackwell” B200 accelerator (preview)

 “As AI-targeted hardware rapidly advances, the value of the MLPerf Training benchmark becomes more important as an open, transparent forum for apples-to-apples comparisons,” said Hiwot Kassa, MLPerf Training working group co-chair. “Everyone benefits: vendors know where they stand versus their competitors, and their customers have better information as they procure AI training systems.”

MLPerf Training v4.1

The MLPerf Training benchmark suite comprises full system tests that stress models, software, and hardware for a range of machine learning (ML) applications. The open-source and peer-reviewed benchmark suite provides a level playing field for competition that drives innovation, performance, and energy efficiency for the entire industry.

The latest Training v4.1 results show a substantial shift in submissions for the three benchmarks that represent “generative AI” training workloads: GPT3, Stable Diffusion, and Llama 2 70B LoRA fine-tuning benchmark, with a 46% increase in submissions in total across those three. 

The two newest benchmarks in the MLPerf Training suite, Llama 2 70B LoRA and Graph Neural Network (GNN), both had notably higher submission rates: a 16% increase for Llama 2, and a 55% increase for GNN. They both also saw significant performance improvements in the v4.1 round compared to v4.0 when they were first introduced, with a 1.26X speedup in the best training time for Llama 2 and a 1.23X speedup for GNN.

“The MLCommons Training working group regularly updates the benchmark to keep pace with industry trends, and the trend in submissions we are seeing validates that our open process is giving stakeholders the information they are looking for. In this round we see continued steady, incremental progress in improving AI training performance,” observed Shriya Rishab, MLPerf Training working group co-chair.

Continued Robust Industry Participation

The MLPerf Training v4.1 round includes 155 results from 17 submitting organizations: ASUSTeK, Azure, Cisco, Clemson University Research Computing and Data, Dell, FlexAI, Fujitsu, GigaComputing, Google, Krai, Lambda, Lenovo, NVIDIA, Oracle, Quanta Cloud Technology, Supermicro, and Tiny Corp.

“We would especially like to welcome first-time MLPerf Training submitters FlexAI and Lambda,” said David Kanter, Head of MLPerf at MLCommons. “We are also very excited to see Dell’s first MLPerf Training results that includes power measurement; as AI adoption skyrockets, it is critical to measure both the performance and the energy efficiency of AI training.”

Participation in the benchmarking process by a broad set of industry stakeholders, as well as by academic groups, strengthens the AI/ML ecosystem as a whole and helps to ensure that the benchmark is serving the community’s needs. We invite submitters and other stakeholders to join the MLPerf Training working group and help us continue to evolve the benchmark.

View the results

To view the full results for MLPerf Training v4.1 and find additional information about the benchmarks, please visit the Training benchmark page.

About ML Commons

MLCommons is the world leader in building benchmarks for AI. It is an open engineering consortium with a mission to make AI better for everyone through benchmarks and data. The foundation for MLCommons began with the MLPerf benchmarks in 2018, which rapidly scaled as a set of industry metrics to measure machine learning performance and promote transparency of machine learning techniques. In collaboration with its 125+ members, global technology providers, academics, and researchers, MLCommons is focused on collaborative engineering work that builds tools for the entire AI industry through benchmarks and metrics, public datasets, and measurements for AI risk and reliability.

For additional information on MLCommons and details on becoming a member, please visit MLCommons.org or contact participation@mlcommons.org.

The post New MLPerf Training v4.1 Benchmarks Highlight Industry’s Focus on New Systems and Generative AI Applications appeared first on MLCommons.

MLPerf Training v4.0

The MLPerf Training benchmark suite comprises full system tests that stress machine learning (ML) models, software, and hardware for a broad range of applications. The open-source and peer-reviewed benchmark suite provides a level playing field for competition that drives innovation, performance, and energy efficiency for the entire industry.

MLPerf Training v4.0 includes over 205 performance results from 17 submitting organizations: ASUSTeK, Dell, Fujitsu, Giga Computing, Google, HPE, Intel (Habana Labs), Juniper Networks, Lenovo, NVIDIA, NVIDIA + CoreWeave, Oracle, Quanta Cloud Technology, Red Hat + Supermicro, Supermicro, Sustainable Metal Cloud (SMC), and tiny corp.

MLCommons would like to especially welcome first-time MLPerf Training submitters Juniper Networks, Oracle, SMC, and tiny corp. 

Congratulations to first-time participant SMC for submitting the first-ever set of power results for MLPerf Training. These results highlight the impact of SMC’s immersion cooling solutions for data center systems. Our industry-standard power measurement works with MLPerf Training and is the first and only method to accurately measure full system power draw and energy consumption for both cloud and on-premise systems in a trusted and consistent fashion. These metrics are critical for the entire community to understand and improve the overall efficiency for training ML models – which will ultimately reduce the energy use and improve the environmental impact of AI in the coming years.

The Training v4.0 results demonstrate broad industry participation and showcase substantial performance gains in ML systems and software. Compared to the last round of results six months ago, this round brings a 1.8X speed-up in training time for Stable Diffusion. Meanwhile, the best results in the RetinaNet and GPT3 tests are 1.2X and 1.13X faster, respectively, thanks to performance scaling at increased system sizes.

“I’m thrilled by the performance gains we are seeing especially for generative AI,” said David Kanter, executive director of MLCommons. “Together with our first power measurement results for MLPerf Training we are increasing capabilities and reducing the environmental footprint – making AI better for everyone.”

New LLM fine-tuning benchmark

The MLPerf Training v4.0 suite introduces a new benchmark to target fine-tuning a large language model (LLM). An LLM that has been pre-trained on a general corpus of text can be fine-tuned to improve its accuracy on specific tasks, and the computational costs of doing so can differ from pre-training.

A variety of approaches to fine-tuning an LLM at lower computational costs have been introduced over the past few years. The MLCommons Training working group evaluated several of these algorithms and ultimately selected LoRA as the basis for its new benchmark. First introduced in 2021, LoRA freezes original pre-trained parameters in a network layer and injects trainable rank decomposition matrices. Since LoRA fine-tuning trains only a small portion of  the network parameters, this approach dramatically reduces the computational and memory demands compared to pre-training or supervised fine-tuning.

“Fine-tuning LLMs is a notable workload because AI practitioners across many organizations make use of this technology. LoRA was the optimal choice for a state-of-the-art fine-tuning technique; it significantly reduces trainable parameters while maintaining performance comparable to fully fine-tuned models,” said Hiwot Kassa, MLPerf Training working group co-chair.

The new LoRA benchmark uses the Llama 2 70B general LLM as its base. This model is fine-tuned with the Scrolls dataset of government documents with a goal of generating more accurate document summaries. Accuracy is measured using the ROUGE algorithm for evaluating the quality of document summaries. The model uses a context length of 8,192 tokens, keeping pace with the industry’s rapid evolution toward longer context lengths.

The LLM fine-tuning benchmark is already achieving widespread adoption, with over 30 submissions in its initial round.

Learn more about the selection of the LoRA fine-tuning algorithm for the MLPerf Training benchmark in this blog.

New GNN benchmark for classification in graphs

MLPerf Training v4.0 also introduces a graph neural network (GNN) benchmark for measuring the performance of ML systems on problems that are represented by large graph-structured data, such as those used to implement literary databases, drug discovery applications, fraud detection systems, social networks, and recommender systems. 

“Training on large graph-structured datasets poses unique system challenges, demanding optimizations for sparse operations and inter-node communication. We hope the addition of a GNN based benchmark in MLPerf Training broadens the challenges offered by the suite and spurs software and hardware innovations for this critical class of workload,” said Ritika Borkar MLPerf Training working group co-chair.

The MLPerf Training GNN benchmark is used for a node classification task where the goal is to predict a label for each node in a graph. The benchmark uses an R-GAT model and is trained on the 2.2 terabyte IGBH full dataset, the largest available open-source graph dataset with 547 million nodes and 5.8 billion edges. The IGBH database is a graph showing the relationships between academic authors, papers, and institutes. Each node in the graph can be classified into one of 2,983 classes.

The MLPerf Training team recently submitted MLPerf R-Gat to the Illinois Graph Benchmark (IGB) leaderboard which helps the industry keep track of the state of the art for GNN models, encouraging reproducibility. We are pleased to announce that their submission is currently #1 with a 72% test accuracy.  

Learn more about the selection of the GNN benchmark in this blog.

View the results

To view the full results for MLPerf Training v4.0 and find additional information about the benchmarks, please visit the Training benchmark page.

About MLCommons

MLCommons is the world leader in building benchmarks for AI. It is an open engineering consortium with a mission to make AI better for everyone through benchmarks and data. The foundation for MLCommons began with the MLPerf benchmarks in 2018, which rapidly scaled as a set of industry metrics to measure machine learning performance and promote transparency of machine learning techniques. In collaboration with its 125+ members, global technology providers, academics, and researchers, MLCommons is focused on collaborative engineering work that builds tools for the entire AI industry through benchmarks and metrics, public datasets, and measurements for AI Safety.

For additional information on MLCommons and details on becoming a member or affiliate, please visit MLCommons.org or contact participation@mlcommons.org.

The post New MLPerf Training Benchmark Results Highlight Hardware and Software Innovations in AI Systems appeared first on MLCommons.

Generative AI has captured the public’s attention and imagination; its uses can be widespread and revolutionary. Large language models (LLMs) can perform language-related tasks and power the advanced conversational abilities of services such as ChatGPT, Gemini, Perplexity AI, and many more.

The MLPerf Training benchmark suite seeks to provide a standard means for measuring performance in machine learning (ML) model training, so naturally, the MLPerf Training working group has been watching these developments closely and considering how to expand the benchmark suite to cover emerging training scenarios.

 Training LLMs can be broadly classified into two phases: pre-training and fine-tuning. 

• Pre-training is the crucial initial phase where the neural network is trained on a large and diverse dataset for generic language understanding such as grammar, idioms, facts, and the subtleties of different contexts. The dataset for pre-training typically consists of trillions of words from books, articles, websites, and social media. This phase is akin to developing a general understanding of language as the model learns to predict the next token based on past context. The MLPerf GPT-3 benchmark captures the performance of the pre-training phase of the LLM development life-cycle.

• Fine-tuning adapts a pre-trained model to specific tasks or domains by further training the model on a smaller, task-specific dataset to enhance its performance. This process boosts training efficiency by reducing computational intensity and enhances performance on specific tasks without starting training over from scratch. 

Fine-tuning has been widely adopted by the industry because of its cost effectiveness and reduced infrastructure requirements which make it highly accessible. With this consideration in mind, the MLPerf Training working group formed a special task force in 2023 to explore and prioritize the inclusion of a fine-tuning benchmark in MLPerf Training.

Model selection

The task force evaluated numerous model candidates for inclusion, ranging from smaller models like Llama 2 7B and Mixtral 7B to mid-range models such as Falcon 40B, MPT 30B, and Llama 2 70B. After thorough evaluation and deliberation, the task force selected Llama 2 70B as the model that most aligned with its objectives.

Several factors contributed to this decision:

• Model diversity: One of the goals of MLPerf Training is to curate a small set of models with diverse characteristics so that the overall suite represents the much larger universe of trained models. With the right mix of models, the benchmark results can be generally and broadly useful. The existing MLPerf Training suite includes two language models: BERT (340 million parameters) and GPT-3 (175 billion parameters). The addition of Llama 2 70B (70 billion parameters) adds another distinct model size along with some architectural diversity thanks to features such as group query attention (GQA), SwiGLU activation function, and rotary embeddings.

• Community engagement: The task force considered Hugging Face contributions and leaderboards as solid indicators of community engagement and industry adoption. The Llama 2 70B model emerged as the frontrunner in terms of community engagement at the time of benchmark development. Since then, Meta has released Llama 3 70B, which was trained on a larger dataset with an improved tokenizer. However, the majority of the network layers are common across the two models, so per-iteration performance on Llama 2 will likely track closely to performance on Llama 3.

• Licensing flexibility: For an open benchmark consortium, model licenses must be flexible enough to offer unlimited access to participating submitters at minimum and, ideally, to the press and public in general. This requirement typically reduces the candidate pool to models (and datasets) with permissive licenses such as Apache 2.0, MIT, or similar. Thanks to Meta’s support, MLCommons is making the Llama 2 family of models available to MLCommons members for the purpose of conducting benchmarking with the MLPerf benchmark suites.

• Efficient benchmarking integration: Llama 2 70B offers easy deployment, simplifying benchmarking and integration for the task force. Also, model scales are growing exponentially, so large language models may not fit on a single accelerator anymore. To train on multiple accelerators, AI practitioners will have to employ methods like model parallelism to split the model efficiently across devices. Choosing a moderate-sized model like Llama 2 70B creates opportunities for submitters to showcase such software innovations. 

Selecting a fine-tuning technique 

Next, the task force investigated three different techniques for optimizing large language models for downstream tasks: supervised fine-tuning (SFT) which operates on training an entire model on curated datasets; reinforcement learning (RLHF) which utilizes human feedback to guide model training; and parameter-efficient fine tuning (PEFT) which updates only a subset of the model parameters.

PEFT stood out among these three approaches as the best fine-tuning technique for training because of its computational efficiency and architectural simplicity. Low-rank adaptation (LoRA) is a PEFT technique which freezes pre-trained model weights while injecting trainable low-rank decomposition matrices. We chose LoRA as the preferred fine-tuning technique for MLPerf Training due to its reduced parameter counts and memory footprint, efficient training process, lower barrier to entry, and popularity (which was likely driven by the prior factors.) Also, LoRA fine-tuning has different performance characteristics than existing MLPerf benchmarks.

Dataset selection

For this benchmark, the task force chose the SCROLLS (Standardized CompaRison Over Long Language Sequences) government report dataset. The dataset is made up of question-summary pairs based on reports written by government research agencies including Congressional Research Service and U.S. Government Accountability Office. 

Context length is the amount of historical data the model uses to understand the context before making a forecast. Enabling LLMs to process and understand larger context lengths is crucial for capturing nuanced relationships and generating coherent and relevant text. The task force experimented with various context lengths and opted for the longest context that fit within a single, eight-accelerator system: 8192 tokens. That’s the equivalent of roughly 6,144 words and is significantly longer context than the existing language models in MLPerf Training, which have sequence lengths of 512 and 2048 for BERT and GPT3, respectively.

Benchmark technical details

The Llama 2 70B LoRA benchmark is based on the Hugging Face ecosystem and tools in order to represent a common developer experience. Hugging Face is one of the most widely used environments for fine-tuning due to the extensive and easy-to-use Transformers library, which provides researchers and developers with pre-trained models. Developers can adapt these models with user-friendly and well-documented APIs, enabling efficient and effective fine-tuning. 

To ensure the representative nature of this benchmark, the task force followed the most common convention among Hugging Face users for fine-tuning LLMs by leveraging AutoModelForCausalLM. AutoModelForCausalLM is a versatile tool provided by Hugging Face designed for tasks involving causal language modeling. This model is primarily used for generating text, where the goal is to predict the next word in a sequence, given the preceding words. 

The MLPerf Training v4.0 benchmark also includes:

• Loss metric: The benchmark uses cross entropy of the next token prediction for the loss function (much like MLPerf GPT3).

• Target accuracy metric: The standard metric for text summarization is the ROUGE score. Due to the computationally intensive and time-consuming nature of computing ROUGE scores, the typical approach involves analyzing the convergence graph of the evaluation loss. Once convergence is detected, ROUGE scores are computed to evaluate performance. Our testing of this method unveiled a strong correlation (R score of 0.9) between evaluation loss and ROUGE scores. Note that, unlike the training loss, the evaluation loss is calculated only over the target tokens.

• Selected region of convergence: To ensure fairness, each submitter is required to run the benchmark 10 times. Results are then scored using Olympic averaging. 

Conclusion

In this blog post, we shared the insights, motivation, and process behind the creation of the new MLPerf Llama 2 70B LoRA fine-tuning benchmark. This benchmark is less computationally intensive than GPT-3, so it should serve as an excellent starting point for working with language models and lower the barrier to entry for MLPerf participants. We are thrilled by the community’s interest in the benchmark, as evidenced by 30 submissions in MLPerf Training v4.0 for this benchmark, and we hope to see this enthusiasm continue to grow.

The post LoRA selected as the fine-tuning technique added to MLPerf Training v4.0 appeared first on MLCommons.

The field of AI has been rapidly advancing over the past decade, and the MLPerf^® Training benchmark suite is constantly evolving to capture the zeitgeist. New benchmarks are added based on MLCommons^® member consensus and customer interests. 

For MLPerf Training v4.0, we are pleased to introduce a graph neural network (GNN) training benchmark based on the Relational Graph Attention Network (R-GAT) model. The benchmark is a result of almost a year of close collaboration between teams from Alibaba, Intel, and NVIDIA. 

About graph neural networks 

Graph neural networks (GNNs) have emerged as an important class of models to employ with graph-structured data such as social graphs. GNNs are used in a wide range of areas such as recommendation systems, fraud detection, knowledge graph answering, and drug discovery, to name a few. From a computational perspective, sparse operations and message passing between nodes of the graph make GNNs stand apart from other benchmarks in the training suite and present new challenges for system optimization and scalability.  

Dataset and model 

Commercial applications of GNNs typically use very large graphs that do not fit into the memory of a single node. To reflect the multi-node nature of the workload, we sought to use the largest available public dataset. Our initial proposal was to use the MAG-240M dataset from Open Graph Benchmarks, consisting of ~240 million nodes and ~1.7 billion edges. With the release of the Illinois Graph Benchmark (IGB) dataset in March 2023, we settled upon using the heterogeneous version of the dataset (IGBH-Full) for the benchmark. For more details, please refer to the IGB publication and repo. 

The IGBH dataset is a real-world citation graph consisting of ~547 million nodes and  ~5.8 billion edges, with more than 40% labeled nodes. The dataset schema is illustrated in Figure 1. The benchmark’s task is to classify the paper nodes of the graph among 2,983 available topics. We also augment the dataset by adding reverse edges to improve accuracy. 

Figure 1: IGB-Heterogeneous dataset schema (source). 

While there are a variety of architectures for GNNs, we chose to use the R-GAT model due to its popularity. This model leverages the attention mechanism proposed in the transformer model to focus on data subsets that are of greatest importance. After a thorough exploration of model architectures, we arrived at a configuration that achieves 72% classification accuracy on the validation set. 

Technical details 

We use a R-GAT model with three layers with [5,10,15] fanout, hidden dimension of 512, and four attention heads. The model is trained from scratch on a predetermined training subset of the graph (60% of the nodes) using the Adam optimizer. The eval dataset is fixed to be a 1/40th fraction of the full validation dataset–about 0.8 million nodes–with accuracy evaluated 20 times per epoch. To obtain a representative result, submitters must run the benchmark 10 times with different random seeds. Results are then scored using Olympic averaging.

The reference code is implemented using Alibaba’s GraphLearn-for-Pytorch (GLT) framework. GLT is a library for large-scale GNN training in Pytorch which supports distributed CPU/GPU-based sampling and training and is fully compatible with PyG. The reference code is designed to run on both NVIDIA GPUs and Intel Xeon platforms.

As we developed the benchmark, we had a healthy debate on the rules governing the benchmark so that the typical industry use case is reflected. For multi-node runs, the graph must be partitioned so that features of a given graph node are read exclusively from one training node to ensure feature-fetching over the network. We also spent considerable effort understanding the sampling differences between PyG and DGL, the two popular GNN frameworks. While the two samplers are not mathematically equivalent, we allow submitters to use either version out of the box because the convergence differences between them were minimal in our experiments. We have also contributed a PyG-style sampler in DGL for community usage.

We recently submitted MLPerf R-GAT to the Illinois Graph Benchmark (IGB) leaderboard which helps the industry keep track of the state of the art for GNN models, encouraging reproducibility, and we are pleased to announce that our submission is currently #1 with a 72% test accuracy.  

Conclusion

We have described the motivation and the process of creating the new MLPerf GNN training benchmark. This benchmark will allow for fair evaluation of various systems, frameworks, and optimization techniques today and is expected to influence the design of future AI systems. We are delighted by the community interest in this benchmark, as shown by the number of submissions to MLPerf Training v4.0, and hope to see this interest grow further. 

Further details of the benchmark and the reference implementation can be found here. 

The post Introducing the MLPerf Training Benchmark for Graph Neural Networks  appeared first on MLCommons.

• The MLPerf Training v3.1 suite, which measures the performance of training machine learning models.
• The MLPerf HPC (High Performance Computing) v.3.0 benchmark suite, which is targeted at supercomputers and measures the performance of training machine learning models for scientific applications and data. 

MLPerf Training v3.1
The MLPerf Training benchmark suite comprises full system tests that stress machine learning models, software, and hardware for a broad range of applications. The open-source and peer-reviewed benchmark suite provides a level playing field for competition that drives innovation, performance, and energy-efficiency for the entire industry.

MLPerf Training v3.1 includes over 200 performance results from 19 submitting organizations: Ailiverse, ASUSTek, Azure, Azure+NVIDIA, Clemson University Research Computing and Data, CTuning, Dell, Fujitsu, GigaComputing, Google, Intel+Habana Labs, Krai, Lenovo, NVIDIA, NVIDIA+CoreWeave, Quanta Cloud Technology, Supermicro, Supermicro+Red Hat, and xFusion. MLCommons would like to especially congratulate first-time MLPerf Training submitters Ailiverse, Clemson University Research Computing and Data, CTuning Foundation, and Red Hat.

The results demonstrate broad industry participation and highlight performance gains of up to 2.8X compared to just 5 months ago and 49X over the first results, reflecting the tremendous rate of innovation in systems for machine learning. 

Significant to this round, is the largest system ever submitted to MLPerf Training. Comprising over 10K accelerators, it demonstrates the extraordinary progress by the machine learning community in scaling system size to advance the training of neural networks.

MLPerf Training v3.1 introduces the new Stable Diffusion generative AI benchmark model to the suite. Based on Stability AI’s Stable Diffusion v2 latent diffusion model, Stable Diffusion takes text prompts as inputs and generates photorealistic images as output. It is the core technology behind an emerging and exciting class of tools and applications such as Midjourney and Lensa.

“Adding Stable Diffusion to the benchmark suite is timely, given how image generation has exploded in popularity,” said Eric Han, MLPerf Training co-chair. “This is a critical new area – extending Generative AI to the visual domain.”

MLCommons added the GPT-3 benchmark to MLPerf Training v3.0 last June. In just five months, the LLM benchmark has shown over 2.8X in performance gains. Eleven submissions in this round include this large language model (LLM) using the GPT-3 reference model, reflecting the tremendous popularity of generative AI. 

“GPT-3 is among the fastest growing benchmarks we’ve launched,” said David Kanter, Executive Director, MLCommons. “It’s one of our goals to ensure that our benchmarks are representative of real-world workloads and it’s exciting to see 2.8X better performance in mere months.”

MLPerf HPC v3.0 Benchmarks
The MLPerf HPC benchmark is similar to MLPerf Training, but is specifically intended for high-performance computing systems that are commonly employed in leading-edge scientific research. It emphasizes training machine learning models for scientific applications and data, such as quantum molecular dynamics, and also incorporates an optional throughput metric for large systems that commonly support multiple users.

MLCommons added a new protein-folding benchmark in the HPC v3.0 benchmark suite: the OpenFold generative AI model, which predicts the 3D structure of a protein given a 1D amino acid sequence. Developed by Columbia University, OpenFold is an open-source reproduction of the AlphaFold 2 foundation model and has been the cornerstone of a large number of research projects since its creation. 

MLPerf HPC v3.0 includes over 30 results – a 50% increase in participation over last year, and  includes submissions by 8 organizations with some of the world’s largest supercomputers: Clemson University Research Computing and Data, Dell, Fujitsu+RIKEN, HPE+Lawrence Berkeley National Laboratory, NVIDIA, and Texas Advanced Computing Center. MLCommons congratulates first-time MLPerf HPC submitters Clemson University Research Computing and Data and HPE+Lawrence Berkeley National Laboratory.

The new OpenFold benchmark includes submissions from 5 organizations: Clemson University Research Computing and Data, HPE+Lawrence Berkeley National Laboratory, NVIDIA, and Texas Advanced Computing Center,

HPC v3.0 Performance Gains
The MLPerf HPC benchmark suite demonstrates considerable progress in AI for science that will help unlock new discoveries. For example, the DeepCAM weather modeling benchmark is 14X faster than when it debuted, illustrating how rapid innovations in machine learning systems can empower scientists with better tools to address critical research areas and advance our understanding of the world. 

“The addition of OpenFold follows the spirit of the MLPerf HPC benchmark suite: Accelerating workloads with potential for global-scale contribution. We are excited for the new addition as well as the increased participation in the latest submission round.” said Andreas Prodromou, MLCommons HPC co-chair.

View the Results
To view the results for MLPerf Training v3.1 and MLPerf HPC v3.0 and find additional information about the benchmarks, please visit the Training and HPC benchmark pages.

About MLCommons
MLCommons is the world leader in building benchmarks for AI. It is an open engineering consortium with a mission to make machine learning better for everyone through benchmarks and data. The foundation for MLCommons began with the MLPerf benchmarks in 2018, which rapidly scaled as a set of industry metrics to measure machine learning performance and promote transparency of machine learning techniques. In collaboration with its 125+ members, global technology providers, academics, and researchers, MLCommons is focused on collaborative engineering work that builds tools for the entire machine learning industry through benchmarks and metrics, public datasets, and best practices.

For additional information on MLCommons and details on becoming a member or affiliate, please visit MLCommons.org or contact participation@mlcommons.org.

The post New MLPerf Training and HPC Benchmark Results Showcase 49X Performance Gains in 5 Years appeared first on MLCommons.