Today, we are announcing that MedPerf has added WebUI capabilities, making it easier for benchmark creators to orchestrate. The WebUI extends Medperf’s capabilities to the browser, providing an intuitive interface built on its CLI. Running entirely on the local machine, the WebUI executes the same commands and functions as the CLI, offering a more visual, accessible way to interact with Medperf without changing how it works under the hood. Access is secured by a unique token generated at launch, ensuring that access remains restricted to the user who launched it. By combining the power of the CLI with the convenience of a browser, Medperf WebUI aims to deliver a user-friendly experience to all Medperf users, including non-technical experts (e.g. clinicians).

We tested the WebUI in our current clinical study for glioblastoma to facilitate the evaluation of AI models on 1568 brain scans across 7 data sites from North America and Europe. Through the WebUI the study’s principal investigator, Dr. Evan Calabrese, followed a step-by-step workflow to create a validation record accompanied by the data preparation script, the metrics script and the model under evaluation, which was pre-trained on 10208 brain scans across 53 data sites from 5 continents (North America, South America, Europe, Asia, and Australia) in the first phase of the study via federated learning using Intel’s OpenFL. Through the WebUI, the representative from each evaluation site followed a step-by-step workflow to prepare the data locally and initiate the evaluation. Evaluation results were monitored via the principal investigator’s dashboard on the WebUI.

We also created an online tutorial for new MedPerf users to learn how to use the new WebUI in easy steps which can be found here: Full Tutorial (WebUI) – MedPerf.

Dr. Evan Calabrese, the lead principal investigator for the clinical study mentioned:

“The MedPerf WebUI was beneficial for the validation phase of the Federated Learning for Postoperative Segmentation of Tumors (FL-PoST)/Federated Tumor Segmentation (FeTS) 2.0 study, which required international collaboration across ~10 different academic medical centers to validate our brain tumor segmentation model trained on over 10,000 individual MRI exams from >35 institutions. In particular, the familiar web-based interface enabled efficient collaboration across time zones without the need for live support, even for those with limited technical literacy. Using the MedPerf WebUI, we were able to work asynchronously across seven individual external sites without any significant support delays. Once we completed the validation phase, the results could be easily viewed or downloaded from the WebUI on our end. It’s clear to me that this WebUI will be the future of MedPerf, enabling broader collaboration without increased support burden.”

Other feedback from clinical and academic contexts included:

“Click and click, step-by-step, all necessary information included in the webUI manual. Web UI was easy to follow, and intuitive to understand the current evaluation status!”

Jaeyoung Cho, CCIBonn.ai, University Hospital Bonn

“The MedPerf Web UI offers an intuitive and user-friendly interface that can be easily operated by both engineers and clinicians with minimal learning effort. As a standardized evaluation platform, it simplifies the benchmarking of models across datasets using multiple performance metrics, facilitating both clinical deployment and future research. I particularly appreciate its integration with compute cluster environments via Docker or Singularity, which makes installation and setup straightforward for any user.”

Rachit Saluja, PhD student, Cornell University & Weill Cornell Medicine

“The interface was easy to learn, and having the post-pipeline segmentations easily accessible through ITK-snap in the UI made processing the segmentations much quicker. It was very convenient to receive multiple labels to show the different components of the segmentations.“

Erik Elgeskog, University of Gothenburg

A Call to Action

We invite the research and medical AI community to explore the new MedPerf WebUI and experience a more accessible way to benchmark and validate models. Your participation is vital to our growth; we encourage you to test the interface, collaborate with peers, and report both your technical hurdles and your success stories to help us refine the platform. Beyond testing, this is an invitation to make a tangible impact: join us in our ongoing AI-based clinical studies and help accelerate the adoption of trusted medical AI

About MedPerf

MedPerf is an open benchmarking platform that aims at evaluating AI on real-world medical data. MedPerf follows the principle of federated evaluation in which medical data never leaves the premises of data providers. Instead, AI algorithms are deployed within the data providers and evaluated against the benchmark. Results are then manually approved for sharing with the benchmark authority. This effort aims to establish global federated datasets and develop scientific benchmarks to reduce the risks of medical AI, such as bias, lack of generalizability, and potential misuse. We believe this two-pronged strategy will enable clinically impactful AI and improve healthcare efficacy.

Learn more at https://mlcommons.org/working-groups/data/medical/.  

About MLCommons 

MLCommons is an open engineering consortium with a mission to make machine learning better for everyone. The organization produces industry-leading benchmarks, datasets, and best practices that span the full range of ML applications—from massive cloud training to resource-constrained edge devices. Its MLPerf benchmark suite has become the de facto standard for evaluating AI performance.

Learn more at www.mlcommons.org.

The post MedPerf adds WebUI capabilities to make federated benchmarking more user-friendly appeared first on MLCommons.

The AlgoPerf: Training Algorithms Competition

To make building useful neural network models less time-consuming and costly, we need better training algorithms. The MLCommons Algorithms working group has developed the open-source AlgoPerf: Training Algorithms benchmark to measure how much faster neural networks can be trained through advancements in underlying training algorithms, such as better optimizers or more effective hyperparameter choices. 

The AlgoPerf: Training Algorithms benchmark evaluates the time training required for different training algorithms across multiple realistic deep learning workloads when running on a fixed hardware configuration. To encourage generally useful methods, submissions must fully specify any required workload-specific tuning. Participants could choose to submit under two separate tuning rulesets: the external tuning ruleset, designed to simulate tuning with a limited amount of parallel resources, or the self-tuning ruleset, designed to simulate fully automated tuning on a single machine.

Participation

The first iteration of the AlgoPerf: Training Algorithms competition attracted 18 submissions (with 15 being scorable) from 10 different teams. Scoring involved over 4000 individual training runs across the 14 workloads used in the benchmark. Participants included researchers from Concordia University, ELLIS Tübingen, Google, Max Planck Institute for Intelligent Systems, Meta AI, Meta Platforms, Michigan State University, Mila, Samsung AI, UCLA, UT Austin, the University of Cambridge, the University of West Indies, and the Vector Institute. 

The submissions collectively explored many interesting techniques and implementation choices, including submissions using both of our supported frameworks, JAX and PyTorch. As required by the rules, all submissions are released publicly under an Apache 2.0 open-source license.

The Winners & Results

Congratulations to Aaron Defazio (Meta), Alice Yang (Meta), and Konstantin Mishchenko (Samsung AI) who came in first place in the self-tuning ruleset with their “Schedule Free AdamW” submission (see Table 2). For the external tuning ruleset (see Table 1, below), first place goes to the “Distributed Shampoo” submission of Hao-Jun Michael Shi, Tsung-Hsien Lee, Anna Cai, Shintaro Iwasaki, Wenyin Fu, Yuchen Hao, and Mike Rabbat (all Meta).

The external tuning ruleset saw four submissions beating the challenging prize-qualification baseline, improving over the state-of-the-art training algorithm. The “Distributed Shampoo” submission provides an impressive 28% faster model training compared to the baseline. “Schedule Free AdamW” was the only submission in the self-tuning ruleset that beat the prize-qualification baseline, improving over it by providing an 8% faster neural network training process.

Congratulations to the winners and all participants for their contributions to advancing neural network training algorithms!

To receive a cash prize, the competition rules require that at least one other submission outperforms the prize qualification baseline for the relevant ruleset, AND that none of the authors of this competing submission share an affiliation with either of the two MLCommons Algorithms working group chairs. This condition was met for the external tuning ruleset, and thus a cash prize of $25,000 will be awarded by MLCommons for the first place submission. Despite the outstanding performance of the first place submissions in the self-tuning ruleset, the prize requirement was not met, since several competing submissions involved overlapping affiliations with the working group chairs, and the prize qualification baselines were quite difficult to beat. The working group’s goal in designing the AlgoPerf: Training Algorithms benchmark competition was to, first and foremost, make sure that any submission that performed well under our rules had to achieve something truly impressive, and we are delighted that the first place submissions in both rulesets managed to produce such exceptional results.

To view the full results of the AlgoPerf: Training Algorithms competition, including the workload-specific performances of each submission, please visit the AlgoPerf results page. We plan to release a paper with a more in-depth discussion of the results after we are done analyzing them in detail.

The next steps for AlgoPerf

The first iteration of AlgoPerf: Training Algorithms demonstrated that neural network training can be accelerated significantly by improving the underlying training algorithms. This iteration was only the first step in driving innovation in machine learning algorithms. Now that we can reliably measure progress in training algorithms, we anticipate rapid progress in this field, both in terms of new research and better methods. The working group is already hard at work planning for the future of the benchmark. If you are interested in shaping this future, developing and scoring any particular submissions, or collaborating on research that builds on top of our benchmark, please consider joining the working group. 

Acknowledgments

We extend our sincere thanks to Google for their generous support in providing computational resources to score and evaluate all submissions across the workloads. Our gratitude also goes to the entire MLCommons organization for supporting the Algorithms working group and funding the $50,000 prize pool. Special thanks are due to the members of the Algorithms Working Group who developed, implemented, and managed the benchmark competition. We particularly want to thank Priya Kasimbeg, the Engineering Lead of the working group, who led the scoring process.

About MLCommons and the Algorithms Working Group

MLCommons is the world leader in building benchmarks for AI. It is an open engineering consortium with a mission to make AI better for everyone through benchmarks and data.

The AlgoPerf: Training Algorithms benchmark was developed by the MLCommons Algorithms Working Group. Researchers from a variety of academic institutions and industry labs serve on the working group. The group’s mission is to create a set of rigorous and relevant benchmarks to measure neural network training speedups due to algorithmic improvements. For additional information on the Algorithms Working Group, and details on how to become a member or contribute to the benchmarks, please visit the working group website or reach out to algorithms-chairs@mlcommons.org.

The post Announcing the results of the inaugural AlgoPerf: Training Algorithms benchmark competition appeared first on MLCommons.

Faster training allows researchers to build more capable machine learning (ML) models, but unlocking the most valuable capabilities requires improvements in every part of the training pipeline. The MLPerf Training benchmark suite has been extremely successful in encouraging innovation in neural network training systems, but more work needs to be done to encourage innovation in training algorithms. Improved training algorithms could save time, computational resources, and lead to better, more accurate, models. Unfortunately, as a community, we are currently unable to reliably identify training algorithm improvements, or even determine the state-of-the-art training algorithm. To accelerate this work, the MLCommons® Algorithms Working Group is delighted to announce the AlgoPerf: Training algorithms competition, which is designed to measure neural network training speedups due to algorithmic improvements (e.g. better optimizers or hyperparameter tuning protocols). 

The AlgoPerf: Training algorithms benchmark is a competitive, time-to-result benchmark that runs on a fixed system and compares training algorithms on multiple deep learning workloads (see Table 1 below). In contrast, in MLPerf Training submitters typically compete on the ML training systems. For AlgoPerf, however, the hardware and lower-level software environments are fixed, so submitters must develop and compete on the basis of more efficient algorithms.

To ensure that the benchmark incentivizes generally useful training algorithms, submissions must simultaneously perform well across multiple workloads, including some randomized ones. This includes a wide variety of workloads across many domains to ensure that the results are broadly applicable and relevant to more ML practitioners. The competition will determine the best general-purpose method, measured by an aggregated score across all workloads. The current workloads are listed below.

To further encourage generality, competition submissions must automate—and strictly account for—any workload-specific hyperparameter tuning they perform. Submissions are allowed under two separate tuning rulesets: an external tuning ruleset meant to simulate tuning with a fixed amount of parallel resources, or a self-tuning ruleset meant to simulate tuning on a single machine.

The Competition is Open NOW!

The AlgoPerf: Training algorithms benchmark competition opens on November 28, 2023, and is scheduled to close on March 28, 2024. To enter the competition please see the instructions on the competition website. Additionally, the accompanying technical report motivates and explains the design choices of the benchmark.

Sponsorship & Prize Money

MLCommons is offering a total prize pool of $50,000, to be awarded by a committee, for the top-performing submissions in each tuning ruleset. 

We would also like to express our gratitude to Google for their generous support in providing computational resources to score the top submissions, and resources to help score promising submissions from submitters with more limited resources.

About the MLCommons Algorithms Working Group

The MLCommons AlgoPerf: Training algorithms benchmark was developed by the MLCommons Algorithms Working Group. Researchers from a variety of academic institutions and industry labs serve on the working group. The group’s mission is to create a set of rigorous and relevant benchmarks to measure neural network training speedups due to algorithmic improvements. For additional information on the Algorithms Working Group and details on how to become a member or contribute to the benchmarks, please visit the working group website or reach out to algorithms-chairs@mlcommons.org.

The post Announcing the MLCommons AlgoPerf Training Algorithms Benchmark Competition appeared first on MLCommons.