A brain tumor (glioblastoma) is a rare disease with devastating outcomes for life expectancy. AI has the potential to improve the diagnosis and prognosis of brain tumor patients through a multitude of automated processes – including boundary definition/segmentation, tumor classification, and tumor quantification. Thanks to the Federated Tumor Segmentation (FeTS) consortium, coordinated by Dr. Spyridon Bakas at Indiana University School of Medicine and the Response Assessment in Neuro-Oncology (RANO) cooperative group, the MedPerf team demonstrated evaluation of a clinically valuable AI model – designed by Dr. Evan Calabrese’s team at Duke University – on real-world brain tumor MRI data.

With MedPerf on Google Cloud’s Confidential Computing, we aim to advance medical AI research worldwide while securely evaluating models on medical data. Using Google Cloud’s Confidential Space, a Trusted Execution Environment (TEE), medical data, model weights, and benchmarks are fully protected throughout the benchmarking process.

The Problem: Life-Threatening Disease, Locked-Down Data

Protecting both AI models and patient data in the medical space is paramount, driven by privacy, ethical, sovereign, and regulatory considerations. At the same time, measuring the performance of AI models on real-world patient data is critical for enabling trust and adoption of AI among clinicians, patients, regulators, and payors.

The MLCommons community built MedPerf to address exactly this gap. MedPerf uses a federated approach – introduced to healthcare by Dr. Bakas’ team – in which AI models move to the medical data owners (such as healthcare organizations, hospitals, and data brokers), rather than requiring data to move to the benchmark operator. This allows data owners to keep their data private while sharing only summarized results with the benchmark operator.

However, while this federated approach protects patient data, it does not fully address concerns around model theft, intellectual property, and benchmark integrity. Without additional guarantees, there is no assurance that model weights won’t leak or that benchmark results won’t be tampered with at the time of execution – raising serious concerns about trust and participation in the benchmark itself.

The Solution: MedPerf on Google Cloud Confidential Space

MedPerf’s integration with Google Cloud’s Confidential Space solves exactly this problem: protecting both model weights and benchmark results while simultaneously safeguarding medical data.

This integration brings four key benefits to MedPerf users:

1. A global test dataset for healthcare AI – enabling evaluation at unprecedented scale
2. Patient data is protected – data never leaves the data owner’s secure environment
3. Model IP is protected – model weights remain encrypted and inaccessible to unauthorized parties
4. No data sharing required – results are summarized and encrypted before transmission

The Demo: A Step-by-Step Secure Benchmark

A benchmark authority first publishes a benchmark on the MedPerf server, describing the benchmark’s aim, data preparation guidelines, and evaluation criteria. Data owners and AI model owners then register metadata for their private assets with the MedPerf server and link their registrations to the benchmark.

Private assets are encrypted using Google Cloud’s Cloud Key Management Service (Cloud KMS) and uploaded to a Google Cloud Storage bucket. Access to Cloud KMS and the bucket is governed by a Google Cloud workload identity pool, which allows access only if the request originates from a confidential virtual machine with specific measurements – such as the benchmark container hash and input hash.

When it’s time to run, data owners execute the benchmark container inside a Confidential Space virtual machine. The container receives cryptographic evidence of the trusted execution environment’s configuration, which is exchanged for access tokens from the model owner’s and data owner’s workload identity pools. These tokens retrieve and decrypt the model and dataset, inference is executed, results are encrypted with the data owner’s public key, and the encrypted results are returned to the data owner’s bucket.

The video below demonstrates a brain tumor segmentation AI model running on a clinical dataset in Google Cloud:

Watch the Demo

Voices from the Community

“My experience testing federated learning on Google Cloud has shown that the future of medical AI lies in secure, scalable, and collaborative cloud environments. Moving beyond the controlled lab setting to test these workflows in a production-ready infrastructure provided a unique opportunity to evaluate the performance and security of federated learning in real-world clinical applications. This collaboration between the FeTS community, MLCommons, and Google Cloud demonstrates how scalable cloud solutions can accelerate the development of high-precision diagnostic tools in neuroradiology.” – Dr. Yury Velichko, Northwestern University

“Federated Learning offers a paradigm shift in healthcare multi-site collaborations that expedites access to unprecedented amounts of knowledge from diverse patient populations, towards robust generalizable AI models with real clinical impact. Security and privacy are prime considerations to ensure that FL-related assets are always protected. Training and evaluating AI models on confidential compute infrastructure, using MLCommons’ MedPerf on Google Cloud, describes a complete exemplary solution for large-scale real-world clinically-relevant AI studies in healthcare, providing the ability to track how well an AI model continues to perform after it has received regulatory approval, for example, from the FDA.” – Dr. Spyridon Bakas, Indiana University School of Medicine

“One of the greatest barriers to trustworthy medical AI has been the tension between the need for real-world data and the need to protect it. The integration of MedPerf with Google Cloud’s Confidential Space is a tangible solution to that tension – one that the MLCommons community has worked hard to bring to life. We are proud to demonstrate what is possible when open collaboration meets production-grade security infrastructure.” – Alexandros Karargyris, MedPerf Lead, MLCommons

Join the MedPerf Community

MedPerf is an open-source effort developed by the medical AI community to address real community needs. Thanks to contributions from the MLCommons community, MedPerf has already supported impactful clinical AI studies, including the FeTS challenge and FeTS 2.0 for brain tumors.

We know we still have much more to do – and we need your help. We are looking for:

• Research investigators with the ambition to lead global clinical AI studies
• Computational experts who want to build disruptive AI on real-world medical data
• Infrastructure engineers to improve real-world benchmarking technologies
• Privacy and governance experts to provide input on medical data access controls and policy
• Regulatory experts to provide feedback on bringing regulatory alignment to real-world benchmarking

Acknowledgments

Contributors to the MedPerf confidential compute integration include Google Cloud team lead Nelly Porter, Dr. Spyridon Bakas from Indiana University School of Medicine, Dr. Yury Velichko from Northwestern University, and Dr. Amber Simpson from the University of Alberta (formerly at Queen’s University).

The post MedPerf Meets Google Cloud Confidential Computing: Secure AI Benchmarking for Brain Tumor Research appeared first on MLCommons.

The MLCommons® Science working group is pleased to announce the availability of MLCommons Science GitHub, a series of 4 open source benchmarks. As a new addition to the MLCommons suite of benchmarks, these tools are aimed at domain scientists, machine learning experts, and students. The goal of MLCommons Science GitHub is to uncover and gather novel Machine Learning (ML) solutions to improve scientific discovery by improving accuracy on critical problems. Each benchmark includes open datasets, and reference models with algorithms and software. These rigorously documented benchmarks will also be used as tutorials and courses and run on new systems.

Join Us

We are calling on the international community to join us for an open call for participation to help increase model accuracy and/or extend uses with other datasets or scientific fields. The call for participation will have rolling submissions described in the overall policy and submissions documents. Participants should share their findings on the MLCommons Science Results GitHub. Ideal submissions will include a trained network that exceeds the target accuracy of one of the reference models for one of the benchmarks, together with a description of the improvements made.

Our Work

The MLCommons Science working group is part of the MLCommons research initiative which spans work in Algorithms, DataPerf, Dynabench, Medical, Science, and Storage. It collaborates with industry and research, with a shared mission to accelerate machine learning innovation to benefit society. The Science benchmarks differ from the flagship MLPerf benchmarks of MLCommons in terms of an emphasis on education and the impact on scientific discovery rather than system performance. New benchmarks are currently being considered. If you wish to join the open collaboration to discuss this work, contribute new challenges, or have any questions, please contact science@mlcommons.org or visit the science working group page. The working group meets every 2 weeks.

About MLCommons

MLCommons is an open engineering consortium with a mission to benefit society by accelerating innovation in machine learning. The foundation for MLCommons began with the MLPerf benchmark in 2018, which rapidly scaled as a set of industry metrics to measure machine learning performance and promote transparency of machine learning techniques. In collaboration with its 50+ founding partners – global technology providers, academics and researchers, MLCommons is focused on collaborative engineering work that builds tools for the entire machine learning industry through benchmarks and metrics, public datasets and best practices.

The post New ML Benchmarks for Scientific Discovery appeared first on MLCommons.