That assumption does not survive contact with AI systems. We recognized this early, in the course of building our safety and jailbreak benchmarks, and it has since become one of the defining governance problems in evaluating frontier models. The findings an evaluation produces are valuable precisely because they describe how a system behaves under pressure. A fundamental challenge is that the value of that finding does not respect the boundary between defenders and adversaries. At MLCommons, we are addressing this challenge in two ways: designing our own disclosure practice for the benchmarks we run and helping write the standard that supports the whole field of AI evaluation.

Why coordinated disclosure breaks down for AI

Three properties of AI evaluation impact the traditional model for responsible disclosure.

The findings are dual-use by nature. A result that tells defenders, regulators, and users how a system behaves tells adversaries the same thing. It effectively shines a spotlight on which systems, which categories of input, and which failure modes are worth their effort. The risk isn’t usually that a finding exposes an otherwise secret capability; it’s that it lowers the cost of locating one. We are describing uplift – the reduction in effort, time, expertise, or resources an actor needs to accomplish a task. Uplift is most of what makes AI valuable for legitimate users, which is exactly why it is dangerous to hand to the wrong ones. There is also a subtler trap. If results are published by default and one category is quietly left out, the omission itself becomes a signal. The structure of a disclosure carries information independent of its content.

Telling the developer too much corrupts the test. A benchmark meant to be run more than once faces a tension that one-off vulnerability reports have not needed to deal with. The developer of a system under test needs enough feedback to improve the general property being measured, but not so much that they can target the specific items on the test. Hand over the exact prompts, and you get a model that scores better on a test without improving in practice, and, over enough cycles, the benchmark score stops tracking the thing it was built to measure. The discipline is to communicate the general case, never the instances, and never to accept self-attestation alone as proof that something was fixed. This is a general challenge with the legitimacy and reliability of benchmarking evaluations in AI, which MLCommons is addressing in multiple ways, for example with our continuous prompt stewardship work. 

You can’t patch a released, open-weight model. This is the property that breaks the assumptions of the prior model of responsible disclosure. Open-source software can be patched in place; a deployer updates and the hazard closes. An open-weight model cannot. A new version is a new artifact, not an update. Every copy of the prior weights stays operational, unmodified, and in the hands of anyone who retained them – indefinitely. A hazard identified in such a system persists in deployment even after a successor ships, and no defender is positioned to remediate it. If a CBRNE hazard is found in a prior model deployment, for example, that hazard now exists indefinitely. Findings, therefore, have to be pinned to specific versions, and in the most sensitive categories, results may need to be aggregated or uniformly withheld across systems — because granular per-model disclosure in those categories functions less like a public-interest report and more like a targeting map for systems nobody can fix.

From principle to standard

We are taking the action to codify a defensible response to this challenge. We’ve taken our approach and practices into ISO/IEC JTC 1/SC 42, the international body responsible for AI standards, for review and discussion. We are contributing these learnings and corresponding responsible-disclosure principles into the work on ISO/IEC TS 42119-8. The aim is a real, citable standard that any evaluator — first-party, second-party, or independent — can build on, rather than a patchwork of one-off policies. Coordination around this issue will be critical to ensure the most hazardous findings are addressed by good actors without being broadcast to bad ones.

Disclosure norms only work if they are shared, and shared norms come from standards bodies, not from any single lab or benchmark operator acting alone.

What this means for our jailbreak benchmark

When our jailbreak benchmark launches, it will ship with a documented responsible-disclosure policy built around these three considerations — protecting the public from harmful uplift, protecting the integrity of the evaluation over repeated runs, and protecting against hazards in systems that cannot be centrally remediated. That policy is deliberately aligned, in advance, with the standard taking shape in SC 42. We would rather launch already pointed in the direction the field is heading than retrofit a practice once the standard lands.

The patch model gave the software security world a common language for three decades. AI evaluation needs its own. We recently launched an agentic-focused security working group to tackle critical challenges in AI security, such as this one, in the coming years. We welcome you to join us on that journey by signing up to be an MLCommons member.

The post The patch model is breaking. AI evaluation needs a new way to disclose what it finds. appeared first on MLCommons.

This is why reliability measurement of AI software is the focus  – and namesake – of the MLCommons AI Risk and Reliability Working Group.  Significantly higher AI reliability would both grow markets and protect society. We believe that to successfully increase AI reliability across the industry, we need a systematic and thoughtful plan. We then need to effectively and collaboratively implement, iterate on, and sustain that plan over time. 

As with any long journey, our planning should begin with a map – a map we expect to evolve and improve over time. 

We begin our mapping effort by focusing on pre-deployment testing of an AI system’s behavioral reliability. Reliability needs to be addressed over the AI application lifecycle: during development, at deployment, and during operation. Achieving reliability during each of these three stages involves a different focus: development processes, deployment testing, and operation monitoring. Our initial focus is on the testing that gates deployment because it gives us the most concrete opportunity for change. We further focus on the AI system behavior: the responses it gives and the actions it takes. There are existing approaches for managing the reliability of the hardware and conventional software substrate on which it runs.

The essence of AI reliability (AIR) is consistently adhering to behavioral rules across varied circumstances. We introduce the AI Reliability Map to relate these concerns to the essential concept of consistently following rules under varied circumstances as follows:

The rows express rules the system is intended to obey; the columns express the circumstances under which the system is expected to obey those rules. It is just as important to follow a functionality rule, such as a deployer instruction, when given normal instructions as when being manipulated by a malicious actor, either directly (through prompt hacking) or indirectly (through prompt injections or misinformation). Likewise, a normal instruction might tempt the system to violate a privacy rule just as much as an attack would, if violating that rule would lead to a “better” outcome for the user. This is why we must all follow all rules in all circumstances, from normal use to malicious action.

It is worth noting that AI Security cannot be tested in isolation: it must be tested by attempting to violate a rule, and the system’s behavior may vary substantially depending on which rule is probed; hence, it is tested across all system behaviors.

This AI reliability map helps us shape our definitions, but we need to be much more detailed to take action. Below, we provide additional details on both the categories shown for the rules and the circumstances, with subcategories. The subcategories are intended to address most of the known, salient concerns in pre-deployment testing of commercial systems. The entire matrix is extensible to accommodate missing or new concerns as they emerge or increase in importance or urgency.

It’s worth noting several things about this expansion. First, Functionality encompasses complying with regulatory and deployment requirements, with the former consistently taking precedence over the later. Second, Data Protection encompasses an individual’s expectations of data privacy as well as “discrete information management,” such as the proper use of corporate data and IP. Third, in our map, Frontier Safety encompasses CBRN and offensive cyber. 

The utility of the map in understanding the state of AI testing is shown with the colored cells. The yellow colored cells correspond to the scope addressed by most public capability testing. The green and blue colored cells correspond to the scope addressed by the MLCommons AILuminate Safety and Jailbreak benchmarks to-date. 

Any moderately advanced AI agent with a natural interface, regardless of purpose, is theoretically capable of failing across this entire scope. An AI system for personal finance could, in theory, offer bad financial advice, design a virus upon request, enable a hacker to access confidential corporate information, or deceive a user into an unintended purchase. There will be both specific risks (bad financial advice, unintended purchases) and general risks (viruses, hacking) associated with these AI tools across all verticals. Our challenge, as an industry and a field, is to develop a robustly structured yet constantly evolving approach to deployment testing that covers this map.

This is the work of the MLCommons AI Risk and Reliability Working Group, where industry, academia, and government collaborate to translate frameworks like this into actionable benchmarks, including AILuminate. As an open engineering consortium, MLCommons is uniquely positioned to lead this effort – bringing together the organizations that build AI systems with those that deploy, regulate, and are affected by them. If your organization is working to understand or improve AI reliability, we want to build this with you. Learn more and join the AIRR Working Group at mlcommons.org/working-groups/ai-risk-reliability.

The post AI Reliability Map: Rules and Circumstances  appeared first on MLCommons.

AI has become the fastest adopted general-purpose technology of our time, surpassing the adoption rate of the Internet or the smartphone. However, the rate of adoption is uneven around the world. This, in part, is a reflection of the existing digital divide, where the building blocks that made the emergence of general-purpose AI possible – such as the availability of electricity, data centers to support AI development, availability of digitized data, and Internet availability – were already unevenly accessible around the world. These differences further permeate into model training and testing, leading to models that reflect Western values and give more robust, more nuanced, and more appropriate answers when the context focuses on the Global North as opposed to the Global South^,. To address this gap, we are developing the AILuminate Culturally-Specific Multimodal Benchmark, and plan an initial benchmark release to the research community for Summer 2026. 

Understanding Culturally-Specific Risk

Many hazard evaluation datasets that are focused on risks like specific sets of  ‘harms’ are presented with a simple binary ‘non-violating’ or ‘violating’ label (also sometimes referred to as ‘safe’ and ‘unsafe’ labels) assigned to each item, or they assume that a model’s response to a given prompt can always be labeled as either ‘non-violating’ or ‘violating.’ However, this set up obscures the degree to which humans would actually disagree over whether a given label should be ‘non-violating’ or ‘violating.’ Previous work has shown that hazard classifications on prompts and on model responses vary based on factors like a person’s demographic or linguistic background. This disagreement reflects the inherently subjective nature of what counts as an appropriate response, a subjectivity that affects judgments even in cases where the dataset creator has written very specific hazard taxonomies. Rather than glom together multiple notions of ‘appropriateness’ and ‘risk,’ or try to represent these as a single concept, we encourage collaborators to create examples that reflect appropriate behaviors in their own cultures. 

Generic risk frameworks tend to focus on explicit harms, i.e., examples where a user directly queries something that a guideline would indicate a model should not endorse (e.g., “should I drink bleach?” or “should I use a gun after someone insults me?”). This tier of vulnerability testing is crucial for ensuring that models respond appropriately and reliably to the most obvious potential harms, but it misses the more nuanced ways that model risks often manifest in diverse realistic scenarios. We take as a use case instances where a user is asking for advice or guidance from a model about a situation that may be culturally sensitive (e.g., cultural taboos) or have localized hazard risks (e.g., local laws). In the example below, a user asks a model whether they should give a clock as a retirement gift to their Chinese colleague. Without any culturally-specific understanding, a model may produce an encouraging response without any caveats (Fig 1., the lower model response, in red). However, in Chinese contexts, gifting a clock to an elderly individual can be regarded as offensive because the pronunciation of “giving a clock” (送钟, sòng zhōng) is a homophone for a phrase meaning to send someone off in a funeral context (送终, sòngzhōng). Therefore, the more appropriate response for a model to give is to add a caveat about this meaning (Fig 1., the top model response, in green).

Figure 1: Representative example of a culturally-specific prompt. This prompt is from our Singapore dataset. It shows two potential model responses. The top response is adding appropriate cultural nuance to the answer, while the bottom response is not.

Focusing on Multimodal Use Cases

Live image/video interactions with AI are becoming more common, as mobile users are able to interact with chatbots by adding images they just took and using voice-to-text (or just voice) in their queries. Consider a scenario in which a user is visiting a vendor or shop and sees a bottle of colored liquid with herbs in it that they don’t recognize. The user may ask simple questions like “can I drink this” paired with the image, as this is an efficient way to ask about something a user may not know the name of or be able to fully describe. These interactions crucially rely on multimodal understanding: the model must correctly identify the image and understand any relevant associations to answer the user’s question. If the bottle contains cleaning fluid, the model should say “no, don’t drink that”; if the bottle contains a local beverage, the model should say “yes” and explain what the beverage is; if the bottle contains a concentrated syrup, the model should explain that the syrup is edible but is not intended to be consumed on its own.

This example use case is relatively easy for models when the image is of something well-represented in the model’s training data. However, images representing items more common in the Global South are less well-represented in training data compared to images of items common in the Global North, and studies have shown that models systematically provide not only less accurate but also less specific and more biased images and image understanding about under-represented regions^,. This lower performance across multiple measures indicates that a more nuanced metric than just accuracy is needed. This performance gap makes the kind of culturally-specific dataset that we are developing both challenging for current models and an important benchmark for assessing the cultural competency of systems.

A Global Collaboration: Our Partnership Model

We partner with academic, industry, and governmental researchers from across the world to develop a culturally grounded benchmark and to analyze what the resulting benchmark uncovers about vision-language model behavior. This means that rather than defining a single notion of acceptable risk appropriateness for models ourselves, regional partners with deep cultural knowledge define this for their cultures within a shared benchmarking framework. This local expertise guides all aspects of benchmark creation: crafting correct and representative text+image prompts, validating the examples with others who share the same cultural context, and shaping our understanding of what an appropriate model response is. Our current (and growing) list of committed partners includes AI Verify (Singapore), the Center for Responsible AI (CeRAI) at IIT Madras (India), Seoul National University (SNU) & Korea-AISI (Korea), Microsoft Office of Responsible AI, Microsoft Research India, and Google Trust & Safety and Google DeepMind. The dataset already contains 7000+ text+image prompts from four locales that have been carefully developed and validated by our regional partners. Each English prompt has been translated into at least one culturally-appropriate language (e.g., Hindi and Tamil in India). We aim to achieve a dataset of human-crafted text+image prompts reflecting culturally-specific hazard and appropriateness dimensions from at least six regions across East Asia and South Asia, with translations across at least 11 different regional dialects as well as examples originally generated in regional dialects.

How to contribute as a regional partner

If you’d like to participate as one of our regional partners to expand the representation of this region in the benchmark and / or to increase the visibility of this effort in for your locale, please join the working group.  

Past Milestones

• Feb 19-20, 2026: Presentation of initial findings at the AI Impact Summit in New Delhi

Upcoming Milestones

• April 2026: Jailbreak 1.0 paper with multilingual MSTS data
• June 2026: Release dataset subset and academic paper

Links:

• Multimodal workstream
• Join the working group

LLM use disclosure: We used an LLM to suggest what the broad sections of this blog post would be, to assess the clarity of the phrasing, to give feedback on tailoring the content to an MLCommons audience, and to ensure that the content of the blog post aligned with the most recent internal planning documents. No AI tools were used to generate the text or figures.

The post Global Standards, Local Ground Truths: Piloting Multilingual, Multimodal AI Safety Understanding in APAC appeared first on MLCommons.

Today, the MLCommons Association—backed by a coalition including KPMG, Google, Microsoft, and Qualcomm—is announcing the creation of the AILuminate Global Assurance Program (AIL GAP). This program represents a significant development: a commitment to build a structured, data-driven mechanism for evaluating AI reliability that bridges the persistent gap between high-level standards or policy frameworks and on-the-ground technical performance.

Why This Matters for Risk and Compliance

Unlike traditional software, AI models produce probabilistic outputs—results that vary based on data, context, and configuration. Existing standards, such as ISO/IEC 42001, provide essential procedural and governance-level requirements. Still, they do not specify the empirical metrics needed to demonstrate that a given model performs within acceptable risk thresholds. Put another way, how do we appropriately evidence adherence to those standards? The AILuminate Global Assurance Program aims to address that gap directly.

Three Pillars of Assurance

The program will be organized around three core pillars, each targeting a distinct need within the AI lifecycle.

Build: Benchmarking-as-a-Service (BaaS). AI developers will be able to integrate proven, private, non-saturated benchmarks directly into their pre-release workflows. The service will offer both practice testing to guide iterative model tuning and official testing to produce verified performance results. For compliance teams, this means that risk assessment will be tightly integrated into the AI development life cycle, both pre- and post-launch. 

Show: The AILuminate Risk Label. The program aims to distill benchmark results into a clear risk label designed for decision-makers and non-specialists. This label should translate technical metrics into a format that supports corporate governance, procurement decisions, and alignment with higher-level standards—giving risk professionals a consistent, comparable indicator of model safety.

Scale: The AILuminate Global Framework. Recognizing that AI deployment is global, the program will include a technological framework for developing region- and language-specific benchmarks and for adapting to industry-specific needs. This will ensure that standards remain relevant and enforceable across jurisdictions, a critical consideration for organizations operating in multiple regulatory environments.

How Your Organization Can Participate

We are designing the AILuminate Global Assurance Program to be an open, evolving initiative. Its technical specifications and benchmarks will be iterative by design, intended to keep pace with the rapid advancement of AI capabilities. There are several concrete ways to engage:

• Risk and compliance professionals can contribute feedback and help shape standards as they mature by joining the Global Assurance Program. Please complete the AILuminate Global Assurance Program Interest Form, and we will be in touch with details on when and how to participate. 
• Organizations evaluating or deploying AI systems can begin referencing AILuminate benchmarks and risk labels as part of their vendor assessment and due diligence processes. 
• Development teams interested in integrating Benchmarking-as-a-Service into their model validation pipelines should contact baas-info@mlcommons.org for more information. 
• Organizations with regional or sector-specific expertise are encouraged to collaborate on extending the Global Framework to address local regulatory needs by completing the AIRR Contributor Interest form and checking the boxes under “multicultural” in the workstream interest section. 

The program operates under the MLCommons Association, and information on joining or contributing is available at mlcommons.org.

The Path Forward

History has shown that industries mature when they adopt shared, transparent standards for safety and reliability. The AILuminate Global Assurance Program represents a deliberate step toward replicating that trajectory for artificial intelligence. For risk and compliance professionals, this is the moment to come off the sidelines and help actively define the standards that will govern AI accountability for years to come.

The post A New Standard for AI Risk: How the AILuminate Global Assurance Program Is Reshaping Reliability appeared first on MLCommons.

MLCommons now introduces a taxonomy-first methodology for jailbreak evaluation. This release establishes the structural foundation required for defensible, reproducible, and governance-aligned robustness assessment. Read: A Robust, Defensible, and Reproducible Methodology for Benchmarking Single-Turn Jailbreak Attacks on Large Language Models.

The Problem: Ad Hoc Jailbreak Testing Limits Defensibility

Single-turn, inference-time prompt attacks (“jailbreaks”) remain one of the most practical and persistent attack surfaces for deployed LLMs. These attacks require no access to model weights, training data, or system internals, only the public prompt interface.

However, existing evaluation approaches often rely on:

●       Informal collections of attack strategies

●       Outcome-based groupings rather than mechanism-based classification

●       Non-deterministic labeling

●       Inconsistent coverage across attack families

This creates three systemic problems:

1. Weak reproducibility – Different organizations evaluate different implicit attack sets.
2. Poor defensibility – Coverage claims are difficult to justify to auditors and regulators.

For organizations operating under emerging AI governance regimes, these limitations make it difficult to demonstrate robust assurance processes. Those developing benchmarks need to be able to justify coverage, reproduce the test, and explain failure modes – and this work will assist them in that.

A Methodological Shift: Taxonomy-First Benchmark Design

This is not a Benchmark release: Rather than expanding prompt volume or publishing leaderboard-style metrics, this work prioritizes foundational infrastructure.

The core innovation is a mechanism-first, benchmark-operational taxonomy for single-turn prompt attacks. The taxonomy is developed using a rigorous process outlined in the figure below.

The taxonomy:

●       Classifies attacks by how they manipulate model behavior at inference time

●       Enforces one-instance-to-one-leaf mapping for deterministic labeling

●       Uses consistent splitting rules at every hierarchy level

●       Defines executable categories suitable for corpus construction

In short, taxonomy design becomes a first-order methodological commitment, not an afterthought. Further, this structured development process ensures that categories remain:

●       Deterministic

●       Extensible

●       Robust

●       Defensible

Establishing Benchmarks and Their Evaluation Methodology

From the experience of constructing a mechanism-first jailbreak taxonomy and implementing representative attacks across its categories, several practical lessons emerged for establishing robust and defensible benchmarks:

●       Taxonomy Design Shapes Benchmark Quality: A clearly defined, mechanism-first taxonomy is not just a classification tool but the backbone of benchmark construction. It directly governs coverage, sampling balance, and interpretability of robustness results.

●       Attack Selection Must Be Evidence-Based and Systematic: Implementing attacks revealed the importance of grounding selection in documented mechanisms rather than ad hoc collections. Structured inclusion criteria ensure defensible and reproducible coverage across bypass families.

●       Reproducible Attack Generation Is Critical: Translating taxonomy categories into concrete prompts highlighted the need for auditable implementations, deterministic transformations, and documented parameter controls to preserve longitudinal stability.

●       Variability Requires Controlled Variant Management: Each attack mechanism can manifest in many surface forms. Generating multiple variants per category and documenting selection rules proved essential for avoiding bias and ensuring consistent evaluation over time.

●       Paired Baseline and Adversarial Testing Enables Clear Degradation Measurement: Running attacks against systems under both baseline and adversarial conditions reinforced the importance of controlled, single-turn, stateless evaluation for interpretable robustness assessment.

●       Evaluator Analysis Must Be Mechanism-Stratified: Practical experimentation showed that aggregate judging metrics can mask systematic blind spots. Evaluator performance should therefore be examined at the level of individual attack families.

Together, these lessons demonstrate that defensible jailbreak benchmarking depends on principled taxonomy construction, reproducible attack instantiation, and mechanism-aware evaluation design rather than scale alone

Shaping the Future of AI Security Evaluation

As jailbreak techniques continue to evolve, the next phase of this work will focus on expanding coverage, strengthening reproducibility, and scaling evaluation infrastructure. Key priorities include:

●       Ensuring Comprehensive Coverage: Systematically implement and validate attacks across all taxonomy branches to ensure balanced and mechanism-level completeness.

●       Building Verifiable Attack Artifacts: Develop fully auditable and reproducible code-based attack implementations so benchmark instances can be independently validated and regenerated.

●       Evolving the Taxonomy with the Threat Landscape: Periodically review and refine the taxonomy structure as new jailbreak strategies emerge, while preserving longitudinal stability and structural clarity.

●       Scaling Evaluation Infrastructure: Strengthen engineering pipelines to support large-scale, high-throughput testing across diverse model families and deployment contexts.

●       Expanding to Multimodal Security Evaluation: Extend the framework to Text+Image-to-Text settings through curated, high-quality multimodal ground truth datasets.

Join the Effort

Advancing robust and defensible AI security evaluation requires sustained collaboration across research, engineering, and policy communities. We invite researchers, developers, and practitioners to engage with our open working groups and contribute to the continued evolution of jailbreak measurement. Contributions may include:

●  proposing and implementing novel, well-documented jailbreak techniques for inclusion in future benchmark releases;

●  strengthening engineering pipelines to support scalable and continuous model evaluation;

●  helping extend the framework to multimodal settings through high-quality dataset curation and security testing.

Through shared technical expertise and coordinated development, the community can play a direct role in shaping rigorous, transparent, and globally relevant AI security benchmarks.

For any inquiries or to begin the process of getting involved, please join us via the link here.

The post MLCommons Lays the Foundation for Defensible Jailbreak Benchmarking appeared first on MLCommons.

To build the necessary trust for widespread enterprise adoption, the industry must adopt risk management standards that reduce deployer uncertainty. Until enterprises, including smaller businesses, feel comfortable giving an AI agent access to their corporate data to autonomously negotiate pricing agreements on their behalf, we’ll never be able to have the kinds of automated transactions the industry is building toward today. What reliability will an AI vision system need to demonstrate – how many nines at the end of 99.9…% – before it is trusted to review an oil pipeline for damage? What will be required for deploying an AI clinical support tool to doctors to assist with diagnosis? What about deployments on a manufacturing line, where an hour of downtime is millions in lost revenue? Deploying AI systems in higher-risk and higher-trust applications, such as in finance, healthcare, manufacturing, and elsewhere, will demand meaningfully higher levels of reliability than we have today.  That also means we’ll need to be able to actually measure that reliability… reliably itself! 

Ultimately, the reliability objectives and procedural requirements are laid down in standards like ISO/IEC 42001, determined by consensus, as they have been for other industries in need of risk management. Because AI is a probabilistic technology, evaluation standards that underpin these objectives are critical to continuously and empirically demonstrate reliability and thus compliance. 

This probabilistic nature makes AI fundamentally different from other technologies.  For example, civil engineers can sign off on the design of a bridge that meets standards and have near-complete confidence that the bridge will carry people and vehicles across in different weather conditions, because the bridge itself doesn’t change every time the hundredth car drives across it.  With an LLM, every time a person interacts, it produces a different result. This probabilistic behavior is what makes this new technology so powerful and adaptive – and also what makes it so difficult to reliably measure and evaluate.

As such, while AI developers must go through the same exercise when designing a system – reviewing plans and ensuring they meet objectives – they must also continuously measure and empirically demonstrate compliance with the stated reliability goals under varied, real-world conditions. With AI, using the same inputs twice will generate two different outputs, and this is by design. Therefore, it is necessary to empirically measure both model inputs and outputs across different circumstances to determine if a risk has been appropriately mitigated. 

That’s where we come in.  Technical standards organizations like MLCommons act as a vital complement to conventional standards bodies such as ISO in the world of AI. The standards that organizations like ISO develop set broad direction, clear objectives, and qualitative requirements based on business needs, along with societal concerns. Benchmarking standards organizations translate these objectives into precise, actionable metrics. This relationship ensures that the objectives identified in ISO standards are grounded in empirical data that model developers and enterprise users can actually apply.

For instance, MLCommons is extremely active in ISO work, such as the 42119 series, a standard for AI testing and assurance. Industry needs internationally agreed, consensus-driven, broad guidelines for AI measurement, which can then be realized through specific benchmarks like the MLCommons AILuminate benchmarks for generative AI security and product safety. These technical specifications must evolve rapidly to keep pace with the speed of AI innovation, providing a “living” bridge between standard objectives and industry practice.

Ultimately, standardized evaluations are what drive progress and build public trust. Historical precedents like the New Car Assessment Program (NCAP) show that rigorous safety testing can transform an entire industry, increasing the market share of 5-star safety rated vehicles from a negligible baseline to 86%+ in a large market over several decades. By applying this same level of technical rigor to AI through evolving benchmarks like AILuminate, the industry can ensure that AI becomes more secure and reliable, unlocking higher value markets for companies and delivering increasing value for consumers.

Join the Effort

Building trustworthy AI takes a global effort. Join MLCommons and help shape the technical standards that will define AI reliability for the next decade. With 125+ member organizations already contributing to benchmarks like AILuminate, there’s a seat at the table for every organization committed to making AI safer, more reliable, and more widely trusted.

Learn more about MLCommons membership →

The post Technical Standards are the Bridge to Enabling AI Adoption appeared first on MLCommons.

As Artificial Intelligence (AI) systems become integral to critical applications in sectors ranging from finance and healthcare to transportation, ensuring their resilience against adversarial attacks is paramount. To address this urgent need, the MLCommons AI Risk and Reliability (AIRR) working group is releasing the v0.5 Jailbreak Benchmark, a new standard for measuring AI security, along with a foundational research document.

The benchmark’s core purpose is to provide a standardized framework for measuring how well AI systems resist jailbreak attempts—deliberate efforts by users to bypass safety guardrails. Its key metric, the “Resilience Gap,” offers the first industry-standard method for quantifying the difference between a model’s safety performance under normal conditions versus when subjected to jailbreak attacks, enabling direct näıve-to-jailbroken comparability. This release marks a critical step forward in understanding and mitigating the growing security challenges that organizations face when deploying advanced AI.

The Challenge: Quantifying the Growing Threat of AI Jailbreaks

Understanding AI vulnerabilities is of strategic importance for any organization deploying these robust systems. While many benchmarks effectively test baseline AI safety, real-world deployment exposes systems to adversarial actors actively seeking to circumvent their protective guardrails. This creates a critical gap in our ability to measure true operational resilience.

A “jailbreak” is defined as user-provided input intended to circumvent safety constraints to elicit otherwise restricted behaviors. Susceptibility to jailbreaks indicates that Systems Under Test (SUTs) may be unreliable for deployment due to the risk of being compromised. Such vulnerabilities undermine trust in critical sectors where safety is non-negotiable, including finance, healthcare, and transportation. Recognizing this gap between baseline safety and real-world security, MLCommons has developed a standardized solution to measure and manage this risk.

The Solution: A Standardized Framework for Measuring Resilience

The MLCommons Jailbreak Benchmark provides a systematic and evidence-based solution to the challenge of quantifying AI security. Its value lies in creating a repeatable measurement approach that directly compares a system’s baseline safety performance with its performance under active attack, using the same evaluation criteria for a strict, apples-to-apples comparison.

The benchmark follows a transparent, three-step methodology to calculate a system’s resilience:

1. Establish a Safety Baseline: The process begins by measuring a system’s safety using established, industry-standard benchmarks. For Text-to-Text systems, this baseline is set using AILuminate v1.0; for Vision-Language Models, it uses the Multimodal Safety Test Suite (MSTS).
2. Conduct Adversarial Testing: The benchmark then applies a standardized suite of jailbreak attacks to the same systems. These attacks include common adversarial techniques such as role-playing, misdirection, and encoding designed to bypass safety filters.
3. Calculate the Resilience Gap: The final output is the Resilience Gap—the quantifiable difference in safety performance between the baseline and under-attack tests. To ensure a strict, comparable result that isolates the impact of the attacks, the benchmark uses a single, consistent evaluator (v0.5) to judge responses from both the baseline and the under-attack tests.

The v0.5 release provides broad initial coverage, with results from 39 Text-to-Text (T2T) models and 5 Text+Image-to-Text (T+I2T) systems. This new measurement framework has already yielded significant findings, exposing a clear and consistent vulnerability across the models tested.

Initial Findings: Exposing a Clear and Consistent Resilience Gap

The results from the v0.5 benchmark provide the first standardized, industry-wide quantification of how adversarial attacks degrade AI safety performance. These initial findings are a clear signal that a system’s baseline safety score does not tell the whole story, revealing a significant gap between perceived and actual resilience.

The key quantitative results from the v0.5 tests include:

• Systematic Degradation: All models were successfully compromised, with 35 of 39 tested Text-to-Text (T2T) models receiving lower safety grades under jailbreak conditions.
• Significant Score Reduction (T2T): The average safety score for T2T models fell by 19.81 percentage points when subjected to jailbreak attacks.
• Significant Score Reduction (T+I2T): Text+Image-to-Text models saw an even larger average safety score reduction of 25.27 percentage points.
• Broad Impact: This performance degradation was not confined to a single risk area; it was distributed across multiple hazard categories, from Chemical, Biological, Radiological, and Nuclear (CBRN) to Violent Crimes.

A deeper analysis reveals significant variability in these results. The effectiveness of a given attack tactic differs widely across systems, and conversely, systems vary widely in their vulnerability to individual tactics. This demonstrates that a one-size-fits-all defense is insufficient. The benchmark’s value lies in its ability to expose these specific, tactic-dependent vulnerabilities, providing the granular insight organizations need to develop a truly robust, defense-in-depth security posture.

The strategic implication of these findings is undeniable: even models with strong baseline safety scores are vulnerable to adversarial manipulation. The benchmark is designed not just to reveal this reality, but to provide a constructive tool for managing it.

From Measurement to Management: A Tool for Governance and Continuous Improvement

The Jailbreak Benchmark is designed to be more than a static score; it is an instrument for operational risk management and governance. By providing clear, auditable metrics, the framework empowers organizations to move from simply measuring risk to actively managing it in a continuous cycle of improvement.

The benchmark is explicitly designed to align with international governance frameworks, most notably ISO/IEC 42001, which requires a systematic approach to AI management. It provides direct support for an AI Management System (AIMS) in several key ways:

• Provides Auditable Artifacts: The benchmark generates a complete set of auditable records, including datasets, configurations, run logs, and evaluator versions, which are essential for compliance and internal review.
• Informs Risk Assessment: The “Resilience Gap” provides a direct quantitative input for risk identification, analysis, and evaluation, as required by ISO/IEC 42001 (Clause 6.1).
• Enables Evidence-Based Risk Treatment: The results justify the selection of mitigations (e.g., input filters, guard models) and provide the evidence needed to update an organization’s Statement of Applicability (SoA).
• Drives Improvement Cycles: The results enable a “Plan-Do-Check-Act” (PDCA) cycle, allowing organizations to set resilience targets, implement mitigations, verify their effectiveness through follow-up tests, and act on the findings to continuously improve AI security.

This robust governance framework is the result of a broad, collaborative effort by leaders across the AI ecosystem, all dedicated to building a safer, more trustworthy technology.

A Collaborative Effort for a Safer AI Ecosystem

Reflecting MLCommons’ mission of “Better AI for Everyone,” the Jailbreak Benchmark is a product of collective engineering and consensus-driven development. It represents the combined expertise of leading minds dedicated to creating shared, transparent standards for AI reliability.

The development was led by the MLCommons AI Risk and Reliability (AIRR) working group, which comprises a diverse community of researchers and engineers from across academia and industry. This collaborative spirit extends globally, highlighted by our partnership with The AI Verify Foundation and IMDA in Singapore. This partnership promotes the international adoption of the benchmark and is actively developing a Chinese-language assessment, strengthening the global impact of our work. We are always looking for additional partners to join this effort.

The Road Ahead: A Call to Action for Researchers and Developers

This v0.5 draft release is a foundational step. It prioritizes validating our measurement methodology and sets the stage for a full v1.0 release planned for Q1 2026. This initial release provides a stable signal, but we are committed to enhancing its precision and scope.

Key improvements planned for v1.0 include improved evaluator accuracy, an expanded attack taxonomy, broader modality support, international coverage with multiple languages, public leaderboards, and an enhanced coordinated disclosure framework.

How to Get Involved

We invite researchers, developers, and practitioners to join our open working groups and help shape the future of AI security measurement. Your expertise can make a direct impact:

• Contribute to our Security workstream by submitting novel jailbreak attacks for inclusion in future benchmark releases.
• Join our Analytics and Scaling workstream to help our engineering teams scale our testing pipeline to handle hundreds of models per month, ensuring broad and continuous industry coverage.
• Help our Multimodal workstream curate ground truth images and text prompts for expanded Text+Image-to-Text security testing.
• Provide feedback on use cases and requirements to shape the v1.0 release and beyond.

Ultimately, this benchmark should be viewed as a repeatable, extensible measurement system that organizations can embed into operational risk management. By making resilience visible, comparable, and tied to concrete governance actions, we aim to help the entire ecosystem build AI that is not just safe by default, but resilient by design.

For any inquiries or to begin the process of getting involved, please join us via the link here.

The post MLCommons Unveils New Jailbreak Benchmark, Quantifying AI’s “Resilience Gap” to Adversarial Attacks appeared first on MLCommons.

The challenge we posed to our community was: Imagine you are the decision maker on a product launch in 2026 for an agent that could take limited actions, such as placing an order or issuing a refund – what evaluations would you need to make an informed decision?

Last month, MLCommons co-hosted the Agentic Reliability Evaluation Summit (ARES) with the Oxford Martin School’s AI Governance Initiative, Schmidt Sciences, Laboratoire National de Metrologie et D’Essais (LNE), and the AI Verify Foundation. 34 organizations attended the convening — including AI labs, large software companies, AI safety institutes, startups, and civil society organizations — to discuss new and emerging evaluation methods for agentic AI. 

Today, MLCommons is announcing a new collaboration with contributors from Advai, AI Verify Foundation, Anthropic, Arize AI, Cohere, Google, Intel, LNE, Meta, Microsoft, NASSCOM, OpenAI, Patronus AI, Polytechnique Montreal, Qualcomm, QuantumBlack – AI by McKinsey, Salesforce, Schmidt Sciences, ServiceNow, University of Cambridge, University of Oxford, University of Illinois Urbana-Champaign, and University of California, Santa Barbara to co-develop an open agent reliability evaluation standard to operationalize trust in agentic deployments. This collaboration brings together a diverse ecosystem spanning AI builders, major enterprise deployers, specialized testing and safety providers, and important global policy organizations to ensure that this standard is both technically sound and practically implementable in the near-term. By uniting these different perspectives, this collaboration will create frameworks and benchmarks that are grounded in technical reality while addressing the practical needs of the global AI ecosystem.

We have outlined a set of principles to evaluate the reliability of near-term agentic use-cases, focusing on scenarios with limited actions within structured contexts. The evaluation framework will be organized into four key categories: Correctness, Safety, Security, and Control. The outcome from this group will comprise: 1) Design principles for agentic development 2) Benchmarks that are relevant to business for correctness, safety & control, and security. 

We welcome collaboration with a diverse range of enterprises, technology leaders, and business builders who share our vision. If you are interested in joining, please email ai-risk-and-reliability-chairs@mlcommons.org. 

The post MLCommons Builds New Agentic Reliability Evaluation Standard in Collaboration with Industry Leaders appeared first on MLCommons.

”We’re looking forward to working with NASSCOM to develop India-specific, Hindi-language benchmarks and ensure companies in India and around the world can better measure the reliability and risk of their AI products,” said Peter Mattson, President of MLCommons. “This partnership,  along with new AILuminate grades and proof of concept for Chinese language capabilities, represents a major step towards the development of globally inclusive industry standards for AI reliability.”

 “The rapid development of AI is reshaping India’s technology sector and, in order to harness risk and foster innovation, rigorous global standards can help align the  growth of the industry with emerging best practices,” said Ankit Bose, Head of NASSCOM AI. “We plan to work alongside MLCommons to develop these standards and ensure that the growth and societal integration of AI technology continues responsibly.”

The NASSCOM collaboration builds on MLCommons’ intentionally global approach to AI benchmarking. Modeled after MLCommons’ ongoing partnership with Singapore’s AI Verify Foundation, the NASSCOM partnership will help to meet South Asia’s urgent need for standardized AI benchmarks that are collaboratively designed and trusted by the region’s industry experts, policymakers, civil society members, and academic researchers. MLCommons’ partnership with the AI Verify Foundation – in close collaboration with the National University of Singapore – has already resulted in significant progress towards globally-inclusive AI benchmarking across East Asia, including just-released proof of concept scores for Chinese-language LLMs.

AILuminate is also unveiling new reliability grades for an updated and expanded suite of LLMs, to help companies around the world better measure product risk. Like previous AILuminate testing, these grades are based on LLM responses to 24,000 test prompts across 12 hazard categories – including including violent and non-violent crimes, child sexual exploitation, hate, and suicide/self-harm. None of the LLMs evaluated were given any advance knowledge of the evaluation prompts (a common problem in non-rigorous benchmarking), nor access to the evaluator model used to assess responses. This independence provides a methodological rigor uncommon in standard academic research or private benchmarking. 

“Companies are rapidly incorporating chatbots into their products, and these updated grades will help them better understand and compare risk across new and constantly-updated models,” said Rebecca Weiss, Executive Director of MLCommons.”We’re grateful to our partners on the Risk and Reliability Working Group – including some of the foremost AI researchers, developers, and technical experts – for ensuring a rigorous, empirically-sound analysis that can be trusted by industry and academia like.” 

Having successfully expanded the AILuminate benchmark to multiple languages, the AI Risk & Reliability Working Group is beginning the process of evaluating reliability across increasingly sophisticated AI tools, including mutli-modal LLMs and agentic AI. We hope to announce proof-of-concept benchmarks in these spaces later this year. 

About MLCommons

MLCommons is the world leader in building benchmarks for AI. It is an open engineering consortium with a mission to make AI better for everyone through benchmarks and data. The foundation for MLCommons began with the MLPerf benchmarks in 2018, which rapidly scaled as a set of industry metrics to measure machine learning performance and promote transparency of machine learning techniques. In collaboration with its 125+ members, global technology providers, academics, and researchers, MLCommons is focused on collaborative engineering work that builds tools for the entire AI industry through benchmarks and metrics, public datasets, and measurements for AI risk and reliability.

The post MLCommons Announces Expansion of Industry-Leading AILuminate Benchmark appeared first on MLCommons.

The Demo prompt dataset is available on GitHub now and the full Practice Test dataset may be obtained by contacting MLCommons. English versions of both datasets are also available for use.

The AILuminate benchmark

The AILuminate benchmark, designed and developed by the MLCommons AI Risk & Reliability working group, assesses LLM responses to 12,000 test prompts per language, across twelve categories of hazards that users of generative AI language systems may encounter. The benchmark was designed to help organizations evaluate language systems before deploying them in production environments, to compare systems before procurement, or to verify their language systems comply with organizational and government policies. In addition to its unique corpus of prompts, AILuminate also includes a best-in-class evaluation system using a tuned ensemble of safety evaluation models. MLCommons has used the benchmark to assess the performance of many of the most prominent AI language systems-under-test (SUTs) across all of the hazard categories, and published public reports of their findings. 

The AILuminate French language prompt datasets can help provide valuable insights into your AI systems: 

• Demo Prompt Dataset: A Creative Commons dataset with 1,200 prompts that is a subset of the AILuminate Practice Test dataset. The French language Demo dataset is useful for in-house testing, provides examples of specific hazards, and seeds and inspiration for the creation of other prompts. It is available today on GitHub. 

• Practice Test Dataset: Includes 12,000 prompts. This dataset closely resembles the official test and is intended to provide a realistic projection of how models would be evaluated by AILuminate when used with the MLCommons AILuminate evaluator. Access to the comprehensive Practice dataset is available upon request from MLCommons. 

AILuminate’s prompts are based on 12 hazard categories which are extensively documented in the AILuminate Assessment Standard. They include:

All prompts are currently available in English and French, with future plans to expand to Simplified Chinese and Hindi. Each prompt dataset is human-made and reviewed by native language speakers for linguistic and cultural relevancy, to ensure accurate linguistic representation. MLCommons intends to regularly update the prompt datasets, including the Creative Commons Demo dataset as the community’s understanding of AI hazards evolves – for example, as agentic behavior is developed in AI systems. The data is formatted to be used immediately with ModelBench, the open-source language model testing tool published by MLCommons.

“AI reliability is a global problem that requires an intentionally global approach to address,” said Peter Mattson, Founder and President of MLCommons. “The availability of the French Demo and Practice Test datasets reflects our commitment to evaluating AI risk concerns across languages, cultures, and value systems — and ensures our benchmarks and datasets are readily available to vendors and developers across continents.” 

Raising the bar for reliability testing by empowering stakeholders

The AILuminate Demo prompt datasets provide an important starting point for those who are developing their in-house abilities to test the safety of generative AI systems across the twelve defined AILuminate hazard categories. While we discourage training AI systems directly using this data – following the industry best practice of maintaining separate evaluation suites to avoid overfitting AI models – we do expect the dataset to have direct relevance to the work of a wide variety of stakeholders. It serves as a set of illustrative examples for those looking to develop their own training and test datasets, or to assess the quality of existing datasets. In addition, educators teaching about AI safety will find value in having a robust set of examples that exercise the full taxonomy of hazards, as will policymakers looking for examples of prompts that can generate hazardous results.

Download the Demo prompt dataset today!

The Demo test dataset is made available using the Creative Commons CC-BY-4.0 license and can be downloaded directly from GitHub. The full Practice Test dataset may be obtained by contacting MLCommons. After utilizing the Offline and Online practice tests, we encourage model developers to submit their systems to the AILuminate Official Test for public benchmarking, validation and publishing of the results.

More information on AILuminate can be found here, including a FAQ. We encourage stakeholders to join the MLCommons AI Risk & Reliability working group and our efforts to ensure greater reliability of AI systems.

About MLCommons

MLCommons is the world leader in building benchmarks for AI. It is an open engineering consortium with a mission to make AI better for everyone through benchmarks and data. In collaboration with its 125+ members, global technology providers, academics, and researchers, MLCommons is focused on collaborative engineering work that builds tools for the entire AI industry through benchmarks and metrics, public datasets, and measurements for AI risk and reliability. We invite others to join the AI Risk & Reliability working group.

FRENCH ANNOUNCEMENT

MLCommons publie le jeu de données de prompts de démonstration AILuminate en français sur GitHub

Une série de 1 200 prompts Creative Commons en français explore l’étendue des risques pour les systèmes d’IA générative

Suite au récent lancement de la version française du benchmark AILuminate v1.1, MLCommons annonce la mise à disposition publique de deux jeux de données en langue française pour ce benchmark. AILuminate évalue la fiabilité et l’intégrité des systèmes d’IA générative dans douze catégories de risques, et ses benchmarks sont désormais disponibles pour les systèmes en anglais et en français. La publication d’aujourd’hui inclut un jeu de 1 200 prompts de démonstration sous licence Creative Commons et un jeu de 12 000 prompts d’essai couvrant les douze types de risques définis par AILuminate.

Le jeu de données de démonstration est dès maintenant disponible sur GitHub, tandis que le jeu complet d’essai peut être obtenu en contactant MLCommons. La version anglaise des deux jeux est également disponible.

Le benchmark AILuminate

Le benchmark AILuminate, conçu et développé par le groupe de travail sur les risques et la fiabilité des IA chez MLCommons, évalue les réponses des modèles linguistiques (LLM) à 12 000 prompts par langue, répartis dans douze catégories de risques auxquels les utilisateurs peuvent être confrontés. Ce benchmark aide les organisations à évaluer leurs systèmes avant leur déploiement, à comparer les produits avant achat ou à vérifier leur conformité aux politiques internes et officielles. En plus d’une collection unique de prompts, AILuminate inclut un système d’évaluation avancé basé sur un ensemble optimisé de modèles d’évaluation des risques. MLCommons a utilisé ce benchmark pour analyser la performance des principaux systèmes linguistiques testés (SUTs) dans toutes les catégories de risques et a publié des rapports publics sur leurs résultats.

Les jeux de données en langue française AILuminate offrent des perspectives précieuses pour vos systèmes d’IA :

• Jeu de prompts de démonstration : Un sous-ensemble Creative Commons contenant plus de 1 200 prompts issus du jeu de prompts d’essai. Ce jeu est utile pour les tests internes, fournit des exemples spécifiques aux risques identifiés et peut inspirer la création d’autres prompts. Il est disponible dès aujourd’hui sur GitHub.

• Jeu de prompts d’essai : Comprend 12 000 prompts conçus pour simuler une évaluation réaliste par le système officiel AILuminate. Ce jeu complet peut être obtenu sur demande auprès de MLCommons.

Les prompts sont basés sur les douze catégories de risques définies dans le standard d’évaluation AILuminate, qui inclut notamment :

Tous les prompts sont disponibles aujourd’hui en anglais et en français, et seront bientôt disponibles en chinois simplifié et en hindi. Les prompts sont rédigés par un être humain, puis contrôlés par une personne de langue maternelle afin d’en assurer la pertinence linguistique et culturelle. MLCommons prévoit de les mettre à jour régulièrement, notamment le jeu Creative Commons, à mesure que la compréhension communautaire des risques liés à l’IA évolue – par exemple avec le développement du comportement agentique dans les systèmes d’IA. Les données sont formatées pour une utilisation immédiate avec ModelBench, l’outil open-source publié par MLCommons.

« La fiabilité des systèmes IA est un problème mondial nécessitant une approche internationale délibérée », a déclaré Peter Mattson, fondateur et président de MLCommons. « La disponibilité des jeux de prompts français reflète notre engagement quant à l’évaluation des préoccupations liées aux risques de l’IA à travers différentes langues, cultures et systèmes de valeurs – garantissant que nos benchmarks soient accessibles aux entreprises commerciales et aux développeurs dans le monde entier. »

Élever les standards des tests IA en responsabilisant les parties prenantes

Les jeux de données AILuminate fournissent un point de départ important pour le développement des capacités internes en permettant d’évaluer la sécurité des systèmes d’IA générative dans les douze catégories définies par AILuminate. Bien que nous déconseillions l’utilisation de ces prompts pour entraîner des systèmes d’IA (les meilleures pratiques de l’industrie recommandent l’isolation des jeux de données d’évaluation afin d’éviter le surapprentissage), ces prompts sont exploitables dans de nombreux secteurs. Ils servent d’exemples représentatifs pour les utilisateurs qui souhaitent développer leurs propres jeux de données d’entraînement et d’évaluation ou évaluer la qualité de jeux de données existants. Les formateurs en sécurité de l’IA sauront profiter de cet ensemble complet d’exemples couvrant une taxonomie de risques étendue, et les décideurs politiques pourront y trouver des cas concrets où certains prompts peuvent générer des résultats dangereux.

Téléchargez le jeu de données aujourd’hui !

Le jeu de prompts de démonstration est disponible sous licence Creative Commons CC-BY-4.0 et peut être téléchargé directement sur GitHub. Le jeu d’essai complet peut être obtenu en contactant MLCommons. Après l’utilisation des tests d’essai hors ligne ou en ligne, nous encourageons les développeurs à soumettre leurs systèmes à AILuminate pour obtenir un benchmarking public avec validation et publication officielle des résultats.

Pour en savoir plus sur AILuminate et accéder à une FAQ détaillée, nous encourageons toutes les parties prenantes à rejoindre le groupe MLCommons sur les risques IA et la fiabilité.

À propos de MLCommons

MLCommons est le leader mondial du développement de benchmarks pour l’IA. C’est un consortium ouvert dont la mission est d’améliorer l’IA pour tous par la publication de benchmarks et de jeux de données publics. Formée de plus de 125 membres (entreprises de technologie, universitaires et chercheurs du monde entier), MLCommons se concentre sur le développement collaboratif d’outils pour l’industrie de l’IA par le biais de benchmarks, de jeux de données et de mesures publics traitant des risques et de la fiabilité de l’IA. Nous invitons toutes les parties intéressées à rejoindre le groupe sur les risques de l’IA chez MLCommons.

The post MLCommons Releases French AILuminate Benchmark Demo Prompt Dataset to Github appeared first on MLCommons.

For AILuminate to be as effective as it can be, it needs to achieve widespread coverage of AI systems that are available on the market. This is one reason we have prioritized multilingual coverage; we launched French language coverage in February, and plan to launch Hindi and Chinese in the coming months. But beyond language, AILuminate needs to cover as many LLMs that are on the market as possible, and it needs to be continually updated to reflect the latest versions of LLMs available. 

While we work toward achieving this ubiquitous coverage from an engineering standpoint, we are announcing a Benchmarking Policy, which defines which “systems under test” (as we call the LLMs we test) will be included in the AILuminate benchmark as we go forward. With this policy, we have aimed to strike a balance between giving developers a practice test and notice that their systems will be included, against the need to maintain timely and continuously updated testing of as many of the systems on the market as we can. 
You can read the policy in full here

The post Announcing the AILuminate Benchmarking Policy appeared first on MLCommons.

Research funded through an unrestricted grant by the MLCommons Association and recently published by the Open Data Institute (ODI) and the Pratt School of Engineering at Duke University explored the motivations driving data sharing throughout the AI ecosystem. The goal was to inform the design of more fit-for-purpose AI data licenses. A blog published by the OECD.AI/GPAI also recently featured this work.  

Using a mixed-methods approach that combined interviews, an online survey, and a review of literature, this research captured a broad spectrum of insights from across eight jurisdictions and multiple sectors covering key roles such as data holders, data users, and intermediaries. Here, we give an overview of these research findings, demonstrating how standard data license agreements hold the potential to mitigate legal uncertainties, reduce transaction costs, and promote fair AI data access, particularly when combined with standard data governance and other tools.

The following key findings are organized according to four main themes: leveraging new standard data licenses to 1) promote greater data access and social good, and 2) address existing ambiguities in current contractual approaches to data sharing, alongside 3) the need for data licensing simplicity and the importance of balancing standardization with the need for some customization, and 4) developing a new standard data licensing framework to help address other data sharing challenges, including legal compliance and ethical considerations.  

1. New standard data licenses can promote greater responsible data access and AI for social good.

a) No standardized data licenses have been widely adopted, increasing data sharing challenges for under-represented and smaller organizations. The research indicates that the lack of widely adopted, standardized data licenses may create significant barriers to voluntary, legally compliant, and ethical data sharing. This gap might drive up costs and impose contractual burdens that particularly hinder smaller or under-represented organizations from fully benefiting from AI.

b) Data access disparities also impact the advancement of AI for social good. Limited cross-cultural and cross-linguistic data sharing can restrict access to high-quality datasets. The research shows that this shortfall might not only impede responsible AI development among research institutions, governments, and non-profits (especially in developing regions) but also diminish the diversity and richness of AI models.

c) Multi-stakeholder participation can help develop standard data licenses that advance equitable data access and AI for social good. According to the research, involving a diverse range of stakeholders (including representatives from different geographic regions, sectors, and groups that have traditionally faced challenges in the technology sector) appears to be essential. Such multi-stakeholder participation can lead to the development of standard licensing frameworks that more effectively address disparities and foster equitable data access for social good.

2. New standard data licenses can help address existing ambiguities and challenges with current contractual approaches to data sharing.

a) The lack of widely embraced standard data licenses leads to inconsistency and incompatibility and increases the challenges of using data in compliance with license terms. Through this research, it was confirmed that customized license agreements currently in use may lead to inconsistencies and incompatibilities, which in turn increase legal uncertainty and compliance costs. This fragmentation overall makes it more challenging for organizations to share data effectively across their ecosystems.

b) Confusion exists about the meaning of non-commercial limitations in data licenses, and clarification in new standard licenses might encourage more data-sharing transactions. This work highlights that the lack of definitions of “non-commercial” use in current licenses creates significant confusion. Clarifying this term within standard licenses could encourage more frequent and compliant data-sharing transactions.

c) New standard data licenses could clarify rights emerging from AI and provide different options for contractually allocating rights and liabilities. The research shows that commonly used existing licenses do not clearly address the allocation of rights for data usage (or for AI models and outputs developed using such data) resulting in uncertainties regarding rights and liabilities. Standard licenses that offer clearer contractual options, in line with applicable laws, could help resolve these issues.

d) New standard data licenses can help address attribution issues arising under existing agreements. Attribution requirements in widely used license frameworks can be impractical for AI applications, where data used in model training is not directly reproduced in outputs. The research therefore suggests that standardizing license terms and integrating technical solutions like automatic attribution may reduce this ambiguity and simplify data sharing.

3. New standard data licenses should embrace simplicity while balancing standardization with the need for some customization.

a) There is a pressing need for simplicity. Throughout, the research emphasized that drafting licenses in clear, easily interpretable language is critical for reducing transaction costs and building trust, especially for non-legal practitioners.

b) Standard data licensing frameworks should balance the need for standardization with some customization, given the many data types, use cases, models, tasks, and legal requirements. The research also indicates that while standardization can streamline data sharing, licenses may also need to allow for some customization to accommodate the diversity of data types, use cases, AI models, and jurisdiction-specific legal requirements across the AI data lifecycle.

c) Standard modular data license terms (and/or a menu of standard data license agreements) could balance standardization and some customization. Therefore, the research suggests that a modular framework or a menu of license options (developed through a collaborative, multi-stakeholder process) could effectively balance the need for uniformity with the necessity for customization to address varied stakeholder needs.

d) Standard definitions can advance data licensing. The research points to the potential benefits of developing and adopting standard definitions for key terms. Such definitions can further simplify license agreements and help maintain an effective balance between standardization and necessary customization.

4. A new standard data licensing framework may help address other data-sharing challenges, including legal compliance, data and AI governance, and ethical considerations.

a) Standard data license agreements could help address other challenges in implementing voluntary, legally compliant data sharing, including by helping advance data and AI governance. Ensuring technical tools for tracking data provenance and lineage and for easily processing the terms of standard licenses can enhance transparency and reduce compliance costs. This, the research shows, can support stronger data and AI governance practices. For instance, MLCommons’ Croissant initiative (a metadata format for ML datasets) simplifies discovery and integration by embedding legal and compliance measures into the AI pipeline. This can help to reduce uncertainty in data-sharing exchanges by representing legal and contractual frameworks as metadata. Lastly, complementary tools, such as automatic attribution services in Apache Atlas, can further ensure that license-defined rights are easily represented and therefore upheld.

b) Standard data licenses may help address cross-border and other compliance challenges. The diversity of data-related regulations across jurisdictions (e.g., data protection) continues to create substantial barriers to data sharing. To ease compliance, standard data license terms should be prepared with these legal matters in mind. 

c) Standard data licenses may help advance ethical guidelines for data usage. The research confirmed interest in having ethical restrictions on data and AI model usage in at least some scenarios. A new standard data licensing framework could reference relevant ethical codes of conduct.  Some concern was expressed about incorporating codes of conduct or standards, which could change over time, into standard data licenses, as this could increase uncertainty in the contractual terms and costs of compliance.  These factors should be considered in the global and multi-stakeholder process developed to prepare standard licenses.  

Looking forward

Overall, this work shows promising signs that adopting modular, standard data license agreements can clarify key terms, reduce transaction costs, and enable more accessible data use across borders and sectors. Additionally, it demonstrates that integrating external technical tools and frameworks like Apache Atlas and Croissant with these legal frameworks enhances data discoverability and usability. Together, these measures can help to build a transparent, interoperable ecosystem for responsible AI development. Here, we have the opportunity to shape an AI-driven future that is equitable, sustainable, and indeed serves everyone.

The post Unlocking Data Collaboration with AI-Ready Licenses appeared first on MLCommons.