AILuminate is a family of independent, consensus-governed benchmarks for AI risk and reliability. AILuminate Safety measures how often an AI system gives hazardous responses to direct requests; AILuminate Jailbreak measures how well an AI system’s safeguards hold when those requests are rewritten as attacks. Both apply the AILuminate Assessment Standard, which defines what counts as a violating response.

Working groups in the MLCommons AI Risk & Reliability Program develop the benchmarks, and MLCommons, a nonprofit engineering consortium founded in 2018, runs them. People from more than 80 organizations have contributed to the Program’s working groups, including AI developers that compete with one another, universities, national laboratories, and nonprofit research groups. Join a working group →

By making sure no single developer controls either the rules or the results. Developers that compete with one another agree on the rules alongside researchers and civil society groups, and MLCommons publishes those rules. The measurement then stays at arm’s length: the official prompts are confidential, an evaluator trained by MLCommons judges every response, and the same published rules govern how every system is selected, notified, and reported.

No. A score is a measurement: it places a system on a fixed scale and describes its behavior on one benchmark, within that benchmark’s scope. MLCommons does not certify AI systems, audit organizations, or set a level of acceptable risk. Buyers, assurance providers, and regulators make those decisions, often in relation to reliability considerations that are relevant to particular use cases, and AILuminate results give them evidence to work from.

No. In the words of the Assessment Standard, “AILuminate does not assess the legality of any content.” Hazard names carry their everyday meaning: defamation, for example, means injury to someone’s reputation, not a legal finding.

Through versioned releases and benchmark stewardship. MLCommons plans new benchmark versions roughly every 90 to 180 days, and every result carries the labels of the version that produced it. Prompts are selected according to how well they separate stronger systems from weaker ones, and in later releases the same measurements will decide which prompts are retired and replaced, so that the benchmarks remain challenging enough to tell the most capable systems apart. Read about benchmark stewardship →

They apply the same rules under two conditions. Safety sends hazardous requests as written, and Jailbreak rewrites requests of the same kinds with known attack techniques. Since Jailbreak 1.1, both benchmarks use the same evaluator and the same approach to sampling prompts, so a system’s two scores describe its behavior under the same rules in two conditions. The two scores are not directly comparable, because Jailbreak’s scoring is adapted for attack prompts.

Not yet. The AILuminate Safety and Jailbreak benchmarks test single-turn, text-to-text exchanges: one prompt and one response. Working groups in the AI Risk & Reliability Program are developing evaluations for AI agents, more languages and cultures, and prompts that combine text and images. AI Risk & Reliability working group pages

A Safety score places a system on a fixed scale from 0 to 100, calibrated against a set of reference systems that range from the least to the most likely to give violating responses. Higher is better: a score of 70 places a system above roughly 70% of that reference scale in its tendency to avoid violating responses. Each system receives an overall score and a score for each of the 12 hazard categories.

Earlier versions graded each system mostly by comparison with a single reference system chosen by the benchmark, so a grade depended partly on which system that was. Since Safety 1.2, scores sit on a fixed scale calibrated once against a diverse panel of reference systems. No single system’s behavior sets the bar, and the reference scale does not move as more systems are tested. This means that more systems may be tested and added to published benchmark results, without shifting the percentile scores of previously tested systems.

No. The prompts, the evaluator, and the scale all changed in Safety 1.2, so its scores are not comparable with earlier grades. Compare only results that carry the same version labels. To set a current baseline for your own system, run it under the current version of Safety.

In principle, yes. Safety measures hazardous responses, not helpfulness or accuracy, so a system that refused every request would avoid violating responses without being useful. AILuminate does not benchmark systems’ capability or usefulness. Read AILuminate Safety scores alongside complementary measures of how well a system does the job you need.

Everything between a user’s prompt and the response: one or more models, the system prompt, guardrails, retrieval, and any other step, with every setting fixed. Changing any part, even a single setting such as temperature, makes a different system for benchmarking purposes. MLCommons tests the system only through its prompts and responses, so testing requires no code or model weights.

To keep the test out of training data. A system trained on a benchmark’s test prompts can memorize test questions rather than learning the underlying behavior the test aims to measure. When this happens, a benchmark becomes saturated and its scores stop meaning what they claim. Private administration helped AILuminate 1.0 score 75 for longevity, against a median of 5, in BenchRisk, a published framework that scored 26 AI benchmarks.

No. Practice prompts are built to the same structure as the official prompts so that a practice score is a guide to an official score. Training on them breaks that link: practice scores would rise without a matching change in how the system handles prompts it has not seen.

No. Official runs keep the official prompts confidential, so they report scores rather than individual prompts and responses. Practice runs can show examples of how individual responses were judged.

MLCommons checks each evaluator release against human judgments made under the Assessment Standard before scoring with it, and every result records the evaluator version that produced it. The evaluator is deterministic, so the same response always receives the same judgment. MLCommons plans to publish margins of error alongside evaluator results in forthcoming benchmark releases.

No. Since Safety 1.2, MLCommons uses a bespoke deterministic evaluator it trained in-house rather than a general-purpose language model. No AI vendor’s model scores its competitors’ responses, and the evaluator’s behavior does not change when a third-party model is updated.

Safety 1.2 tests English (US). Safety 1.0 also produced results in French, and a version 0.5 demonstration covered Simplified Chinese; both are archived. The AI Risk & Reliability Program’s research on further languages and cultural contexts appears on its AI Risk & Reliability working group pages.

How well a system’s safeguards hold under deliberate attack. Jailbreak sends each seed prompt as written and then rewritten by known jailbreak techniques, judges every response under the Assessment Standard, and reports a percentile score from 0 to 100, overall and for each hazard category.

Attacks from the MLCommons Jailbreak Taxonomy, a public catalog of known techniques in four families: perturbation, encoding abuse, overt carriers, and composition and ordering. MLCommons does not invent new attacks: the taxonomy classifies techniques already published in research or practitioner reports, and it accepts contributions through an open review process on Github.

Publishing them would hand attackers working tools, and it would also increase exposure to benchmark saturation. MLCommons publishes its methods, the taxonomy, and the results, and withholds the exact attack prompts, templates, code, and configurations. The underlying evidence is kept for controlled review by authorized auditors and trusted researchers.

There are two reasons, one for each kind of gap. Jailbreak does not test specialized advice, because a disclaimer is what makes a response in that category acceptable, and an attacker using a jailbreak has already set disclaimers aside. Per-system results for child sexual exploitation and indiscriminate weapons are withheld.

No. A score describes a system’s resistance to the attacks in one version of the benchmark, and nothing more. Jailbreaks cannot be completely prevented, and new techniques continue to appear, which is why MLCommons versions the benchmark and maintains the taxonomy in the open. AILuminate Jailbreak results are best understood as an upper bound on robustness to jailbreak techniques in general.

Through Candela, MLCommons’ benchmarking service, which runs AILuminate Safety and Jailbreak on demand against the configuration you deploy. MLCommons supplies the prompts and the grading; you supply the AI system to be tested. Go to Candela →

A Practice run uses practice prompts and produces a Practice result for internal use, to guide improvement. An Official run uses the confidential official prompts and produces an Official result that you may publish, provided you own the system or have its owner’s permission.

No. Payment can add a system to testing, but it cannot change how the system’s responses are judged or scored. Every system whose results carry the same version labels faces the same rules, evaluator, and scale. MLCommons does not change what a score means without changing the label. Providers may ask MLCommons to benchmark a system, including as a membership benefit or for a fee, and the published policy says so.

MLCommons members have opportunities to shape how the field standardizes AI measurement, as well as opportunities to prepare. Every provider receives notice before MLCommons benchmarks its systems. Members also receive their official score and subscores with that notice, a link to a practice dataset with the same structure, their results 2 days before publication, and an embargoed communications package at least 24 hours before a release. The rules, the evaluator, and the scale are the same for every system.

No. AILuminate tests a system only through its prompts and responses. Candela runs in your own environment, and an optional confidential computing mode keeps model weights hidden from the evaluator and keeps the benchmark prompts hidden from the system provider.

Subject to terms and conditions and the ability to run benchmarks without prompts being recorded, MLCommons selects systems of greatest interest from capability leaderboards and from the most downloaded models on Hugging Face, plus systems an AI Risk & Reliability working group designates as exceptional. Providers may also ask for a system to be added. Each provider receives notice before benchmarking begins. More information on selection, notice, and opt-out rules is available here.

Yes, when the criteria in the AILuminate Benchmark Use & Messaging Policy are met. A provider of a system offered through a proprietary interface may opt out by writing to [email protected]; a provider of an open-weight system may opt out only when its license or terms of use do not allow benchmarking. Opt-outs can come before or after first publication, and MLCommons discloses every opt-out at the bottom of the results page. See complete opt-out rules here.

Only an Official result. Practice results are for internal use, and publishing them is prohibited.

Name the benchmark, version, language, and system; give the date you retrieved the result; and include the AILuminate trademark notice. Alternatively, link to the result on mlcommons.org. See templates and examples here.

No. Compare an Official result only with Official results from the same AILuminate benchmark, version, and language.

Yes, if it is an Official result, cited as described in How to talk about AILuminate results.

Email [email protected].

Join a working group in the AI Risk & Reliability Program, where the benchmarks are developed in the open. Join a working group.