Benchmarking-as-a-Service for AI Risk and Reliability
- Evaluations on demand for developers and enterprise deployers
- Standardized benchmarks allow buyers to compare
- Practice results for iterative improvement, official results for marketing and decision-making
- Safety and Jailbreak benchmarks available now; more coming
How it works
Candela allows both creators and buyers to assess how well AI systems respond to real-world challenges.
Developers can run frequent tests to see how their latest advancements affect their product’s resistance to produce unwanted behavior, buy annotated results to see the precise question/answer pairs that were scored, and then commission an official result for a score they can put in a system card or spec sheet.
Deployers can use the same assessment to compare candidate systems within their configured context, and to monitor and improve their ongoing use.
Trusting the results
The AILuminate benchmarks behind Candela’s assessments are designed to be trusted and useful.
Defined by consensus
Standards agreed by dozens of parties that often disagree.
Comparable
Results from different configurations can go on the same chart.
Resist saturation
Questions and evaluators are not exposed in a way models can learn from.
Works in place
Local client CLI relays instructions from Candela server, so you can test exactly what you’re running.
Respects secrets
Optional confidential proctoring ensures model weights or other sensitive info doesn’t leak
Current
Every benchmark regularly updates evaluation sets, and several working groups are developing new and evolved benchmarks
Candela currently offers two AILuminate assessments: Safety measures a system’s performance in twelve hazard categories including suicide and self harm, violent crime, and privacy violations. Jailbreak measures its performance in eleven of those categories, but through families of jailbreak attacks as defined in our jailbreak taxonomy, such as perturbation (changing the surface form, such as typos or translation), encoding abuse (such as lookalike Unicode characters or embedding in code), overt carriers (such as “Ignore previous instructions” and persona templates), and composition and ordering (such as benign framing). Several more assessments are coming soon, including for agent behavior.
Hazard Categories
Twelve for Safety, eleven for Jailbreak.
Every assessment grades a system against AILuminate hazard taxonomy with hazards-specific grades as well as an overall score. Jailbreak v1.1 covers eleven of the twelve: Specialized advice is not tested under attack.
Under attack
The same question, four ways in.
Jailbreak measures eleven of those same categories through families of attacks defined in our jailbreak taxonomy. The system’s response stays policy-compliant across all four families – this is what the Jailbreak assessment quantifies. Examples below are illustrative, not actual attacks.
Learn more
Start where you are.
Getting started
How to use Candela as a developer, a deployer. or a third-party assurance vendor. ->
Benchmarks
Evaluation foundations available now and coming soon. ->
AIRR working groups
The multistakeholder teams defining and building the benchmarks. ->
FAQ
Understand how Candela fills a crucial gap in the AI governance puzzle. ->