A defensible methodology for jailbreak benchmarking
We published A Robust, Defensible, and Reproducible Methodology for Benchmarking Single-Turn Jailbreak Attacks on Large Language Models, the MLCommons v0.7 release that sits between the v0.5 draft and the v1.0 benchmark.
A pile of adversarial prompts is not a standard. This paper puts a mechanism-first taxonomy at the center of benchmark construction: how each jailbreak actually works, deterministic labeling rules, and taxonomy-guided attack selection. The taxonomy is public; the specific attacks used in the private benchmark are not. That split is the point: you can strengthen defenses against the whole landscape without overfitting a hidden test set, and auditors get a method they can actually inspect.