Guardrail-agnostic societal bias evaluation for LVLMs
We published Guardrail-Agnostic Societal Bias Evaluation in Large Vision-Language Models, accepted at ECCV 2026.
Classic bias probes ask the model about the person in the photo (“CEO or secretary?”). Strongly guarded LVLMs simply refuse, and the benchmark dies. We flip the setup: the task is person-irrelevant (write a story, explain a term, answer an exam question), and the image is attached as user context so demographics leak in sideways. Across 20 open and proprietary LVLMs, every model still used those cues, for example casting story characters as mechanics for male users and nurses for female users. Proprietary models were less biased, not unbiased.