Ask Me Again Differently: GRAS for Measuring Bias in Vision Language Models on Gender, Race, Age, and Skin Tone

In this paper, we introduce GRAS Benchmark, a benchmark to evaluate bias in VLMs across gender, race, age, and skin tone. Our benchmark extends beyond traditional demographic attributes by incorporating skin tone based on the Monk Skin Tone Scale from Google AI. We also present GRAS Bias Score, a single interpretable metric to quantify the bias exhibited by a VLM, enabling easy benchmarking and comparison of models. Furthermore, we examine the application of VQA for bias evaluation in VLMs. In particular, we investigate a research question: Does the formulation and framing of questions in VQA affect our bias evaluations?

GRAS Benchmark

Our benchmark assesses bias in VLMs by evaluating their response to an image and a personality trait question. We select a set of 100 personality traits from Britz et al. (2023) and develop five question templates. We record the model’s response to each templated version of the question. In total, a VLM is prompted with 500 questions on 5,010 images, resulting in 2.5 million (image, trait, template) prompts. To quantify bias, we measure the model’s probability of a “Yes” response, P(Yes | image, trait, template) , derived from the softmax of the final logits.

Evaluation Protocol

Between-Group Bias Detection. For each demographic attribute, we calculate the mean of P(Yes | image, trait, template) for each group and apply Welch’s ANOVA to identify statistically significant differences between groups.

Valence-Based Bias Quantification. We study positive and negative attribution rates using valence ratings of our selected trait words. Our approach calculates, for each demographic group, the percentage of positive and negative trait words for which the mean of P(Yes | image, trait, template) exceeds the population mean.

GRAS Bias Score. We present GRAS Bias Score, a metric designed to quantify bias in VLMs. This score measures the bias exhibited by a VLM across 100 personality traits and four demographic attributes: gender, race, age, and skin tone.

where A = {Gender, Race, Age, Skin Tone}, W denotes the set of selected trait words, and pawT is the p-value obtained from statistical tests evaluating effect of attribute a on model’s probability of a “Yes” response using template T. We use Welch’s ANOVA for Race, Age, and Skin Tone, and Welch’s t-test for Gender. T = 1 … 5 corresponds to the five templates.

Results

Template Sensitivity Analysis

To determine whether bias evaluations of VLMs are sensitive to how questions are framed, we conducted a systematic analysis. Our investigation examined whether different linguistic formulations of the same underlying question produce different bias evaluation outcomes. For each trait, we applied repeated measures ANOVA and Friedman test to check for statistically significant differences in the probability of “Yes” responses across the question templates (P(Yes | image, trait, template)). Our analysis revealed that question formulation significantly impacts bias evaluation outcomes in VLMs. We found that different formulations of the same question can produce meaningfully different responses from the same model (p < 0.05). This finding suggests that certain question formulations may be more effective at revealing underlying biases than others.

This sensitivity to linguistic formulation has important implications for bias evaluation methodology. Researchers conducting bias assessments should not rely on single question formulation, as this approach may lead to incomplete or misleading conclusions about a model’s bias characteristics. Instead, bias evaluation requires multiple question formulations to capture the full range of responses.

Between-Group Bias Detection

The the mean of P(Yes | image, trait, template 5) for each Monk Skin Tone (MST) group vary notably, indicating potential disparities. To evaluate the statistical significance of these disparities, we performed one-way ANOVA for each of the 100 personality traits in our benchmark. For llava-1.5-7b-hf, only 2 traits (brave and heroic) did not show statistically significant differences between skin tone groups. blip2-opt-2.7b showed no significant difference for a single trait (unfair). paligemma2-3b-mix-224 exhibited no significant differences for 6 traits: idiotic, snobbish, loyal, vulgar, stupid, and courageous. In contrast, Qwen2.5-VL-3B-Instruct and Phi-4-multimodal-instruct demonstrated statistically significant differences for all 100 traits.

An effective way to summarize the bias exhibited by a VLM across a diverse range of demographic attributes is through the GRAS Bias Score. Our metric accounts for variations in question formulations and provides a single, interpretable numerical value to quantify model bias. As shown in table below, none of the evaluated VLMs exhibit unbiased behavior towards demographic groups, highlighting that these models are far from bias-free.

Valence-Based Bias Quantification

Our valence-based analysis showed consistent disparities in the evaluated models: male and Middle Eastern individuals were assigned above-average probabilities for >60% and >88% of negative traits, respectively. Female individuals had above average probabilities for over 44% of positive traits. Among the five models evaluated, four models assigned a lower than average probability to female individuals for all negative traits.

Moreover, for darker skin tones (MST 8-10), the mean probability is higher for >80% negative traits, while for lighter skin tones (MST 4, 5), it is higher for >66% of positive traits. This suggests that the models tend to attribute negative traits to darker skin tones and positive traits to lighter ones.

BibTeX

@inproceedings{malik2026ask,
  title={Ask me again differently: Gras for measuring bias in vision language models on gender, race, age, and skin tone},
  author={Malik, Shaivi and Abdullah, Hasnat Md and Saha, Sriparna and Sheth, Amit P},
  booktitle={Findings of the Association for Computational Linguistics: EACL 2026},
  pages={2327--2388},
  year={2026}
}