Vision Language Models Are Biased
Published on: 2025-06-12 16:47:30
Finding: State-of-the-art Vision Language Models achieve 100% accuracy counting on images of popular subjects (e.g. knowing that the Adidas logo has 3 stripes and a dog has 4 legs) but are only ~17% accurate in counting in counterfactual images (e.g. counting stripes in a 4-striped Adidas-like logo or counting legs in a 5-legged dog).
VLMs don't actually "see" - they rely on memorized knowledge instead of visual analysis due to bias.
... Read full article.