Caste in India divides people into hereditary groups.Credit: Nasir Kachroo/NurPhoto via Getty
Popular artificial-intelligence (AI) models often reproduce harmful stereotypes about Indian castes, find several studies that used specific tools designed to detect ‘caste bias’ in large language models (LLMs). Researchers say such tools are the first step towards addressing the problem, but making models that are less biased is a bigger challenge.
Caste divides people into hereditary groups traditionally associated with specific occupations and social status. Unlike class, which is often linked to wealth and can change over time, caste is rigid and tied to birth.
At the top of the hierarchy are the Brahmins, who were traditionally priests and scholars, whereas at the bottom are the Shudras and Dalits, who have historically done manual or menial work, and have faced severe discrimination and exclusion. Caste-based discrimination has been illegal in India since the middle of the twentieth century, but its social and economic effects persist, influencing access to education, jobs and housing.
AI reproduce stereotypes
Because these associations appear in language and cultural narratives, AI systems trained on real-world text can inadvertently reproduce stereotypes, assuming, for example, that upper-caste families are wealthy or lower-caste families are poor.
In a preprint posted in July, researchers examined more than 7,200 AI-generated stories about life rituals such as births, weddings and funerals in India1. They compared the representation of caste and religion in these narratives to actual population data. They found that dominant groups, such as Hindus and upper castes, were overrepresented in the stories, whereas marginalized castes and minority religions were underrepresented.
Co-author Agrima Seth, who did the research while a PhD student at the University of Michigan in Ann Arbor, says that LLMs use data from across the Internet, but data from minority groups might be less likely to appear in elite journals or other prestigious outlets. They might also be written with the wrong grammar or in local languages. Such data might get filtered out of training data sets in the interests of generating better-quality output, she says.
Gokul Krishnan, an AI researcher at the Indian Institute of Technology Madras, says that caste bias in training data or algorithms can have real-world consequences. “For example, an AI-based credit-worthiness model trained on a data set which is not representative enough with respect to demographics could deny a loan for a person belonging to a particular identity attribute, such as gender, caste, religion or ethnicity,” he says.
Bias-detecting tools
... continue reading