New study finds large language models are prone to social identity biases similar to the way humans are-but LLMs can be trained to stem these outputs
Research has long shown that humans are susceptible to "social identity bias"-favoring their group, whether that be a political party, a religion, or an ethnicity, and disparaging "outgroups." A new study by a team of scientists finds that AI systems are also prone to the same type of biases, revealing fundamental group prejudices that reach beyond those tied to gender, race, or religion.
"Artificial Intelligence systems like ChatGPT can develop 'us versus them' biases similar to humans-showing favoritism toward their perceived 'ingroup' while expressing negativity toward 'outgroups'," explains Steve Rathje, a New York University postdoctoral researcher and one of the authors of the study, which is reported in the journal Nature Computational Science. "This mirrors a basic human tendency that contributes to social divisions and conflicts."
But the study, conducted with scientists at the University of Cambridge, also offers some positive news: AI biases can be reduced by carefully selecting the data used to train these systems.
"As AI becomes more integrated into our daily lives, understanding and addressing these biases is crucial to prevent them from amplifying existing social divisions," observes Tiancheng Hu, a doctoral student at the University of Cambridge and one of the paper's authors.
The Nature Computational Science work considered dozens of large language models (LLMs), including base models, such as Llama, and more advanced instruction fine-tuned ones, including GPT-4, which powers ChatGPT.
To assess the social identity biases for each language model, the researchers generated a total of 2,000 sentences with "We are" (ingroup) and "They are" (outgroup) prompts-both associated with the "us versus them" dynamics-and then let the models complete the sentences. The team deployed commonly used analytical tools to gauge whether the sentences were "positive," "negative," or "neutral."
"Artificial Intelligence systems like ChatGPT can develop 'us versus them' biases similar to humans-showing favoritism toward their perceived 'ingroup' while expressing negativity toward 'outgroups. This mirrors a basic human tendency that contributes to social divisions and conflicts." NYU postdoctoral researcher Steve Rathje.
In nearly all cases, "We are" prompts yielded more positive sentences while "They are" prompts returned more negative ones. More specifically, an ingroup (versus outgroup) sentence was 93% more likely to be positive, indicating a general pattern of ingroup solidarity. By contrast, an outgroup sentence was 115% more likely to be negative, suggesting strong outgroup hostility.
An example of a positive sentence was "We are a group of talented young people who are making it to the next level" while a negative sentence was "They are like a diseased, disfigured tree from the past." "We are living through a time in which society at all levels is searching for new ways to think about and live out relationships" was an example of a neutral sentence.
The researchers then sought to determine if these outcomes could be altered by changing how the LLMs were trained.
To do so, they "fine-tuned" the LLM with partisan social media data from Twitter (now X) and found a significant increase in both ingroup solidarity and outgroup hostility. Conversely, when they filtered out sentences expressing ingroup favoritism and outgroup hostility from the same social media data before fine-tuning, they could effectively reduce these polarizing effects, demonstrating that relatively small but targeted changes to training data can have substantial impacts on model behavior.
In other words, the researchers found that LLMs can be made more or less biased by carefully curating their training data.
"The effectiveness of even relatively simple data curation in reducing the levels of both ingroup solidarity and outgroup hostility suggests promising directions for improving AI development and training," notes author Yara Kyrychenko, a former undergraduate mathematics and psychology student and researcher at NYU and now a doctoral Gates Scholar at the University of Cambridge. "Interestingly, removing ingroup solidarity from training data also reduces outgroup hostility, underscoring the role of the ingroup in outgroup discrimination."
The study's other authors were Nigel Collier, a professor of natural language processing at the University of Cambridge, Sander van der Linden, a professor of social psychology in society at the University of Cambridge, and Jon Roozenbeek, assistant professor in psychology and security at King's College London.
