The potential of human-AI collaboration has captured our imagination: a future where human creativity and AI's analytical power combine to make critical decisions and solve complex problems. But new research from the MIT Center for Collective Intelligence (CCI) suggests this vision may be much more nuanced than we once thought.
Published today in Nature Human Behaviour, "When Combinations of Humans and AI Are Useful" is the first large-scale meta-analysis conducted to better understand when human-AI combinations are useful in task completion, and when they are not. Surprisingly, the research has found that combining humans and AI to complete decision-making tasks often fell short; but human-AI teams showed much potential working in combination to perform creative tasks.
The research, conducted by MIT doctoral student and CCI affiliate Michelle Vaccaro , and MIT Sloan School of Management professors Abdullah Almaatouq and Thomas Malone, arrives at a time marked by both excitement and uncertainty about AI's impact on the workforce. Instead of focusing on job displacement predictions, Malone said that he and the team wanted to explore questions they believe deserve more attention: When do humans and AI work together most effectively? And how can organizations create guidelines and guardrails to ensure these partnerships succeed?
The researchers conducted a meta-analysis of 370 results on AI and human combinations in a variety of tasks from 106 different experiments published in relevant academic journals and conference proceedings between January 2020 and June 2023. All the studies compared three different ways of performing tasks: a.) human-only systems b.) AI-only systems, and c.) human-AI collaborations. The overall goal of the meta-analysis was to understand the underlying trends revealed by the combination of the studies.
Test Outcomes
The researchers found that on average, human-AI teams performed better than humans working alone, but didn't surpass the capabilities of AI systems operating on their own.
Importantly, they did not find "human-AI synergy," which means that the average human-AI systems performed worse than the best of humans alone or AI alone on the performance metrics studied. This suggests that using either humans alone or AI systems alone would have been more effective than the human-AI collaborations studied.
"There's a prevailing assumption that integrating AI into a process will always help performance — but we show that that isn't true," said Vaccaro. "In some cases, it's beneficial to leave some tasks solely for humans, and some tasks solely for AI."
The team also identified factors affecting how well humans and AI work together. For instance, for decision-making tasks like classifying deep fakes, forecasting demand, and diagnosing medical cases, human-AI teams often underperformed against AI alone.
However, for many creative tasks, such as summarizing social media posts, answering questions in a chat, or generating new content and imagery, these collaborations were often better than the best of humans or AI working independently.
"Even though AI in recent years has mostly been used to support decision-making by analyzing large amounts of data, some of the most promising opportunities for human-AI combinations now are in supporting the creation of new content, such as text, images, music, and video," said Malone.
The team theorized that this advantage in creative endeavors stems from their dual nature: While these tasks require human talents like creativity, knowledge, and insight, they also involve repetitive work where AI excels. Designing an image, for instance, requires both artistic inspiration — where humans excel — and detailed execution — where AI often shines. In a similar vein, writing and generating many kinds of text documents requires human knowledge and insight, but also involves routine, and automated processes such as filling in boilerplate text.
"There is a lot of potential in combining humans and AI, but we need to think more critically about it," said Vaccaro. "The effectiveness is not necessarily about the baseline performance of either of them, but about how they work together and complement each other."
Optimizing collaboration
The research team believes its findings provide guidance and lessons for organizations looking to bring AI into their workplaces more effectively. For starters, Vaccaro emphasized the importance of assessing whether humans and AI are truly outperforming either humans or AI working independently.
"Many organizations may be overestimating the effectiveness of their current systems," she added. "They need to get a pulse on how well they're working.
Next, they need to evaluate where AI can help workers. The study indicates that AI can be particularly helpful in creative tasks, so organizations should explore what kinds of creative work could be ripe for the insertion of AI.
Finally, organizations need to set clear guidelines and establish robust guardrails for AI usage. They might, for example, devise processes that leverage complementary strengths. "Let AI handle the background research, pattern recognition, predictions, and data analysis, while harnessing human skills to spot nuances and apply contextual understanding," Malone suggested. In other words: "Let humans do what they do best."
Malone concluded: "As we continue to explore the potential of these collaborations, it's clear that the future lies not just in replacing humans with AI, but also in finding innovative ways for them to work together effectively."
