by Last spring, when Karim Lakhani began testing how ChatGPT affected the work of elite business consultants, he thought they would be delighted with the tool. In a preliminary study of two dozen workers, the language robot helped them finish two-hour tasks in 20 minutes.
“I figured they, like me, would think, ‘Great! I can do so much more!” said Dr. Lakhani, a professor at Harvard Business School.
In contrast, the consultants had feelings of unease. They appreciated having done a better job in less time. But ChatGPT’s fast work threatened their perception of themselves as highly skilled workers and some feared becoming too dependent on it. “They were really worried and felt like this would denigrate them and be kind of empty calories for their brain,” Dr. Lakhani said.
After these preliminary tests, Dr. Lakhani and his colleagues devised a larger controlled experiment to measure how ChatGPT would affect more than 750 white-collar workers. That study, which is being reviewed in a scientific journal, indicated markedly contradictory results in the consultants’ work product. ChatGPT greatly improved the speed and quality of work on a brainstorming task, but diverted many consultants from doing more analytical work.
The study also detailed workers’ varying feelings about the tool. One participant compared it to the fire that Prometheus stole from the gods to help mortals. Another told Dr. Lakhani’s colleague Fabrizio Dell’Acqua that ChatGPT looked like junk food: hard to resist, easy to consume but ultimately bad for the consumer.
In the near future, language bots like OpenAI’s ChatGPT, Meta’s Llama, and Google’s Gemini are expected to take on many administrative tasks, such as writing texts, preparation of legal writings and writing letters of recommendation. The study is one of the first to show how technology could affect real office work and office workers.
“It’s a well-designed study, particularly in a nascent area like this,” said Maryam Alavi, a professor at the Georgia Institute of Technology’s Scheller College of Business who was not involved in the experiments. Dr. Alavi, who has studied the impact of new digital technology on workers and organizations, also noted that the study “really points out how much more we need to learn.”
The study recruited management consultants from Boston Consulting Group, one of the largest management consulting firms in the world. The company had prohibited its consultants from using artificial intelligence robots in their work.
“We wanted it to involve a large set of real workers working on real tasks,” said François Candelon, managing director of the company who helped design the experiments.
The volunteers were divided into two groups, each working on a different management consulting problem. Within each group, some consultants used ChatGPT after 30 minutes of training, some used it without instructions, and some did not use it.
One of the tasks was to brainstorm a new type of shoe, outline a persuasive business plan for making it, and write about it persuasively. Some researchers believed that only humans could perform such creative tasks.
They were wrong. Consultants who used ChatGPT produced work that independent reviewers rated about 40 percent higher on average. In fact, people who simply cut and pasted the ChatGPT output scored higher than colleagues who combined their work with their own thoughts. And AI-assisted consultants were more than 20 percent faster.
ChatGPT studies this year in legal analysis and white collar writing assignments They have discovered that the bot helps lower-performing people more than more capable ones. Dr. Lakhani and his colleagues found the same effect in their study.
However, on a task that required evidence-based reasoning, ChatGPT was of no help at all. In this group, volunteers were asked to advise a corporation that had been invented for the study. They needed to interpret spreadsheet data and relate it to mock transcripts of executive interviews.
In this case, ChatGPT made employees trust it too much. Unaided humans had the correct answer 85 percent of the time. People who used ChatGPT without training scored just over 70 percent. Those who had been trained fared even worse, getting the answer only 60 percent of the time.
In interviews after the experiment, “people told us they didn’t check it because it’s so polished and looks so right,” said Hila Lifshitz-Assaf, a management professor at Warwick Business School in Britain.
Many consultants said ChatGPT made them uneasy about how the tool would change their profession and even their perception of themselves. Nearly three in four participants told researchers they were concerned that using ChatGPT would cause their own creative muscles to atrophy, said Candelon of Boston Consulting Group.
“If you haven’t had an existential crisis with this tool, then you haven’t used it much yet,” said another co-author, Ethan Mollick, a management professor at the Wharton School of the University of Pennsylvania.
