Star employees unlikely to benefit most from AI: BCG study

• According to a new study, advisors using AI completed tasks faster and delivered higher-quality results.
• According to the study’s authors, below-average performers saw the greatest benefit from using AI.
• Previous studies have shown that AI assistance had a weak impact on highly-skilled customer service employees.

A recent study conducted at the management consulting firm Boston Consulting Group, or BCG, found that below-average consultants benefited the most from using technology.

Conversely, those who did well saw lower gains.

The study, published Thursday, included 758 BCG advisors globally — or about 7% of the company’s individual contributor-level advisors. They were split into two groups: one with access to GPT-4, the model that currently powers ChatGPT, and one without.

Before the experiment, employees were tested to measure their average level of performance, classifying them into “bottom-half” and “top-half” skilled participants.

They were then assigned a series of practical consulting tasks for a fictional shoe company and their performance was graded by human and AI raters.

Below-average performers saw the biggest benefit from using AI, with their average performance improving by 43%.

Their above-average counterparts saw an average performance increase of only 17% using AI.

The study also found that those who used AI completed their tasks faster and produced higher-quality results than those without access.

However, it also highlighted the dangers of over-reliance on AI.

Advisors with access to AI performed up to 20% worse when presented with tasks beyond the AI’s understanding. In these cases, the AI ​​will present misleading yet plausible responses.

The study was conducted by researchers from Harvard, MIT, Warwick University, and the University of Pennsylvania, along with the Boston Consulting Group.

Ethan Mollick, a Wharton professor and co-author of the study, wrote in a post on X, formerly known as Twitter: “AI acted like levelers: underperformers benefited the most.”

Another study — published in April — examined customer service workers at a Fortune 500 company and found that highly-skilled workers saw “close to 0%” productivity gains with AI assistants.

In contrast, low-skilled agents increased their productivity by 35%.

Certainly, generative AI, as it currently exists, is prone to convincingly present false statements as true. The consequences of these errors can be serious — from botched deaths to poor health outcomes for patients.

The study authors and BCG did not immediately respond to Insider’s requests for comment sent outside regular business hours.