While generative AI may hold promise as an efficient way to help small businesses grow, a study of entrepreneurs in Kenya found real-world limitations for those businesses that need it most.
The more successful entrepreneurs in the study were able to get a 15% performance boost after consulting an AI-powered business mentor. But the low performers did worse, seeing an 8% drop in revenue.
The difference, the researchers found, was that the high performers tended to ask for help on relatively straightforward tasks while those who were struggling sought help with more challenging tasks.
“Generative AI has the potential to significantly influence business performance,” says Nicholas Otis, a doctoral candidate at the Haas School of Business and the paper’s primary researcher. “Our results suggest that whether this impact is positive or negative depends on the tasks that entrepreneurs select for AI assistance.”
The five-month randomized control trial included 640 Kenyan entrepreneurs running fast food joints, poultry farms, cybershops for computer services, and a range of other businesses. Otis ran the study with Berkeley Haas assistant professors Solène Delecourt and David Holtz, as well as Harvard Business School doctoral candidate Rowan Clarke and associate professor Rembrand Koning.
AI business mentor
The researchers spent months building a GPT-4-powered AI business “mentor” that entrepreneurs interacted with via WhatsApp, which is used by 90% of Kenyans as a low-cost messaging platform. The researchers tailored the AI to the Kenyan business environment, and programmed it to give multiple pieces of practical advice for each prompt, with details on implementation. The participating entrepreneurs were then randomly assigned to receive a standard business guide or to work with the AI.
The entrepreneurs could use the AI mentor as much as they wanted on any questions that arose, and they used it in a variety of ways. For example, a restaurant owner considering changing a menu asked for help in thinking through the possibilities and uncertainties in making the decision. In another example, a business owner selling wholesale and retail milk asked for help in expanding offerings to increase profits.
No effect on average
Over a period of five months, the researchers gathered over 4,000 data points on firm performance and thousands of interactions with the gen-AI mentor. In their first analysis, the researchers found no evidence, on average, of a positive effect on business performance from those with access to the AI mentor.
But when they split the sample between those whose business performance was above and below the median at the start of the experiment, differences emerged. As noted, the above-median performers saw profits or revenue climb by 15%, whereas the low-performers’ revenues sagged by 8%.
The researchers found no difference in the number or general quality of the questions asked by the two groups. Rather, the content of the questions differed: Low performers, already struggling with weaker revenues and profits, sought advice on difficult tasks that would be challenging for AI—or even humans— to solve. For example, a farm might face stiff competition or a drought, or a business might need capital to survive.
“…Whether by choice or necessity, low-performing entrepreneurs in our sample asked the AI mentor for assistance with more challenging tasks than high performers,” the researchers wrote.
Implications
The results contrast with recent research that found college-educated workers using gen AI were more productive on well-defined tasks—especially benefitting those with the weakest skills. Another recent paper found gen AI can increase productivity for low-skilled workers, thus reducing disparities overall.
Overall, the researchers conclude that generative AI has the potential to benefit millions of companies in emerging economies through personalized advice. But it also has the potential to widen the gap between high-performers—who could address weaknesses and surge further ahead—and those who are struggling.
“This suggests that for gen AI to really add value to entrepreneurs in more open-ended contexts, they would also need expanded access to complementary skills training and resources—including financial resources,” Holtz says.
Even so, a carefully implemented AI intervention does hold some promise for business development, Delecourt adds.
“An optimistic way to view our results is that we had a positive effect for a subset of the population, with a very low-cost intervention,” she says. “It’s just not a one-size-fits-all solution.”
Read the full paper:
“The Uneven Impact of Generative AI on Entrepreneurial Performance”
By Nicholas G. Otis, Rowan Clarke, Solène Delecourt, David Holtz, and Rembrand Koning
February 2024