“Classified” is an occasional series spotlighting some of the more powerful lessons being taught in classrooms around Haas.
It’s a recent Tuesday evening at Berkeley Haas, and Marissa Maliwanag, MBA 24, has just five minutes to pitch her team’s idea for Tables Together. It’s an online marketplace that big corporations like Google could use to donate surplus food from their employee kitchens to organizations that feed people in need.
“There are matches that need to be made and we want to create a marketplace and solve the problem,” Maliwanag said, ticking off the amount of food that goes to waste in the United States each year.
After a few quick questions for the team, the rapid-fire pitch slam—part of the MBA class called Online Marketplace and Platform Design—continues. Students pitch ideas, among them a private plane rental marketplace to a community for matching skiers and snowboarders with coaches to a marketplace for tailors of bespoke clothing for events like weddings.
All of the pitches serve as practice for the students who are working toward final projects, says Assistant Professor David Holtz, who teaches the class, an elective that enrolls 68 students. The group is a split of mostly full-time and evening & weekend MBA students, on a journey that covers all aspects of online platforms—from A/B testing, network effects, and platform monetization, to reputation systems and discrimination in online marketplaces.
The class aligns with Holtz’s career experience as a former Silicon Valley data scientist. Most recently, Holtz worked for Airbnb, where he first became intrigued by online marketplaces. “I was exposed to a lot of interesting problems including reputation-system design, algorithmic pricing, and experiment design,” Holtz, a member of the Management of Organizations (MORS) and Entrepreneurship & Innovation Group at Haas, says. “To this day, these topics form the backbone of my research, because, in addition to being extremely interesting, they’re also extremely difficult to solve.”
Taking apart the case
During the first half of a recent class session, Holtz asked students to split into groups to discuss one of the week’s assignments: Pick a company on the a16z Marketplace 100 list—Andreessen Horowitz’s ranking of the largest and fastest-growing consumer-facing marketplace startups and private companies—and come up with a new market mechanism that the company might trial using A/B testing.
One MBA student team wrote about the online specialty food marketplace Goldbelly, suggesting that the company might add a feature that prompts site visitors to indicate that they’re trying to buy a gift. Then, Goldbelly could customize searches and provide a more personal message option at checkout.
Holtz then runs students through a business case called “Innovation at Uber: The Launch of Express POOL, a case directly related to some of his marketplace research that examines experiment design in two-sided markets. Set in March 2018, the case follows Uber through the launch of a new product called Express POOL, which offers carpooling riders a cheaper ride if they agree to walk a short distance to and from pick-up and drop-off points and wait a few minutes before being matched to a driver.
In this case, Uber had to decide whether to keep rider wait times at two minutes or change the Express POOL wait time to five minutes mid-experiment. The big dilemma? Uber benefited from a cost-per-ride reduction with a five-minute wait time but didn’t want to make a change that could hurt the user experience. “Even if the company did decide that a longer wait time was preferable, what did that mean for the ongoing experiment the company was running?” Holtz says. “Should they change the product mid-experiment or let the experiment continue running as originally intended?”
In this case, Uber had to decide whether to keep rider wait times at two minutes or change the Express POOL wait time to five minutes mid-experiment.
Holtz then shifts to a whiteboard, where he outlines different types of experiments (also called A/B tests) that marketplace companies like Uber use to test new features.
First is the “bread and butter” user-level test, which Uber could have used to compare the behavior of riders with access to Express POOL to the behavior of those who did not have access to Express POOL. The second kind of test, a switchback experiment, would give all riders and drivers in a given market access to Express POOL for randomly selected 160-minute-long chunks. Over two weeks, Uber would switch Express POOL availability back and forth to compare behaviors.
The third type of experiment Holtz describes, which Uber did use with Express POOL, is a synthetic control experiment. It is the most accurate form of testing, Holtz says, but also the most complicated to run and the “noisiest.” Using the synthetic control experiment, Uber identified two sets of markets that, in aggregate, were as similar to each other as possible. The company then launched Express POOL in one set of cities, but not in the other. By comparing behavior in the two sets of cities, Uber could estimate the impact of both.
Holtz’s knowledge of how to apply A/B tests comes from deep research. He has conducted multiple large-scale experiments analyzing the effects of marketplace design choices on Airbnb. One study examined whether coupons would lead more Airbnb bookers to write more reviews—with the eventual aim of facilitating better matches on the platform and increasing revenue. Comparing behaviors of buyers who received coupons to those who didn’t, he found that the coupons led to additional reviews that were more negative, on average, and that the reviews didn’t affect the number of nights sold on the site or total revenue.
In a separate, widely cited study, he and his co-authors examined the effects of remote work on collaboration among information workers at Microsoft. They scoured anonymized, aggregated data describing emails, calendars, instant messages, video/audio calls, and workweek hours of more than 60,000 U.S.-based Microsoft employees during the COVID-19 pandemic, trying to estimate the causal effects of firm-wide remote work on collaboration and communication. Results showed that under firm-wide remote work, collaboration patterns become more static and siloed, with fewer bridges between disparate parts of an organization.
Impressive guest speakers
For Lena Corredor, MBA 25, knowledge gained in Holtz’s class is providing an opportunity to explore the challenges of building a successful entrepreneurship marketplace, which is her startup idea.
“This class is really eye-opening for me because it’s not as straightforward as it seems,” she says. “When you think about the different sides of a marketplace, one would think if you build it, they will come, but it’s not the case. The design elements he talks about are very important to business success.”
During most classes, Holtz opens with a guest speaker, and his roster includes an impressive industry bench of leaders including Sudeep Das, head of Machine Learning/AI at DoorDash; Martin Manley, co-founder of Alibris and former U.S. assistant secretary of labor; Ania Smith, CEO of Taskrabbit; and Briana Vecchione, a technical researcher at Data & Society’s Algorithmic Impacts Methods Lab (AIMLab); among others.
Roberto Pérez, MBA/MEng 24, an entrepreneur in Mexico before coming to Haas, said they were drawn to the class for two reasons. “First, I knew that the professor had a great background and first-hand experience on this topic,” they say. “Second, I knew that the class would have a lot of guest speakers and that was interesting to me as this level of exposure is very valuable.”
Looking toward the future of online marketplaces, Holtz said he’s excited to see where entrepreneurs will take new technologies, such as generative AI, AR/VR, and blockchain-based tech. To that end, he said he expects the students will hear more from a group of investors and VCs who are guest judges at the last class—Raphael Lee, Vickie Peng, and Lindsay Pettingill.
“They weigh business pitches all the time and will have a better sense than anyone of where we are headed,” he said.