Established with a philanthropic investment of almost $17 million from Robert G. and Sue Douthit O’Donnell, the new center will bring together the best minds from a wide range of fields.
Berkeley, Calif.—Ever since Nobel laureates George Akerlof and Daniel Kahneman created a 1987 UC Berkeley course that broke the rigid barrier between psychology and economics, the university has led the way in bringing the once-disparate disciplines together into the field of behavioral economics.
Professor Ulrike Malmendier (Photo: Copyright Noah Berger)
“We went from neoclassical economics that considered humans to be perfectly rational, to behavioral economics that brought in social psychology,” says Ulrike Malmendier, the Cora Jane Flood Professor of Finance, who will serve as the center’s faculty director. “Now we want to move the needle further, bringing together the best minds for rigorous research on human behavior from the sciences more broadly—including neuroscience, cognitive science, biology, medicine, epidemiology, and genetics.”
Funded with a philanthropic investment of almost $17 million by Bob O’Donnell, BS 65, MBA 66, and his wife, Sue O’Donnell, the center aims to become the preeminent hub for the maturing fields of behavioral economics and finance, bringing together leading researchers from a wide range of disciplines for collaboration, conferences, and bootcamps, as well as funding promising PhD students and postdoctoral scholars. The center will also host the prestigious Behavioral Economics Annual Meeting (BEAM), co-founded by Malmendier, every three years.
A nexus for cross-disciplinary research
O’Donnell says he was inspired by the pioneering work of Kahneman, Akerlof, Malmendier, and others who gave Berkeley its leading position in behavioral economics. “UC Berkeley is dedicated to integrating business education with other disciplines on campus, which is essential in this area,” he says. “It should have a center devoted to continuing this work.”
The center, says Berkeley Haas Dean Ann Harrison, will create a far-reaching impact across UC Berkeley, a research powerhouse with many areas of strength. “The goal is to cut through barriers that traditionally hinder research across disciplines, such as different ways of presenting data and publishing results, and bring people together in a different way than what’s usually done,” she says. “The O’Donnell Center will be the nexus of a new form of cross-disciplinary collaboration that pushes behavioral economics toward the future.”
Beyond ‘homo economicus’
Traditional economics was based on the assumption that human beings are perfectly rational, profit-maximizing “robots”—sometimes referred to as “homo economicus” or “economic man,” Malmendier says. Behavioral economics brought in insights from psychology and human behavior to explore the predictable foibles in our thinking, such as decision-making biases, fears of losing out, lack of self-control, and overconfidence. A classic example is Kahneman’s pioneering work with Amos Tversky on loss aversion, which showed that people are willing to take greater risks to avoid a loss than to secure a gain.
These ideas have been integrated into economics and finance departments around the world and have deeply influenced public policy and practice. For example, after Nobel Laureate Richard Thaler and Cass Sunstein developed the concept of the “nudge”—interventions that spur people to act in their own self-interest, such as enrolling in a retirement savings plan—hundreds of “nudge units” were established in governmental and private-sector organizations around the world.
Many other Berkeley Haas researchers helped pioneer this intellectual revolution, including finance professor Terrance Odean, BA 90, MS 92, PhD 97, the Rudd Family Foundation Chair, who was convinced by Kahneman to pursue a doctorate in finance rather than psychology and whose work reveals investors’ flawed decision making.
O’Donnell, the center’s founding donor, says he often applied insights from behavioral economics during his career as a portfolio manager for a large mutual fund group. “It represents a further step in the evolution of financial theory comparable to the development of the efficient market hypothesis,” he says. “When combined with existing financial theory, I believe that its insights enhanced results for my clients.”
Yet, during the 17 years he taught an investment class in the Berkeley Haas MBA program, O’Donnell says he sometimes encountered skepticism when he introduced ideas from the field. “Indeed, one student asked, ‘Isn’t all this kind of woo-woo?’”, he says. “Several years later, that student told me how perspectives from behavioral economics had helped her career in finance.”
Experience effects
Now, after more than three decades of foundational work, it’s time to move behavioral economics past its adolescence, Malmendier says. “Behavioral economics made progress by including psychology, but we didn’t include all the other sciences.”
Malmendier, whose groundbreaking work on “experience effects” earned her a Fischer Black Prize in 2013 for the top economist under the age of 40 and a Guggenheim Fellowship in 2017, has focused on complex economic behaviors. She has studied how stressful experiences with recessions, layoffs, inflation, housing bubbles, and political repression make consumer and investor behavior more cautious and risk averse for years afterward, and she has explored how stress can affect our health, careers, education, and other aspects of life in dramatic ways.
To further that work, Malmendier aims to bring a wider range of researchers together and break down silos. For example, collaborating with neuroscientists, neuropsychiatrists, biologists, medical researchers, and epidemiologists who have studied stress and trauma could more precisely demonstrate how past experiences shape our actions today and across generations. Stress impacts the big variables that economists study, such as completing an education, choosing an occupation, and deciding to have a family, she says.
“As we walk through life, our outlook on the world changes, especially if we suffer trauma,” she says. “Neuroscience says our brain gets rewired. There may be a long-term impact of stress on our longevity, on our aging, and on our health.”
Questioning the status quo
Malmendier, who now serves on the German Council of Economic Experts, is passionate about the potential of behavioral economics to help leaders create better solutions to the most complex and urgent problems of our time—from fighting climate change to battling inflation and avoiding financial crises. “If leaders keep in mind people’s emotions, their personal histories, and their psychologies, they can engineer ways to make things more predictable and give people more control over events help them live better lives,” she says. “That is our ultimate goal.”
Professor Stefano DellaVigna
Moving the field forward will also involve rigorous research to reexamine what has come before. For instance, a recent paper by center co-founder Stefano DellaVigna, the Daniel E. Koshland Senior Distinguished Professor of Economics and professor of business, with Elizabeth Linos of Harvard, suggests that leaders should get more realistic about nudge policies—and better at incorporating them into practice. Two government nudge units opened their records to allow the researchers to look at all their interventions. By examining 126 randomized controlled trials of nudge policies involving 23 million people in the United States, the researchers found that nudge interventions are on average effective, increasing the desired outcomes by about 8%. However, the effects are less than those in published academic papers—about one-fifth the size. The authors attribute the difference to publication bias, or the tendency toward publishing only large, surprising results.
“Our study stresses the importance of research transparency,” DellaVigna says. “This transparent access is quite unique and shows a further innovative impact of behavioral economics, which has led to more evidence gathering within governments.”
In a second paper, DellaVigna and Linos, along with Department of Economics doctoral student Woojin Kim, found that even when nudge policies are found to be effective, public agencies implement them only about a quarter of the time, often due to organizational inertia.
In addition to Malmendier and DellaVigna, the center will include a host of affiliated researchers from Berkeley Haas and Berkeley Economics, as well as from across the university. They include Berkeley Haas professors Ricardo Perez-Truglia, Ned Augenblick, Don Moore, and Gautam Rao, PhD 14—who will join Haas in January from Harvard University—as well as Dmitry Taubinsky of Berkeley Economics and others. The founding gift will establish a permanent endowment to support the center and some of its ongoing activities.
Minister Nishimura Yasutoshi (center) speaks with Acting Dean Jennifer Chatman (right).
Nishimura Yasutoshi, Japan’s Minister of Economy, Trade and Industry (METI) visited UC Berkeley and the Haas School of Business this week to spread the message that Japan is making significant investments to transform its economy through entrepreneurship and innovation.
While Japan may be best known for its big companies like Toyota and Sony, “They began as startups first of all,” said Minister Nishimura, speaking through an interpreter in a conversation with Haas Acting Dean Jennifer Chatman. “Entrepreneurship is really in the DNA of the Japanese people.”
Invited to campus by the Clausen Center for International Business and Policy as part of events surrounding the Asia-Pacific Economic Cooperation (APEC) Summit in San Francisco, Minister Nishimura’s visit also expanded the collaboration between UC-Berkeley and the Japan External Trade Organization (JETRO). During the event, Caroline Winnett, Executive Director of the Berkeley SkyDeck accelerator, signed a memorandum of understanding with JETRO to further advance entrepreneurship, innovation, and scholarship.
Minister Nishimura also toured SkyDeck, which to date has hosted about 60 Japanese startups through its JETRO partnership.
Berkeley SkyDeck was thrilled to host Yasutoshi Nishimura, the Japanese Minister of Economy, Trade, and Industry, for a visit to SkyDeck, following an MOU signing with @JETRO_InvestJP to launch our second international program. Learn more here: https://t.co/5SXLsxzXfyhttps://t.co/coKXsjQwyA
In addition to the SkyDeck collaboration commemorated at Minister Nishimura’s talk, Berkeley Haas has a long tradition of partnering with Japanese companies and universities to promote innovation and entrepreneurship. In the past year, the Berkeley Haas Entrepreneurship Program has worked with Tohoku University to train top startups from the Sendai region in Lean Launch methodology. Haas has also hosted leading Japanese companies at the Berkeley Innovation Forum to explore building their innovation and entrepreneurial ecosystems.
To achieve its goal of a tenfold increase in the number of startups over the next five years, the Japanese government plans to send 1,000 entrepreneurs to the Bay Area over a five year span, and to invest in university partnerships, noted Haas Continuing Lecturer Jon Metzler, who helped organize the METI visit to Haas.
“The government of Japan is taking a number of measures to stimulate entrepreneurship, increase new venture formation, and nurture entrepreneurs with a more global mindset—including sending promising entrepreneurs to acceleration programs like Berkeley SkyDeck,” Metzler said.
‘Unicorns and decacorns’
METI Minister Yasutoshi Nishimura spoke to the Berkeley community on Japan’s innovation goals.
After an introduction by Associate Professor Matilde Bombardini of the Clausen Center, Minister Nishimura delivered prepared remarks and sat for Q&A with Acting Dean Chatman. He said everyone who has visited Japan in the past few years is surprised by how much it has changed.
“In terms of macroeconomy, over the past 30 years because of deflation, it has been a challenging time for Japan. But now we are in an era when big changes are about to take place,” Nishimura said. Within the population of about 125 million, many entrepreneurs have been content to find success within the country. But Nishimura is encouraging young entrepreneurs to think big and “go global.”
“We are looking toward the emergence of many unicorns and decacorns,” he said.
Nishimura also talked about plans to build a next-generation semiconductor fabrication facility in Hokkaido, which will adopt a 2 nm fabrication process—a major technological leap compared to current fabs in Japan.
In addition to the Haas’ Clausen Center for International Business and Policy, the event was hosted in partnership with the Berkeley APEC Study Center, the Institute for East Asian Studies, the Center for Japanese Studies, and Berkeley SkyDeck, with support from the Haas Asia Business Club. The Japan Society of Northern California also helped promote the event.
Left to right: Chris Bush, executive director of the Institute for Business Innovation; Berkeley Haas Lecturer Jon Metzler; METI Minister Yasutoshi Nishimura; Acting Dean Jenny Chatman; and UC Berkeley Executive Education CEO Mike Rielly.
Ruth Porat, president, CFO, and chief investment officer at Alphabet and Google, ranks among the most influential women in finance, having earned spots on both Forbes’ and Fortune’s“Most Powerful Women” lists.
Porat joined the Nov. 1 Berkeley Haas Dean’s Speaker Series, discussing her rich journey from Wall Street to tech.
As a child, Porat said her parents, both scientists, told her to “learn, study, embrace the new, and work really hard.” “And when I graduated from college, I thought I would end up going into labor law,” she said. Later, while earning an MBA degree at Wharton, her path took a turn while taking an amazing mergers and acquisitions accounting class. “That was the game changer for me,” she said. “I ended up applying to Morgan Stanley, starting my career there, and thinking that was the only thing I would ever do: mergers and acquisitions. But as the world changed, I kept moving around.”
After decades on Wall Street, including serving as CFO of Morgan Stanley from 2010-2015, Porat pivoted to the world of technology, joining Google in 2015 as chief financial officer. At Google, Porat was one of the key team members who worked to shift the company’s business model and its name to Alphabet. (Alphabet is the parent company of Google and several former Google subsidiaries.)
(Watch the DSS interview with Ruth Porat)
Even with her years of finance experience, she said she found herself constantly adapting to the new environment at Alphabet—whether by learning new technical tasks or working with new teams.
“The most important thing, from my perspective, is drawing on what you know, but being authentic about what you don’t,” she said. “So just go with it, learn, know that you’ll figure it out.”
As Google’s longest-serving CFO, Porat continues to utilize her strong finance background to lead.Throughout her career, Porat said she has also discovered the importance of what she calls a “work-life mix.” Unlike the traditional “work-life balance,” which can lead to guilt, she said she prefers to look at life as a kaleidoscope—in which both personal and professional lives co-exist.
“What I always tell people on my teams is, ‘Look, I trust you. That’s why you’re on my team,'” she said. “If there’s something you need to do at home with kids, with family, do it. Don’t feel guilty, don’t check in with me. And when you need to be here, you know, really torquing it for work, you’ll be doing that. It works out over time.”
Read the full transcript:
Good afternoon, fellow Bears. How are you? I’m Jenny Chatman. I’m the acting dean at Haas. It’s a delight to see you all here for this exciting event. Welcome to today’s Dean’s Speaker Series.
I’m thrilled to welcome Ruth Porat, a powerhouse woman in finance. Ruth has broken barriers her entire career from her start at Morgan Stanley in the mergers and acquisitions department to her current role as president, chief investment officer, and CFO at Alphabet and Google. Ruth has seen finance through many different lenses, private, public, and tech. It is said that she has a magic power of charting paths through uncertainty, especially when she led through the financial crisis of 2008. Ruth has also been decisive in initiating change coming into Google in May of 2015 and being a key member of the team that shifted the business model to Alphabet.
A few months later, Ruth even finds the time to mentor others and serve on several boards, including Blackstone, the Council on Foreign Relations, and Bloomberg Philanthropies. In our ever-changing world, I’m grateful that we can learn from someone who has led with resilience and strength throughout her career.
Ruth, thank you so much for taking the time to come to Haas and speak to us today. So, just a note on our process today: You have note cards on your seats. If you have a question anytime throughout the conversation, please jot down that question, and be sure to put your name and which program you’re from and give it to our monitors, and they will get the question to our question asker. So, we’ll be collecting those cards throughout the event, and we’ll save time at the end to answer those questions. So, I’m now going to turn it over to Madhu Gupta and Paula Gutierrez, who are going to moderate today’s discussion. Thank you.
– Good afternoon everyone, and thank you so much Ruth for being here. We’re super excited to have you.
– Just a huge thanks, just wonderful to be with all of you, and thank you for those very, very kind comments. It’s wonderful to be with the group.
– My name is Paola Gutierrez.
– And I’m Madhu Gupta.
– And we’re both second-year, full-time MBA students. As a school with a strong interest and love for technology, our community is really excited to hear from you today. In our time together, we want to take you and our audience through a journey, first learning about your career, then your leadership development, your strong passion and advocacy for women in technology, and then, finally, your outlook on the future. Does that sound good?
– Sounds good. Let’s go.
– So you come from a family of scientists—your father was a physicist, and your mother was a psychologist. And you yourself have three degrees from economics, international relations, industrial relations, and an MBA from Wharton. So tell us a little bit: What got you into the business world, and why finance as a place to begin your career?
Ruth Porat: So I feel blessed to have had the parents I’ve had. They were humble immigrants who were very focused on education. They always said, “Education is your passport for life.” And my dad, as he said, he was a physicist. We ended up in California when I was pretty young. He was at the Stanford Linear Accelerator Center. And as a young child, when I was in elementary school, he used to take me up to Slack, and we would build gadgets. He taught me how to use a soldering iron when I was in elementary school. And for those of you who don’t know what that is, it gets to a pretty high temperature, and you can melt things, so you can combine component parts. We built a radio—it didn’t really play good music, but I was proud as an elementary school kid to build that. And my mom was a psychologist. And as a young child, she said it was really important as a woman to have a career outside of the home. And she showed me that it was possible. And so their message early on was always, “Just learn, study, embrace the new, work really hard.” And when I graduated from college, I thought I would end up going into labor law. One thing led to another, graduate school, open doors. I ended up taking an amazing class—a mergers and acquisitions accounting class was the game changer for me. Go figure. But it really was. And I ended up applying to Morgan Stanley, starting my career there, and thinking that was the only thing I would ever do. Mergers and acquisitions. But as the world changed, I kept moving around.
– Wow, you’ve had an incredible career journey. You spent 27 years in Wall Street, and then 2015, you joined Google as CFO. So walk us a little bit through that journey and how you navigated two of the most challenging industries for women—I mean both banking and technology—and how that transition was for you.
– I think there were, over the years, I came up with a couple of rules that I kept going back to. And one of the most important ones, and I tell my teams this all the time, “Anchor everything in data, and the rest will follow.” And I think as a young woman growing up in the financial services industry, when you have data and you lay it out in a compelling way and it tells a story, and in particular, when you build sensitivity analysis that forces engagement on, “So, what are the key variables? Is it revenue growth, or is it operating expense expectation? Why? Where do you think the world will go?” It sort of just starts to transcend “Who are you,” and, “Why are you there?” And you go right to data, and there was an expression at Morgan Stanley: “The answer is a number.” I think that’s really true. And I always tell my team, “Sensitivity analysis, grab people in a different sort of way.” So anchor in data. I think the other thing that’s really important is: Find a great boss. So, my first project at Morgan Stanley, I was working around the clock, and I never got to go to even internal meetings, let alone client meetings. And finally, after about a month, I was brought into a meeting, and the managing director looked at this guy and said, “So are you the one that I keep hearing about?”, which sort of made it pretty clear he was taking credit for my work, and it taught me, “Go find somebody you really want to work for who will take a risk on you and has your back.” And I did that repeatedly. And probably one of the most important stories for me was, I worked for somebody who eventually became the head of equity sales and trading. And when I was asked to run technology equity capital markets, which is a joint venture between banking and sales and trading, he called me into his office and he said, “I think you’re going to soar, but if you slip, I’m here to backstop you. I will be your senior air cover.” And I just thought that that was such an important phrase. We all need senior air cover, and we all need to be senior air cover for someone. So that’s the second lesson. And then the third, which is sort of a corollary, is when you have that great person, you can also have open-ended conversations about your career. So, my go-to phrase that I think has served me well—and I always advise everyone to use—is: Don’t pinpoint where you want to be and at what time, but say, “What’s my highest and best use,” whenever that opening comes. Because every door that opened for me, virtually every door that opened for me, what came out of that kind of conversation, what’s my highest and best use? And then it would, that person I trusted who took a risk on me, who was backstopping me, would point to something, and we’d have an open dialogue about whether it made sense, whether I wanted to do it. So those would probably be the three most important rules.
– Just a quick follow-up. So many of us, at least MBA students, are looking at pivots in our career. So in that pivot that you made from banking to tech, were there any points of time where you felt a little nervous or kind of unsure of what was what you were going to be facing, and this also very new and growing industry?
– Well, I think pivots are great. I think pivots actually, the most important thing from my perspective is drawing on what you know but being authentic about what you don’t. And so, the pivot from Morgan Stanley to Google wasn’t as dramatic as it seems. I had run tech banking at Morgan Stanley, I’d taken Google public, I knew a lot of the people, but a lot was definitely very new, but probably one of the most important pivots I made. Going back to this conversation, highest and best use, I was asked at one point at Morgan Stanley to take over and run the investment banking business focused on financial services. So banks, insurance companies, asset management. And I had done tech mergers, private equity. So I had a very clear answer, which is, “That sounds horrendously boring.” Like, there’s no way I want to do that. But somebody I trusted who ended up becoming president of Morgan Stanley said, “You got to trust me on this one. Running financial institutions is actually a lot like running tech—25% of the global economy, multi-sector, tech, you’ve got hardware, software, semis, in financial institutions, banking, insurance, et cetera. You got to try this.” And he was trying to be helpful. The first meeting he set up for me was with Larry Fink, who’s the founder and CEO chairman of BlackRock, largest asset manager. And I’m thinking, “OK, this is terrifying.” I know nothing about asset management at this point. So to your question, where there are moments where it’s like, “Why am I here?” Yes, but I concluded, and he was trying to do me a favor, introduced me to one of the most important people in that sector. And I said, “They have a section of BlackRock that is all about technology. I’ll lean into that. I won’t pretend I know asset management, and that’s the most important thing.” Be authentic with what you know—you’ll learn the rest later. And just one last bit, the transition to Google, there was a lot I didn’t know, and I was constantly Googling terms, and I literally, at one point, I’m in a meeting with Larry Page and Sergei Brin, and Larry turns around and sees me doing this, and he said to the group, “She’s Googling everything.” I’m like, “That’s OK. I’m learning.” So just go with it, learn, know that you’re not, you’ll figure it out.
– You’re doing, I guess now, a slightly less dramatic pivot, but you’ve been CFO for 14 years, and now in July, you became president and chief investment officer of Google. So why now? And why the need for this role?
– Well, I felt like the position I’ve had as CFO and still have, it’s a privilege to be in this role, and it’s been extraordinary. And as CFO for eight years and the longest-serving CFO—and I’m still the CFO—and I will, I’m still massively focused on landing our plan and where we go from here. But it just seemed like the right time. Fourteen years as CFO, longest one at Alphabet and at Sundar, and I talked about it, there’s a real opportunity to say, “Where do we sharpen our focus on investments? How can he and I work together on that? And what’s the interplay between investments and global policy?” And I think it’s an exciting time on this planet, and so, it just seemed like the right time. So I’m grateful for it.
– Thank you, Ruth, for sharing your early interest in finance, your career, and how you’ve grown along the way. Shifting gears now a little bit to learn about who Ruth is and the personal experiences that have shaped her. I recently read that, when you interview candidates at Google and Alphabet, you often ask about their battle scars, which, correct me if I’m wrong, but for the audience here, asks, “What has happened to you, and how have you learned from it?” Today, we would love to pose that same question to you and learn a little bit about your personal battle scars and how they have shaped who you are today.
– I do ask that question in every interview. I think it’s a really valuable one to see, “How have people learned, and what pattern recognition do they have as a result of some of the toughest things that you’ve been through in life?” And so, there are probably two worth noting on the personal, ’cause you call, you said personal battle scars on the personal, I’ve had cancer twice. First, in 2002, 2000, I’m sorry, 2001. I’m blocking the whole thing out. 2001, and then again in 2004. And at the time, my kids were very little, first time around 5, 7, and 9, and it was clearly terrifying. I’m completely fine. I’m grateful for the amazing care that I got, but if it taught me anything, it’s to make sure everyone knows: Don’t assume that you can plot out a life on a timeline and defer to later what you want to do now. When I was in the hospital for surgery, I had these three amazing kids. I’ve been married to this amazing man forever. I am really happy with my career there. There were, I was the oldest in the cancer surgical center, and I looked at younger women, and my fear for them was “Had they done what they had wanted to do?” So, all I can say is, whatever it is, “Grab life by the moment.” Every day really is precious. And don’t defer because you don’t actually control the timeline as much as we all would like to think, professional, but I’m fine. So that’s the good news. Medical care is amazing, in terms of professionally, probably the biggest battle scar was going through the financial crisis. And because I didn’t actually go with my instinct on that, no way am I going to do financial services. I did end up running it in 2006, and then crisis, when you were in a bank started in 2007, 2008, secretary Hank Paulson asked if I would, a team of us from Morgan Stanley would, we got seconded to the Treasury Department. I worked with him very closely on Fannie Mae, Freddie Mac, when we went through the various Lehman, et cetera, weekends, and then AIG. And it was extraordinary to go through that and just really see the imperative of making crisp decisions at the right time, in the right way, with the right team.
– Thank you for sharing. I know many of us in this room can draw inspiration from the immense strength and courage with which you’ve navigated these very difficult experiences. Now forging a path in a successful path, I should say, in these industries, does not come without personal sacrifices and evaluating trade-offs. Now, of course, you are no stranger to that, having navigated many challenging periods in your professional life as you just mentioned. So navigating the 2008 financial crisis, a long battle with cancer, and most recently the pandemic and tense economic conditions, what have been the most important lessons that you have learned along the way—and how have they changed the way that you show up and lead every day?
– That is a great question. I think one of the most important things I come back to all the time, I say it frequently, is “the absence of information is filled with dirt.” And when you’re going through any crisis, personal or professional, the absence of information is filled with dirt. Those around you fear the worst. And one of my kids, ’cause I also use this expression at home, said, “Actually, mom, when you’re a teenager, sometimes it’s filled with fun,” and he’s totally right. But fundamentally, the absence of information is filled with dirt. And I think that, to me, the fear that our teams had going through COVID, you don’t know everything when you’re going through a crisis, obviously it’s so rapidly evolving, you don’t want to overstate and then under-deliver, but sharing what you know, how you know it, making it clear you care and you want to bring people along, is absolutely imperative. And when I look at all of you thinking, you’re about to enter the workforce, I tell my teams this all the time because also, it’s really important actually, as when you’re on a team, if you don’t tell your boss what you’re doing, sometimes they assume, “Well maybe they’re not actually doing very much or doing what I want them doing.” Communication is really important. I had a boss who said, you can never over-communicate enough. I think that’s right. That’s probably one of the most important. And then the second really important one is something I learned actually from Hank Paulson, secretary Paulson as we were going through the financial crisis. And as I’m sure many of you remember, at one point, the biggest concern was about Greece, and was Greece at a tipping point? And he said the problem when you’re going through a crisis is, “You have to have the will and the means, the political will and the financial means, too often by the time you have the will, you no longer have the means.” Problems get bigger over time. And I think it’s very instructive for all of us. When you see an issue, it’s easier. First of all, it’s easier to prevent than to fix. And it’s easier to fix early than to let it grow. So act, be decisive, you’re always dealing with least-worst options. There’s no good option. Those would be some of the main lessons.
– Thank you. You touched upon how you have led through these very trying times for the world. And, of course, your incredible leadership has earned you the title of the most powerful woman on Wall Street. And one of the most powerful women in the world by Fortune and Forbes. I want to ask you, what do these labels mean to you, and how do you shoulder this responsibility for the women and minorities around you?
– You know, in the early days, it was uncomfortable to hear something like that. But then I realize what’s really important. I am here because of the many women before me who made this possible, my mom, who said, “Have a career,” and I saw you could be a great mom and have a great career. And oftentimes, I’d come home and wonder, “Why did everybody else get to come home?” And their mom was there, mine wasn’t. But I saw what an important role model to us. And she’s really, that generation cleared away so much our, my generation is trying to clear away more for all of you and other underrepresented groups. So I think it’s really important that we, A., understand and see the various roles that women and other underrepresented groups have in these industries and are there for one another. And really, again, that senior air cover backstop be there for one another. Some of the things that we may each think, we’re idiosyncratically going through it. No, probably a number of us have dealt with it before. And so having that open network is really, really valuable.
– Yeah, just to follow up to that, What do you believe is the biggest priority to get more women into tech and leadership roles?
– Well, I think for every organization, it has to start with tone from the top. ‘Cause tone from the top sets that expectation for all. But that’s not enough. You then, I firmly believe, need to ensure that the organization has invested in process systems, and I’m going to come back to it and data, at the end of the day, set your goals, hold yourself accountable, and see how you’re doing with data. Data will tell you, will make it really clear, elucidate where you’re missing. You’re falling short of expectations. But I think of it like any supply chain issue. If it’s not working, you need to basically go back through, “What is it, do a postmortem, what’s not working, what in our supply chain is not working?” And it starts with clarity about what your objectives are, the data, and then putting process around recruiting, retention, training, visibility, every element of it to make it possible.
– You’ve been a strong champion for women in tech basically for all of your career. Even in 2018, you joined your team in the walkout protesting sexual harassment at Google and at the Wall Street Journal Tech Live conference, you asked, “If the tech industry can make cars that drive themselves, why can’t it do better when it comes to sexual harassment?” Many of our MBA students, myself included—and we spoke a little bit about this before coming out to the stage—are focused on centering diversity, equity, justice, and belonging in their careers and future ventures. When you’re thinking of your role as CFO—and now as the chief investment officer—how do you make decisions to advance DE&I, not only in the makeup of employees and the organizational culture but also in the way that products and services are designed and taken into market?
– That’s a great question and so important. Look, I think it starts with a deeply held view. I have, Sundar (Pichai) has, our senior leadership has, that diversity is not, and diversity, equity, and inclusion in an org, is not just the right thing to do, but it builds a much better, stronger, more capable organization. The diversity of views lets you understand and see things that you don’t otherwise see. And so, everything I said about put process and systems so that you’re making sure that you have at the broadest representation that you’re supporting people throughout their careers, giving them the access, the opportunity is key. And then, as we talked about briefly, the impact on product is absolutely imperative. It goes to the same point, one of my favorite stats, 1.3 billion people on the planet have accessibility challenges. So when we think about Google and our mission to organize the world’s information and make it universally accessible, one of the important thing, universally accessible, 1.3 billion people on this planet are in need of some elements of what we can do with technology that will open the ability, access to information, access to different elements. And so it makes our products better. Everybody should be focused on this. I also, as a CFO, one of the things I’m really proud that we do, is we have a digital skills training program. And it started when our colleagues in New York wanted to find people from underrepresented communities in New York, train them to do IT support. Fast-forward many years, it was such a success, their conclusion was, “If we want to really have an impact, what we should do is create certificates that are portable,” which we now have nine, “Build a consortium.” We now have more than 250 companies in the U.S. More than half of the people who go through this program are from underrepresented groups. And what we’re doing is giving them access to high-growth careers. Well-paying careers. And what is extraordinary to me is, if you want to have sustainable economic growth in society, you need to have everybody participating. You need to address the opportunity gap. And so, we’re addressing it through our products, but also through other efforts where we can say, “How do people participate in this growing, thriving, changing economy?” And so, I am proud of it, it’s very core to the way we think about it.
– That’s amazing. Sometimes there are things that we can’t control based on the society and the systems that we work under. So, I want to switch gears a little bit and talk about the things that we can do personally to set ourselves up for success when we’re facing some challenges in our career. So, you’ve been very vocal about not liking and banishing the term “work-life balance” because you believe it sets us up for failure, both men and women. And you are a big advocate for a work-life mix. So can you kind of speak a little bit to what that means and how you’ve created that for yourself?
– Yes, it’s one of my favorite topics. I love your chapters back and forth here. So look, I think where you said it in the question, work-life balance, like balance, the physics of balance is really hard. And I think that when you’re trying to have balance, all you do is feel guilty that you’re not doing enough at work, you’re not doing enough at home. You haven’t, you’ve let somebody down. So the metaphor, you know, I keep coming back to is, “Life is like a kaleidoscope.” If you have one side that’s equal in size to the other, and one’s yellow and the other’s green, that’s pretty boring compared to what life really is. Lots of different shapes, lots of different colors, they move over time. And so, what I always tell people on my teams is, “Look, I trust you. That’s why you’re on my team. If there’s something you need to do at home with kids, with family, do it. Don’t feel guilty, don’t check in with me. And when you need to be here, you know, really torquing it for work, you’ll be doing that. It works out over time.” And to me, it’s that work-life, mixed kaleidoscope that’s really important. And you’re absolutely right. It is for all of us. All of us need to figure out in our lives, how do we actually combine a mix? And for me, a large part of it was kids, but it may be something else. It may be sports or community service or whatever it is, you need a mix. Because the other point is, if you don’t have a mix, I think you burn out on your job. There’s no way that any job, I love my job, I’m so grateful to have this opportunity, but if this was it, I can’t imagine that it would be completely all nourishing. So, figure out a mix, recognize that you’re, do it whenever you want to do it. No timeline thing. Going back to the first question and then that mix is going to nourish you over time.
– A quick follow-up on, you talked about kids was part of your mix and figuring that out. How did you figure out what was right for you and what felt right for you? If you know that being a mom was going to be a big part of that mix, or was that something that you tested, maybe a few things you didn’t like? Try some other things until you found the right recipe?
– Yeah, I probably, if my kids were here, I don’t know what they would say, but hopefully, it would be good. It does start with the fact that my parents were amazing, and as I said, with a mom who worked, I think that really helped because I had a pattern that I saw actually could work. And I think part of the mix element is I didn’t try and have two separate lives. My kids were very aware of what I was doing. And they met the people with whom I worked, so we played games. If I was working on a deal, I would explain how the globe worked and the markets where they open, and we would follow it. Which may sound like a weird set of games. Some of you’re probably like, “Wow, great mom.” But, you know, I, one thing I have right on my dresser ’cause it means so much to me working during the financial crisis, we were literally working around the clock and that AIG awful weekend, or Monday, Tuesday, Wednesday, rather, I was around the clock, and I came home and my kids had each written a little note to me, yellow sticky note, and it totally reflected their personality. The oldest must have been 14 or something at the time, was like, “Mom chillax, this’ll be fine.” Which is his personality. And then the middle guy was like, “This is so important for the country.” And the little guy was like, Mommy, hurry up and fix this because I want you back.” And it was, but they got it. They got, it was, I was doing something that mattered. And to me, that was because we always talked about it. And also because I was present when in everything important in their lives. So that is a way of saying, I really, this whole thing. Can you have it all? Well, I think you can, you just have to define what that is. Maybe there are plenty of things I haven’t been able to do and plenty of things where I wish I could have done more. But it’s a beautiful mix when you think about it. It’s the kaleidoscope.
– Thank you for sharing.
– Sure.
– Yeah. Thank you for taking us along your thoughts on balancing professional and personal lives. And I personally loved the kaleidoscope analogy, so I’m going to be using that. We are now at your, let’s see, eighth year at Alphabet and Google, and congratulations, what an incredible journey that has been for you. Google, as we know, is known for its transformative world, changing ideas, often being the pioneer of groundbreaking innovation. But investing into such projects requires a high degree of risk and uncertainty. How do you evaluate and measure the impact and potential of these projects against the risk and the potential consequences they may carry?
– Yes, it’s a great question. I think, for all of us, coming out of business goals and focusing on finance and capital allocation is one of the most important things that we do. And I think there are really two parts to your question or two parts to the answer—one I learned really early in my career. I was on a deal at Morgan Stanley, the Gillette hostile takeover defense. It was written up in the book from Good to Great. And the reason it was written up is, the CEO said, “Do whatever you need to do to keep me independent, but you cannot touch my R&D budget, and I will not tell you what it is, but I’m better off dead than existing without that budget.” And it ended up, I learned years later it was the Sensor razor. And I think the message I got as that young associate was, “If you don’t invest for long-term growth, you are starting to sow the seeds of your decline.” And so, you are totally right, there are risks associated with it, but there’s a bigger risk not doing it. And so, really important to invest for long-term growth. Now, that needs to be calibrated, you got to have the engine, but by the time you need that growth to kick in, I have seen too many companies across too many industries where they think about it too late, and then they’ve sort of run out of runway. So then, how do you analyze it, to your question? It’s hard to do. I mean, it starts with, again, data. What are the assumptions that you need to look at? And oftentimes, and I learned this really working at Morgan Stanley and thinking through, “How do you determine what to take public in many instances, the articulation of what is the model, how can it evolve, how does it defy what is already out there in the conventional world,” gives you a sense of what’s the art of the possible but also what’s the aspiration and ambition of the CO. Do you have the right team that’s going to drive to get to the top of the mountain? And so the analytics are hard. It has to be a problem that’s worthy of taking on. We often start with, it’s got to be, have the opportunity to really benefit people around the globe in a meaningful way. It has to be big enough to matter. And then you look at, “How is it transforming? Is it truly a different approach, a different experience?” And so, it’s, yeah, and you have to, and then finally, I’m sorry, set metrics along the way so that you can kind of mark yourself, metrics and milestones along the way.
– Thank you for sharing. It’s really fascinating to sort of hear about that data-driven approach that you just described. All of us MBA students in the room today. We are super excited about the future. We’re very hopeful, but at the same time, we are living in very uncertain and anxious times with the emergence of gen AI and the impact it can have on the workforce. Very tense economic conditions and worsening climate change. And the list just goes on, with that. When you think about the future, Ruth, what keeps you up at night? What helps you keep going and what are you most excited about?
– There’s a lot to be excited about, and I think you’re absolutely right. There’s so much to be concerned about. I do think technology throughout history has been a catalyst to solve problems. It’s been a catalyst for economic growth. When we talk about AI, we want to make sure that we’re bold and responsible. We’ve talked about the fact that, when you have a technology, that “is this extraordinary and transformative?”, and then Sundar, our CEO years ago in Davos said, “It’s like fire. It can sterilize water, it can heat your home, it can burn it down. It needs to be properly governed.” So, I think, when I’m very excited, when I look at what AI can do, if I go back to cancer as an example, when I got to Google, shortly thereafter, our amazing team had a breakthrough in early-stage detection of metastatic breast cancer. I had breast cancer. And so, I call my oncologist at Sloan Kettering and said, “Is this as important as it seems, or is this just kind of Silicon Valley believing itself?” And he said, “You cannot, we cannot transform health care, we cannot democratize health care.” That was his language. “We cannot democratize health care without AI.” And to me, that’s extraordinary, because I had amazing care because I was at an extraordinary institution. But not everybody lives near Berkeley or near Sloan Kettering or you name it. Not everyone lives in these areas. And so the upside, whether it’s in health care, and many cancer docs will believe that we will solve many aspects of cancer in our lifetime. That’s amazing, right? Whether it’s with education, whether it’s applying it to climate change, there’s a lot to be excited about. But we do need all of us focused on, “How do we use this productively in the right way?” Climate change is so self-evident that the pace at which the evidence is in front of us that we need to make sure that we’re individually, collectively applying ourselves. And at Google, what we look at is, “What can we do through products and services that will help more than a billion people on the planet make decisions in their daily life?” And, “What can we do at, across Alphabet to transform tech with technology that can transform solutions?” And, “How can we run our data centers 24/7 on carbon-free energy? What do we need to do?” So, we each, I’m concerned and excited, and I think that’s the right way to think about it because what the solutions are here if we apply ourselves with gusto, and we need to collectively own these issues.
– Thank you, I think we definitely needed that inspiration.
– Last question for today. You were in our shoes once at Wharton a few years ago. What is your-
– A few years ago, that’s so generous. A few decades ago.
– Just a few years ago, we have a room full of MBA students here who are very excited to embark back into their professional lives. What would your advice be, and if there’s one thing that you could do differently when you were in business school, what would that be?
– So I feel like you keep asking a question. I give a two-part answer, apologies. But I graduated from Wharton, I’ll date myself because it’s relevant to the story. I graduated in 1987, I started at Morgan Stanley in August of 1987. And for the historians in the room, October of 1987 was the single worst percentage decline in the history of the stock market. And that record has not been broken. So I started my career six weeks before I thought the world—the finance world—was falling apart. It didn’t, if you now go and actually Google it and look at do a chart, right? August 1987, to the present, it’s a tiny blip. It is irrelevant. In the moment, it felt like everything I’d worked so hard for maybe was gone. And what I ended up doing was I thought I would only be in M&A, and I was never going to do anything other than mergers. But new things came out of banking after that. And so, first of all, no matter what the world throws you, I do think these things open up new doors and keep it in context. It feels much worse in the moment. Number two, be open to new ideas. If you love what you’re doing, stay with it. You don’t need to move around. But if you get to a point where I did a number of times in my career where I felt like I was plateauing, I went back to the, “What’s my highest and best use?” And I was open to change. And just make sure you’re working with great people, ’cause they will coach and train you. They will get you up the curve much faster. I ended up hiring somebody when I got to Google who was close to retiring as controller of Intel. He had a bit more time left. But I said, “Come over to Google because we have amazing young rising superstars, but I want someone with your pattern recognition.” And that’s what he did. He got all of the team up the curve really quickly ’cause he brought so much wisdom from his many years of experience. So, lean on those people. If you’re working with great people, they’re going to love to coach and see you rise up, and you’re going to have fun.
– Well thank you, Ruth. It’s been an absolute honor chatting with you today, and I speak, obviously, on behalf of all of us when I say we’re so grateful for the opportunity to learn from you. With that, I think we’re ready to move on to audience Q&A.
– Cool, thank you. Thank you both for the questions.
– I’m Don Moore. I am the associate dean for Academic Affairs here at Haas. Thank you, Ruth.
– Thank you.
– For this wonderful, insightful discussion.
– Is your mic on?
– Your mic’s not on
– It’s not on? Oh, nope. OK, thanks. So let’s see, I have a few questions from the audience. Given the fact that Google’s on the forefront of the AI revolution, can you take us into the future and share with us what your or Google’s vision is for that future and help the aspiring young professionals in the audience place themselves in that future? How should their careers take account of where things are headed?
– So AI, what’s interesting about it is it’s so, so much a part of every conversation that we’re having, but the reality is, of course, we’ve all been using AI for quite some time. Anytime you take, you do a photo search, or anytime you’re traveling and you want to translate to another language, across the board, contact centers are currently using AI to provide operating leverage to the teams they have there directing questions queries most efficiently. So, it is here. That being said, your question is a great question because the inflection is extraordinary, and I think you’re going to see it in every industry and it’s going to, so wherever you are going, whatever you’re going to do, make sure that you’re fluent and fast and, “How can I use AI to either get closer to customers or think about operating efficiencies in my business? Where should I get operating leverage? How can I use it as a risk tool?” Those are sort of the business answers. When I think about the implications for the world, I’ve hit on a lot of it. And to me, that is what’s exciting, the implication for health. So as an example, our leading AI group, Google DeepMind. So at DeepMind, they open source to the world, something called AlphaFold, where it’s basically giving to the world protein structure as a base for researchers, scientists globally to work on drug discovery, climate change solutions. And what I’ve heard doctors say, researchers say is, “This is the most important foundation for drug discovery.” And to me, these types of things are what get me really excited. So, it’ll be in every part of your life. And I think those of you say, “I want to go into governance around it,” that’s a great part as well. Making sure that it is safe, that bias is addressed. Oftentimes, I’m asked, “Well what’s the role if you’re not a computer scientist?” Plenty. Because much of what we do in the real world today has bias embedded in it. You don’t want to just replicate that. So we like to bring in social scientists, humanists to make sure that what you’re looking at, we’re thinking about how to improve upon this. And then the last point I’d make is, I personally really like the concept that it’s not artificial intelligence—it’s augmented. And for each one of us, it should augment what we do. It’s a tool for us to use, but judgment and bringing teams together, creativity, it is going to be a foundation, just like I’m old enough to remember my first calculator, my first PC. These are all operating leverage, and we should use it that way.
– A follow-up question. Google is at the forefront of organizations racing toward the transformative introduction of general AI, and there are some intelligent observers who worry about the risks that unleashing AGI on the world could pose competitors racing toward that frontier may all want to get ahead of their rivals and may all wish that the government imposes some safety regulations on all the competitors. What sorts of regulations would Google like to see imposed?
– It’s a really important question. The White House actually has done a very good job here bringing industry in. And I think the group of us who’ve come together, and they just issued their executive order on AI, but the principles around governance for AI have been very important to do red teaming, to test what you’re doing, to provide transparency, to look at watermarking, all of these issues where you look at it as a collaborative body trying to ensure that we prevent misuse, abuse, and the downside risk. And I think one of the other things that’s been very important that the White House has also done is, what we don’t need is a U.S. solution and a Europe solution, and fill in the blank. So trying, for example, the G7 came out with some added principles, and everyone’s trying to constructively come together to delineate, “What are the rules of the road that will make a difference?” And we’re pleased to see that that is the way people have approached it
– A couple of times already. Today the issue of climate change has come up. How are you thinking about the current climate crisis from the will means perspective? What’s Google’s role, especially from an investment perspective in helping create a better, more sustainable future?
– Well, I’m very proud that from the earliest days the founder, our founders Larry and Sergey had been focused on climate change. And so, we were the first to be carbon neutral than we’d said in 2012. We wanted to match 100% of our energy consumption with renewables, figuring that if you put basically a bid out in the market, you would catalyze more supply. That, at the time, the team thought it would take about a decade to get there but hit that goal by 2017. And so, a number of years ago, we said, “What we now want to do is work on actually not just matching but never emitting. So let’s run all of our data centers, all of our office campuses on carbon-free energy 24/7.” It is not easy to do, but we’re making a lot of progress on it because, again, collectively, we need to set clear aspirational goals. The data are self-evident, they’re going in the wrong direction. And so, that part of that’s about “how do you use AI.” For example, one of the things that we found is we can make our data centers more efficient. We think we’ve had the most efficient data centers on the planet, and yet, with cooling, you can make them more efficient with an application of AI to measurement and looking at where you cite them so that you can actually use alternate source of energy. If anyone wants to come down to Sunnyvale and see our campus there, we’re really proud of the way we’re building, we’re using in our Sunnyvale facility, geothermal energy, you can see our solar panels on the buildings. And so, building it into everything you do is one, the second I’ve already mentioned, “How do we make sure that our products address what people around the globe are asking for, which is how can we each have an impact?” So eco-friendly routing is the one example, and if you search on flights, “What’s the carbon footprint from flights?”, that should motivate every CEO of a company that is the bottom of the list to move higher up on the list. And so again, it’s another form of saying transparency, and then we have some exciting breakthroughs in our… across what we’re doing in research that are still early,
– Such important contributions. So, the issue of the global climate crisis and wise shared guardrails on the development of AGI both necessitate international cooperation. Shefali is a full-time student in our MBA program and a former U.S. diplomat. She asks, “Could you please share more about your engagement with international relations and how this connects with your career in finance and tech?”
– Great question. So, I look, I firmly am of the view that there’s an interplay between public policy and the types of investments that one makes, would want to make. For example, if you’re citing a data center somewhere, you want to make sure that we can actually protect users’ data. That we don’t actually lose it because there’s an expectation. And so, there is unequivocally a linkage between getting the right policy in place and then people do want to invest, invest responsibly. And I think bringing that lens to all of the discussions is key. I saw that unequivocally during the financial crisis, you, and each one of you working, where you work will have technical insight, and the question is, “What actually is in the art of the possible that can land that doesn’t have unintended consequences?” And what’s very important is, you come at it trying to solve the problem for the country, trying to actually step outside and into the role of the person you’re working with. It is extremely transparent when you’re not doing that, and as somebody who sat on the other side of the table, you want to make sure you’re engaging with people who are constructively trying to solve the issues at hand for the broader society and bringing their technical expertise. So I think that interplay between finance, investment, public policy, is a wonderful nexus that enables us to have stronger public policy.
– Have you been able to apply your studies in international relations to your work?
– Well, it’s funny, because my thesis a million years ago was about nuclear energy and nonproliferation of nuclear weapons. And, at the time, I was very clearly saying this is a concern. Fast-forward many decades, not many years, but many decades. And I actually think to the climate change question, there’s a real opportunity, in particular, given we have smaller scale of nuclear energy opportunities to look at what’s the interplay there. And are there ways that we should be rethinking some of the things that I learned many, many years ago. But absolutely, the lessons of history are lessons we keep learning over and over again. And whether it’s how do you solve, how do you bring together the public-private sector to convene in a way that develops and delivers very responsible governance around AI. There are plenty of history lessons there, or whether we think about where else we can execute. I think every discipline that any one of you has gone through, you will find a nugget in there that’s relevant.
– Nebo Iwenofu is a student in our evening weekend MBA program and asks, “You have to have broad knowledge and deep understanding to be successful at your level. How would you coach a leader in your organization to be both broad and deep in their area of responsibility?”
– I think you develop that over time. Don’t expect too much upfront from yourself. Do what you’re doing—do it really, really well. I think there was one point in my career at Morgan Stanley where I actually stepped back as we were getting to the year-end process, and I realized I was working for too many people, I was supporting too many people. I was in equity capital markets, I think, at the time. And so, I had a whole host of business partners, and I think focus yields results, and so, go deep, have a great boss or series of bosses who will coach and groom you. Then, you get expertise that you can then continue to build upon, and you have plenty of time ahead so you don’t need to rush it. I think it’s very easy to be super impatient and not have that context, and you’ll be surprised how over the years it just sort of is like a patchwork quilt that builds on itself.
– If you could go back in time to deliver a message to your 27-year-old self, what would you tell yourself?
– Probably, “It’s going to be OK.” I don’t actually look on things where I’ve messed up. It’s like mistakes. I view them as learning experience, and so, I don’t actually record them that way because they are learning experiences, and I think just keeping things in perspective and appreciating the kaleidoscope that life does evolve over time. There were plenty of times, actually, I finally have a good answer as I was rambling on and on too. Inarticulately, the most valuable is that you’ll be able to get there and learn and grow and just grab the moments when you see them and enjoy it as you go. Don’t feel guilty, maybe, I don’t know what it was in my life, but I have felt guilt all the time. So I’m very articulate about saying, “Don’t feel guilty.” Work-life mix, not work-life balance. It came from feeling guilty, right? That I wasn’t doing enough, and only when I deconstructed why and got to the kaleidoscope and got to the fact that I have amazing kids and incredible partner in life, did I realize be there in the moment and be wherever else you need to be in that moment. And it’s OK. Guilt doesn’t actually do anything productive in life.
– Thank you. We should wrap it up there. Let me thank our question-askers, Madhu and Paola, and thank Ruth for the wisdom that you shared with us today. Thanks to all of you for coming.
– Rather than ending on the word guilt, I’m just going to say, “Have fun.” You’ve got so much ahead of you. So it’s a pleasure being with all of you, and thank you.
After a series of high-profile research findings failed to hold up to scrutiny, a replication crisis rocked the social-behavioral sciences and triggered a movement to make research methods more rigorous. A six-year effort to test these emerging methods, led by labs at UC Santa Barbara, UC Berkeley, Stanford, and the University of Virginia, has shown they can produce new and highly replicable findings. The paper, published in the journal Nature Human Behavior and co-authored by Berkeley Haas Professor Leif Nelson, is the strongest evidence to date that the methodological reform movement known as open science can lead to more reliable research results.
Image: Robert Kneschke for Adobe Stock
Roughly two decades ago, a community-wide reckoning emerged concerning the credibility of published literature in the social-behavioral sciences, especially psychology. Several large-scale studies attempted to reproduce previously published findings to no avail or to a much lesser magnitude, sending the credibility of the findings—and future studies in social-behavioral sciences—into question.
To confront this crisis, a handful of top experts in the field set out to test whether emerging research practices can produce more reliable results. Over six years, researchers at labs from UC Santa Barbara, UC Berkeley, Stanford University, and the University of Virginia discovered and replicated 16 novel findings with ostensibly gold-standard best practices, including pre-registration, large sample sizes, and replication fidelity.
Their findings, published in the journal Nature Human Behaviour, suggest that with high rigor, high replicability is achievable.
“The major finding is that when you follow current best practices in conducting and replicating online social-behavioral studies, you can accomplish high and generally stable replication rates,”said UC Santa Barbara Distinguished Professor Jonathan Schooler, director of UCSB’s META Lab and the Center for Mindfulness and Human Potential, and senior author of the paper.
Their study’s replication findings were 97% the size of the original findings on average. By comparison, prior replication projects observed replication findings that were roughly 50%.
The paper’s principal investigators were John Protzko of UCSB’s META Lab and Central Connecticut State University, Jon Krosnick of Stanford’s Political Psychology Research Group, Leif Nelson at the Haas School of Business, UC Berkeley, and Brian Nosek, who is affiliated with the University of Virginia and is the Executive Director of the Center for Open Science.
“There have been a lot of concerns over the past few years about the replicability of many sciences, but psychology was among the first fields to start systematically investigating the issue,” said lead author Protzko, who is a research associate to Schooler’s lab, where he was a postdoctoral scholar during the study. He is now an assistant professor of psychological science at Central Connecticut State University.
The question, Protzko said, was whether past replication failures and declining effect sizes are inherently built into the scientific domains that have observed them. “For example, some have speculated that it is an inherent aspect of the scientific enterprise that newly discovered findings can become less replicable or smaller over time,” he said.
The researchers decided to perform new studies using emerging best practices in open science—and then to replicate them with an innovative design in which they committed to replicating the initial studies regardless of outcome.
“It is important to test the replicability of all outcomes,” said Nelson, the Ewald T. Grether Professor in Business Administration & Marketing at the Haas School of Business, UC Berkeley. Scientists and scientific journals will always prioritize emphasizing newly confirmed hypotheses, but consumers of science care just as much about the hypotheses that were not confirmed. We should care about the replicability of both outcomes.”
Replicating 16 new discoveries
Over the course of six years, research teams at each lab developed studies which were then replicated by all of the other labs. In total, the coalition discovered 16 new phenomena and replicated each of them four times involving 120,000 participants.
Across the board, the project revealed extremely high replicability rates of their social-behavioral findings, and no statistically significant evidence of decline over repeated replications. Given the sample sizes and effect sizes, the observed replicability rate of 86%, based on statistical significance, could not have been any higher, the researchers pointed out.
They also ran several follow-up surveys to test the novelty of their discoveries. “It would not be particularly interesting to discover that it is easy to replicate completely obvious findings,” Schooler said. “But our studies were comparable in their surprise factor to studies that have been difficult to replicate in the past. Untrained judges who were given summaries of the two conditions in each of our studies and a comparable set of two-condition studies from a prior replication effort found it similarly difficult to predict the direction of our findings relative to the earlier ones.”
Indeed, many of the newly discovered findings have already been independently published in high-quality journals.
Because each research lab developed its own studies, they came from a variety of social, behavioral, and psychological fields such as marketing, political psychology, prejudice, and decision making. They all involved human subjects and adhered to certain constraints, such as not using deception. “We really built into the process that the individual labs would act independently,” Protzko said. “They would go about their sort of normal topics they were interested in and how they would run their studies.”
Collectively, their meta-scientific investigation provides evidence that low replicability and declining effects are not inevitable, and that rigor-enhancing practices can lead to very high replication rates. Even so, identifying which practices work best will take further study. This study’s “kitchen sink” approach—using multiple rigor-enhancing practices at once—didn’t isolate any individual practice’s effect.
The additional investigators on the study are Jordan Axt (Department of Psychology, McGill University, Montreal, Canada); Matt Berent (Matt Berent Consulting); Nicholas Buttrick (Department of Psychology, University of Wisconsin-Madison), Matthew DeBell (Institute for Research in Social Sciences, Stanford University), Charles R. Ebersole (Department of Psychology, University of Virginia), Sebastian Lundmark (The SOM Institute, University of Gothenburg, Sweden); Bo MacInnis (Department of Communication, Stanford University), Michael O’Donnell, (McDonough School of Business, Georgetown University); Hannah Perfecto (Olin School of Business, Washington University in St. Louis); James E. Pustejovsky (Educational Psychology Department, University of Wisconsin-Madison); Scott S. Roeder (Darla Moore School of Business, University of South Carolina); and Jan Walleczek (Phenoscience Laboratories, Berlin, Germany).
This Veterans Day, we asked some of the many veterans in our community to share their stories.
Phil Ickes didn’t come from a military family. But a desire to explore the world—and make it a safer place–drew him to join the U.S. Army.
Phil Ickes, MBA 24
Ickes, MBA 24, began his military career as an undergraduate in the ROTC program at the University of Pittsburgh. Between his junior and senior years, he studied Arabic in Jordan and volunteered at Syrian refugee camps, which he describes as a turning point.
“One weekend, I met a family that had been severely wounded by a car bomb while they were still living in Syria,” said Ickes, who grew up outside Pittsburgh. “Meeting that family made me feel compelled to do something about it.”
After graduating, Ickes joined the U.S Army, training in the Naval School Explosive Ordnance Disposal (EOD), a school jointly run by the Army, Navy, Air Force, and Marine Corps. Working as s a platoon leader, his team supported the U.S. Secret Service, protecting several presidents, vice presidents, and foreign dignitaries.
As an EOD operations manager, he later directed an operations cell that supported more than 600 soldiers in full-spectrum bomb disposal.
In 2021, Ickes traveled to Syria and Iraq as an EOD company commander. “That experience definitely left me feeling like I left the world better than I found it,” Ickes says. “The act of disarming bombs and rendering explosive devices safe and leaving communities overseas safer than I found them was definitely very rewarding.”
Phil Ickes helped to detonate bombs around the world while serving in the U.S. Army.
Working with robotics and machine learning in the military piqued his interest in tech, and drew him to apply to Haas, where he’s working toward a career in technical product management.
Last summer, he interned at San Mateo-based drone startup Skydio. He said he believes drones have the potential to benefit fields including law enforcement, infrastructure inspections, mapping, and commercial delivery.
“I like products that make people or businesses’ lives easier or more efficient,” he said. “Usually these products are at the intersection of software and hardware.”
On making the transition from military life to an MBA program, Ickes said he immediately found a collaborative spirit and welcoming environment at Haas.
“Everyone is ambitious yet laid back,” he says. “Nobody takes themselves too seriously.”
“Join the Navy, see the world”
Just after turning 30, while working as an accountant in Santa Barbara, Emily Hawkins decided she wanted something more in life.
“It was a great job, but it kind of felt like I was lacking purpose, and it was at that time in your life where your friends are settling down and having families,” said Hawkins, MBA 24.
Emily Hawkins went from boot camp to an aircraft carrier and a seven-month deployment off the coast of Japan in 2010 and 2011.
She’d always been interested in the military and, after some research and inspired by the slogan “join the Navy, see the world,” she decided to enlist.
“My plan was to enlist for one term, which is four years, learn some new skills, have an adventure, be part of something bigger than what I had been doing, and then return to what I thought of as my normal life,” Hawkins said.
Adventure is what she got. Hawkins went straight from boot camp to an aircraft carrier and a seven-month deployment off the coast of Japan in 2010 and 2011.
Hawkins and her team were called in to help with the Fukushima nuclear accident in 2011. “I had a chance to participate in the humanitarian operation, and it was amazing, and it kind of broadened my perspective of what a career in the military could be, and I was hooked,” Hawkins says. “And that was about 14 years ago.”
Hawkins is still serving in the Navy while earning her MBA, but her role has shifted from working on an aircraft carrier to serving as an officer in supply, logistics, and financial management.
While she was accepted to three MBA programs, she chose Haas due to the support of the school’s Veterans Club. “They were by far the most engaged in terms of helping you with your resume, reviewing your essays, talking to you pre- and post-interview,” Hawkins says. “They were so engaged and so supportive that I knew there would be a good community for me.”
Veterans Day, for Hawkins, is a time to reflect. “It’s a time to think about how we as a society view military veterans and the programs we provide for them in terms of education, transition programs, and medical care.”
With approximately 44 million Americans having reported daily feelings of loneliness, U.S. Surgeon General Dr. Vivek Murthy earlier this year declared that we are facing a “loneliness epidemic.”
Juliana Schroeder, the Harold Furst Chair in Management Philosophy and Values Professor and Barbara and Gerson Bakar Faculty Fellow at Berkeley Haas, explored this crisis in a recent TEDxMarin talk, sharing strategies to fight loneliness and live a more connected life. Schroeder is a behavioral scientist who researches the psychological processes behind how people interact and make social inferences about others.
In fact, Schroeder herself once felt plagued by loneliness. She shared that while riding public transit early in her career, she used to feel particularly disconnected from those around her. Technology is only exacerbating our loneliness, she added.
“Avoidance is easier for us than ever,” she said in her talk. “Online avoidance feels convenient, but it carries an inconvenient consequence.”
“Avoidance is easier for us than ever.”
This “inconvenient consequence” extends beyond the emotional impacts of loneliness. In fact, Schroeder noted a meta-analytic review in the journal PLOS Medicine that found sustained loneliness is potentially as harmful to human health as smoking 15 cigarettes a day.
When people refuse to interact with others and instead retreat into their tech devices, Schroeder explained, the cycle is only perpetuated.
“The paradox of loneliness is that people who are lonely don’t want others to think there is something wrong with them,” she explained. “So they further isolate themselves, creating a cycle of loneliness.”
“The paradox of loneliness is that people who are lonely don’t want others to think there is something wrong with them. So they further isolate themselves, creating a cycle of loneliness.”
Strangers on a train
Schroeder, who also holds affiliations with the Social Psychology Department, Cognition Department, and Center for Human-Compatible AI at UC Berkeley, has explored this paradox in her own research, looking at not only the causes behind the loneliness epidemic but also at potential solutions. When completing her doctorate in social psychology at the University of Chicago, for instance, she ran an experiment inspired by her own experiences on the train.
She found that people expected that talking with a stranger on the train would result in the least positive commute experience—but, in reality, it did the exact opposite. Those who engaged with others on the train reported having the most positive commute experience, leading her to beg the question: Why do people choose to avoid each other if they’d be happier connecting?
One of the main reasons, she finds, is that people tend to overestimate the social risk that comes with connecting with others. Citing the rule of social reciprocity, however, Schroeder noted that nearly all participants received a response when engaging with another person on their commute.
Fostering deeper connections
Curing loneliness is about more than just making small talk—it also requires establishing deeper connections with colleagues, friends, and more. In another study published in the journal Psychological Science, Schroeder found that the medium through which communication occurs is crucial to its quality. That is, voice-based communication like in-person and phone conversations—as opposed to text-based communication like texting, emailing, or DMing on social media—changes how words are received and understood, influences word choice, and affects the synchronicity of exchanges.
In fact, research suggests voice conversations enhance brain wave and heart rate synchrony—the neurological and cardiovascular evidence of psychological alignment.
“Combating loneliness means making the deliberate choice to connect when all you want to do is avoid,” Schroeder concluded. “These choices might feel small, but in aggregate, they have the power to change us from feeling lonely to feeling connected.”
The 2023 Finance Fellows: Back row (left to right): Erik Swisher, Renzo Viale Paiva, Gauri Deshpande , Marya Unwala, Martin Lima; front row (left to right) Rogerio Rios, Venky Vuppalapati, Yvonne Mondragón, Isabella Fantini, Hector Alamillo, Daniel Espinoza Birman. Photo: Jim Block
When Yvonne Mondragón finished the Berkeley Haas undergraduate program in 2016, she worked for seven years in finance, planning a long-term career in investment banking.
“I knew I wanted to come back to school in order to pivot into investment banking and work in banking at the highest level,” she said.
Mondragón, MBA 25, is now well on her way, as one of 11 first-year full-time Haas MBA students named among Haas’ 2023 Finance Fellows.
As fellows, the students receive a scholarship award and are assigned mentors—Haas alumni working in finance, including recent graduates and senior executives.
The 2023 fellows include:
Mondragón, for the C&J White Fellowship in Finance.
Isabella Fantini, Renzo Viale Paiva, Marya Unwala, and Martin Lima for entrepreneurial finance.
Venky Vuppalapati, Gauri Deshpande, Hector Alamillo, and Erik Swisher for investment banking.
Daniel Espinoza Birman and Rogério Rios for private equity and investment management.
About 45% of these new fellows are international, reflecting the percentage of the overall MBA class, said William Rindfuss, managing director for Strategic Programs with the Haas Finance Group. Several of the students bring work experience in different finance sectors from their home countries, and are looking to pivot to larger sectors in the U.S.
Vuppalapati, who is from India, said he’s drawn to the excitement of technology investment banking, and closely tracks how world events, the day’s news, and government policy impact financial markets.
“When I think of investment banking, I also think about how much any one deal can impact different people and different industries,” he said. “Tech has the largest impact, so it feels like a great fit.”
Rios, originally from Brazil, said he’s fascinated by innovation in health care, which led him to pursue a MBA/MPH degree.
“Innovation and technology are going to shape the future, and I want to be in a place that would not only give me an opportunity to be close to financial markets but also provide a solid understanding of how business and tech intersect with health care.”
Inspired by the four Berkeley Haas Defining Leadership Principles—Question the Status Quo; Confidence Without Attitude; Students Always; and Beyond Yourself—Rios added that he is seeking to make an impact on the world and give back to his family.
“I’m a first-generation student, so a lot of my efforts are in the spirit of giving back to them and to my community,” he said.
Mondragón, who is also a first-generation college student, said she hopes to serve as a role model.
“Having someone who looks like me in the finance space is so important,” she said. “I have the lived experience of someone who did not benefit from this space. I grew up not having much access to any of the knowledge that I have now.”
Fantini said she is coupling passions for both technology and venture capital at Haas—and adding a lifelong interest in the food supply chain.
“Haas has such an amazing focus on sustainability and food,” Fantini said. “I knew I could stay connected to Silicon Valley, stay connected to venture, and get even more connected to food resources by coming here for an MBA.”
Choosing a Medicare plan is both complicated and consequential. Berkeley Haas research has fueled a new company that simplifies the process, promising to save money and improve health for millions of people.
Photo courtesy of Healthpilot
It’s Medicare open enrollment season, and the tens of millions of retirees who rely on the government program for health care are grappling with an abundance of options, a dearth of information, and no good way to personalize or compare plans.
“It is not a well-functioning market,” says Jon Kolstad, an associate professor at UC Berkeley’s Haas School of Business who studies the economics of health care. “And yet the choices people make have high financial stakes—consumers are typically on fixed incomes—and critical health implications.”
Kolstad has been studying this challenge since he was in graduate school and, with several academic collaborators, recently helped turn a broad foundation of research into a company centered on helping people make better decisions when choosing a Medicare plan. Healthpilot, which launched in late 2020, uses machine learning to compare Medicare plans and suggest options that are personalized to each person’s current circumstances, projected health needs, and risk tolerance. It’s also free to use.
“We recognized the power of this highly predictive algorithm developed to solve a critical unmet healthcare need for seniors who are evaluating Medicare plans,” says Healthpilot CEO Seth Teich. “We believe that Healthpilot’s platform is a transformative technology that empowers consumers to easily navigate through the complexity of plan choices to find, and enroll in, their best coverage option.”
A desperate need for innovation
People with questions about Medicare enrollment have long relied on phone calls to private agents who walk them through the decision. But there’s a problem: “In reality, these brokers, who are supposed to be experts, do no better at selecting a plan than the average person,” Kolstad says.
In a 2021 paper, Kolstad and several colleagues, including UC Berkeley economist Ben Handel, demonstrated that brokers are prone to the same flawed judgment as everyone else. For instance, they place too much weight on a plan’s premium while overlooking other costs, such as out-of-pocket expenses. The result is that consumers working with brokers pay $1,260 more per year on average than they would if they enrolled in the best plan.
This is not the result of brokers’ bad intentions but simply because solving which plan is best for which person “is a very complex computational problem,” Kolstad says. In fact, the 2021 paper found that when brokers were provided with an AI assistant to help them suggest a plan, they saved consumers about $300 per year. (This is a conservative estimate.)
Healthpilot’s promise lies with its ability to use AI to properly weight the many fixed and projected costs of every available plan, sifting carefully—and impartially—through these multidimensional relationships. The algorithm operates by comparing every Medicare enrollee with millions of similar people and then forecasting the likelihood of different medical complications. By pairing this forecast with known information—including, with users’ permission, secure access to the medications people take and the doctors they see, along with information they provide directly—Healthpilot then determines which plan is ideal and ranks the alternatives.
The algorithm also considers individual appetites for risk. “Some people have very little risk aversion, and they would rather have low payments now and gamble on how they fare,” Kolstad says. “The plan that gets recommended to this kind of person should be different than the plan that gets recommended to someone who is very risk averse, who wants a high premium now in order to know that they’ll be covered.”
Benefiting individuals and the marketplace at large
The financial benefits for individuals are straightforward: Choosing the right Medicare plan generally means better coverage and less expense. These savings also accrue to the federal government, which finances Medicare.
A subtler benefit are gains in well-being and even lower mortality rates. One working paper co-authored by Kolstad found that people who have to pay more out of pocket cut back on important health care services. A related paper, co-authored by Ziad Obermeyer of Berkeley Public Health with researchers at Stanford and Harvard, found that a $100 bump in per-month cost-sharing for drugs—exactly the kind of mistake people make without good guidance—increases mortality by 13.4%, as people forego essential drugs such as blood pressure medication.
Healthpilot is able to deliver these financial and health-related benefits to consumers for free because of the structure of the Medicare market. Since Medicare is a valuable source of revenue for insurance companies, the companies pay commissions to brokers for each person that they enroll. If Healthpilot sends someone to Humana, Humana pays; if instead the enrollee goes to Blue Cross-Blue Shield, then Blue Cross-Blue Shield pays.
That’s a crucial point: Because these commission amounts may vary by carrier, human agents may be biased in their plan recommendation based on the commission they are paid. Healthpilot’s algorithm does not factor commissions into its recommendations and does not steer people toward any particular plan or company based on financial incentives. “There’s no distortion in the platform or plan recommendation, which is unique in the industry,” Kolstad says.
This also has the potential to inspire greater innovation and efficiency in the insurance market as a whole—one of Kolstad’s main interests. As a point of comparison, consider the tech market: When a company like Apple creates a product that people like, they buy it; when it creates a product people don’t like, they don’t buy it. This is quickly reflected in the company’s revenue and share price.
Because insurance products are so much more complicated, consumer decisions rarely reflect clear notions about quality; people often enroll, and stay enrolled, in plans that don’t deliver value. Healthpilot, by sorting people into plans that genuinely benefit them, could bring much greater transparency into the marketplace and produce meaningful information for companies to build better plans, Kolstad says.
“For better or worse, we rely on competing private plans in Medicare. That’s the approach we’ve taken because we believe that a private market will offer innovation,” Kolstad says. “Giving customers a greater ability to match with plans that give them the coverage they want and need will reward innovators. That means Healthpilot isn’t just a digital enrollment solution for consumers but can be a tool to make the whole market function more as it should.”
An aerial view of Dominion Energy’s Scott Solar farm in Powhatan, Va. (AP Photo/Steve Helber)
The Haas School of Business is launching the first student-led Climate Solutions Fund, the latest addition to its comprehensive curriculum to equip the next generation of business leaders with the financial skills to accelerate the transition to a low-carbon economy.
Beginning in fall 2024, MBA students can enroll in a new course where they serve as investment managers for the $2.37 million fund, learning how to structure financing in complex private markets by co-investing in real-world deals focused on solutions to climate change.
“As the world moves toward a goal of net-zero carbon emissions by 2050, we need financial leaders with the skills to navigate the economic revolution we are facing,” says Professor Adair Morse, co-founder of theSustainable and Impact Finance Center (SAIF), who conceived of the fund and will lead the course. “This economic revolution will be staggeringly disruptive yet will also be a source of more business opportunities across all parts of the country than we’ve seen in 250 years.”
“As the world moves toward a goal of net-zero carbon emissions by 2050, we need financial leaders with the skills to navigate the economic revolution we are facing.” —Professor Adair Morse
The new fund was made possible by a lead gift from Allan Holt, MBA 76, along with generous founding donations from Larry Johnson, BS 72, Charlie Michaels, BS 78, and his wife Doris, Scott Pinkus, and Professor Laura D. Tyson, former Haas dean and co-founder of SAIF.
“I am thrilled to help Haas take the lead in training leaders in the emerging area of climate finance,” says Holt, a Senior Partner and Managing Director of The Carlyle Group. “Decarbonizing our economy is the critical issue of our time, and I am committed to supporting future leaders who can spur this transition.”
“Decarbonizing our economy is the critical issue of our time, and I am committed to supporting future leaders who can spur this transition.” —Allan Holt, MBA 76
The multi-asset class private Climate Solutions Fund augments Haas’ unique curriculum under SAIF, which teaches investment management with hands-on experiential learning. It rounds out the public markets-focused Sustainable Investment Fund—the first and the largest student-led sustainable investing fund within a leading business school—and the Haas Impact Fund, a seed/startup capital offering.
A new area of finance
The Climate Solutions Fund curriculum will teach students new designs and uses of finance not traditionally taught in mainstream finance courses, where there are dire needs for leadership, according to Morse, who saw the need for this financial expertise while serving as deputy assistant secretary of Capital Access in the U.S. Department of the Treasury from 2021-23.
Financing the climate transition requires a diverse and technical tool kit: An estimated $4 trillion to $5 trillion per year will be needed to reshape global energy, transportation, food, and waste infrastructure, and to help companies reinvent supply chains and integrate new technologies, Morse says.
“This level of reinvestment will require every finance tool available, including designing financial structures to mobilize government programs and work with community and industry partners,” she says. “Our goal is to expand how we teach students to provide the leadership and expertise that corporations, financial entities, startups, governments, and philanthropies will need to navigate this transition.”
“This level of reinvestment will require every finance tool available, including designing financial structures to mobilize government programs and work with community and industry partners.” —Professor Adair Morse
The fund, and the associated MBA course, are the first at a major business school to focus on complex financing strategies within private markets, including growth equity and debt equity; public-private partnerships with federal and state programs; risk mitigation; identifying the underlying technologies to fuel the low-carbon transition; and envisioning new financial products.
Students enrolled in the Climate Solutions Fund course will assess investment opportunities in U.S.–based for-profit companies, working with outside investment partners to structure deals. Following a pitch competition, student managers will select one finalist to co-invest $100,000 to $300,000 annually. The fund is intended to generate positive returns over time so that future generations of students can build off the capital.
Stock photo of a biogas plant and farm (Adobe Stock)
Comprehensive curriculum
In addition to the “fund-as-curriculum” courses, SAIF also offers other applied innovation courses such as the Impact and Climate Investing Practicum, where faculty guide small teams of MBA students who are paired with impact investing firms to to gain hands-on experience with impact investing strategy, mapping, and measurement projects.
The courses count toward theMichael’s Graduate Certificate in Sustainable Business. Open to both full-time and evening and weekend MBA students, the certificate requires 9 units of required coursework. Students can create a pathway that’s focused on either bringing a sustainability lens to a mainstream business function or building expertise into a specific industry such as renewable energy or green infrastructure.
In addition to Morse, SAIF is led by Professor Panos Patatoukas, The L.H. Penney Chair in Accounting, and Tyson.
Five major areas of sustainability
The new Climate Solutions Fund is part of Haas’ larger effort to ensure that all students are educated in the fundamentals of sustainability. Haas launched the first student-managed SRI fund in the early 2000s and is now the only top business school to work across five major sustainable business areas: energy, sustainable agriculture and food, real estate and urban economics, corporate accountability, and sustainable finance and accounting.
The school has combined research on energy conservation and storage, building efficiency, renewable energy sources, and sustainable food with efforts to include climate and equity into the core business curriculum across all programs. All told, Haas offers more than 25 courses with a focus on sustainability.
For students planning careers in managing sustainability challenges in organizations, Haas is also planning to launch a new joint master’s program in 2024 with the Rausser College of Natural Resources to offer an MBA/MS in Climate Solutions.
In a new working paper, Haas post-doc scholar Merrick Osborne examines how bias occurs in AI, and what can be done about it.
While artificial intelligence can be a powerful tool to help people work more productively and cheaply, it comes with a dark side: Trained on vast repositories of data on the internet, it tends to reflect the racist, sexist, and homophobic biases embedded in its source material. To protect against those biases, creators of AI models must be highly vigilant, says Merrick Osborne, a postdoctoral scholar in racial equity at Haas School of Business.
Postdoc Scholar Merrick Osborne’s new paper explores the dark side of AI
Osborne investigates the origins of the phenomenon—and how to combat it—in a new paper, “The Sins of the Parents Are to Be Laid Upon the Children: Biased Humans, Biased Data, Biased Models,” published in the journal Perspectives in Psychological Science.
“People have flaws and very natural biases that impact how these models are created,” says Osborne, who wrote the paper along with computer scientists Ali Omrani and Morteza Dehghani of the University of Southern California. “We need to think about how their behaviors and psychologies impact the way these really useful tools are constructed.”
Osborne joined Haas earlier this year as the first scholar in a post-doc program supporting academic work focused on racial inequities in business. Before coming to Haas, he earned a PhD in business administration at University of Southern California’s Marshall School of Business last year. In their new paper, he and his co-authors apply lessons from social psychology to examine how bias occurs, and what can be done to combat it.
Representation bias
Bias starts with the data that programmers use to train AI systems, says Osborne. While oftentimes it reflects stereotypes of marginalized groups, it can just as often leave them out entirely, creating “representation bias” that privileges a white, male, heterosexual worldview by default. “One of the most pernicious biases for computer scientists in terms of the dataset is just how well-represented—or under-represented—different groups of people are,” Osborne says.
Adding to problems with the data, AI engineers often use annotators—humans who go through data and label them for a desired set of categories. “That’s not a dispassionate process,” Osborne says. “Maybe even without knowing, they are applying subjective values to the process.” Without explicitly recognizing the need for fairer representation, for example, they may inadvertently leave out certain groups, leading to skewed outputs in the AI model. “It’s really important for organizations to invest in a way to help annotators identify the bias that they and their colleagues are putting in.”
Privileged programmers
Programmers themselves are not immune to their own implicit bias, he continues. By virtue of their position, computer engineers constructing AI models are more likely to be inherently privileged. The high status they are granted within their organizations can increase their sense of psychological power. “That higher sense of societal and psychological power can reduce their inhibitions and mean they’re less likely to stop and really concentrate on what could be going wrong.”
Osborne believes we’re at a critical fork in the road: We can continue to use these models without examining and addressing their flaws and rely on computer scientists to try to mitigate them on their own. Or we can turn to those with expertise in biases to work collaboratively with programmers on fighting racism, sexism, and all other biases in AI models.
First off, says Osborne, it’s important for programmers and those managing them to go through training that can make them aware of their biases and can take measures to account for gaps or stereotypes in the data when designing models. “Programmers may not know how to look for it, or to look for it at all,” Osborne says. “There’s a lot that could be done just from simply having discussions within a company or team on how our model could end up hurting people—or helping people.”
AI Fairness
Moreover, computer scientists have recently taken measures to combat bias within machine learning systems, implementing a new field of research known as AI fairness. As Osborne and his colleague describe in their paper, fairness uses complex mathematical formulas to train machine learning systems on certain variables including gender, ethnicity, sexual orientation, and disability, to make sure that the algorithm behind the model is treating different groups equally. Other processes are aimed at ensuring individuals are being treated fairly within groups, and that all groups are being fairly represented.
Organizations can help improve their models by making sure that programmers are aware of this latest research and sponsoring them to take courses to introduce them to these algorithmic models—tools such as IBM’s AI Fairness 360 Toolkit, Google’s What-If Tool, Microsoft’s Fairlearn.py, or Aequitas. Because each model is different, organizations should work with organizational experts in implementing algorithmic fairness to understand how bias can manifest for their specific programs. “We aren’t born knowing how to create a fair machine-learning model,” Osborne says. “It’s knowledge that we must acquire.”
More broadly, he says, companies can encourage a culture of awareness around bias in AI, such that individual employees who notice biased outcomes can feel supported in reporting them to their supervisors. Managers, in turn, can go back to programmers to give them feedback to tweak their models or design different queries that can root out biased outcomes.
“Models aren’t perfect, and until the programming catches up and creates better ones, this is something we are all going to be dealing with as AI becomes more prevalent,” Osborne says. “Organizational leaders play a really important role in improving the fairness of model’s output.”
As the director of product in his last role, Josh Martow, MBA 23, dreaded nagging his team members to make sure work got done. When he arrived at Berkeley Haas, he started mapping out an idea to solve his own problem, which led to the launch of startup Chaser.
In this interview, Martow explains how Chaser makes people more productive.
Could you give us a quick synopsis of what Chaser does?
Chaser follows up with your co-workers on the things they need to do. There are other project management tools out there that are supposed to solve this problem, but they break down because most teams struggle to constantly keep them up to date; no one wakes up in the morning and thinks, “I’m going to check Trello.”
With Chaser, you can delegate a task to anybody from within Slack. Chaser sends them the task, collects progress updates, and follows up until it’s complete. It works like magic because your co-workers never have to open Chaser. They don’t have to sign up for it or even know what it is! Their tasks just arrive in their inbox, and they can click “complete” right there.
How did you come up with the idea?
At my last job, so much of my work as a manager required making sure work was being taken across the finish line, which meant following up with people a lot. It’s not fun to be a nag and feel like you’re a babysitter to your team. It’s also not productive. And on the other end, no one likes to constantly receive these types of messages.
We’re doing with Chaser what Google did for calendars.
We’re doing with Chaser what Google did for calendars. It’s amazing that my friends and co-workers can put events right on my calendar for me and all of the sudden our calendars are synced up. Why doesn’t this exist for to-do lists? Imagine if your co-workers helped populate your to-do list for you and all you had to do was hit “complete” or “change date” and it would reflect on their end too, just like when you RSVP on Google Calendar, or move an event around.
How does a manager use Chaser?
So anyone can add Chaser to a Slack workspace. Once it’s added, just type “/todo,” tag the assignee, write the task, and include a due date, if there is one. The task will appear in your direct messages and Chaser will take it from there! You can follow along in your dashboard, which also lives inside Slack, while Chaser goes out and makes sure it gets done.
For now we’re actually offering Chaser for free, so everyone can give it a try here.
What was your background before coming to Haas?
Before Haas, I was the first employee at a startup Thriver Technologies. I got to wear every hat there. Throughout my time there I led sales, product, growth, and business intelligence. I was just running around doing whatever I could to help set up everything the company needed. We grew it to 150 people and raised a Series B, and after five years, I really got bit by that entrepreneurship bug and decided I wanted to do this myself. I ended up teaming up with the director of engineering and we set off to start our own thing.
Josh Martow presenting at the recent entrepreneurial network event, B-school Disrupt.
How has Haas helped you as an entrepreneur?
The two biggest things for me have been extracurriculars and classmates. For extracurriculars, some free accelerators connect you with mentors, help you hone your pitch, and help you figure out your business. After competing in one of the Demo Days, one of the judges, who was also a VC, ended up putting in $100,000 after hearing the pitch.
But my favorite thing about Haas is the Haasies. There are just a ton of great people who are interested in and want to talk about startups, and everyone comes from such diverse professional backgrounds. Just having people to bounce ideas around with is just so valuable.
What made you want to get an MBA to launch a company rather than launch without going to business school?
I didn’t study business as an undergrad and I wanted formal business training. I also needed time to develop more clarity and conviction around what we wanted to build.
But also, you hear that business school is a great place to start a company. And it’s 100% true.
Not having a full-time job gives you the freedom to explore, and being around Haasies realy helps you with that exploration. Not to mention access to the resources available, the accelerators, and being in the Bay Area. All of these things kind of just make it the perfect place to start a company.
What advice do you have for people considering launching a business while earning an MBA?
The No. 1 piece of advice is to cut out the things that are not 100% critical, and don’t succumb to FOMO when you see classmates doing things you just frankly won’t have time for. I definitely did not appreciate this enough at the start and was signing up for more than I could handle and would often be disappointed when I needed to miss out on things. It would have been a lot easier if I came into it understanding that you can’t do it all while you’re trying to get a startup off the ground.
That said, it’s a great experience, and it’s immensely valuable to be in school learning while thinking about your business and applying the things in class to your business every day.
Anew paper co-authored by Olivia Natan of Berkeley Haas and published in The Quarterly Journal of Economics peers into the black box of airline pricing and finds some surprises.
Photo: Fabio Principe/iStock
Buy your ticket on a Tuesday. Search in your browser’s incognito mode. Use a VPN to pretend you live in Suriname.
“There are so many hacks out there for finding cheaper airline tickets,” says Olivia Natan, an assistant professor of marketing at the Haas School of Business. “But our data shows many of these beliefs are wrong.”
With four colleagues—Ali Hortaçsu and Timothy Schwieg from the University of Chicago, Kevin Williams from Yale, and Hayden Parsley from the University of Texas at Austin—Natan looked deeply into the structure and processes behind how prices are set at a major U.S. airline. The system that she found, which is representative of airlines around the world, was strikingly at odds with what many economists would expect—and most consumers assume.
“We initially didn’t know how to rationalize the things we were seeing,” she says.
Substituting convenience for price
Consider fruit jam at the grocery store. Consumers have many options. If a company raises the price on its strawberry jam, one might fairly assume that this would affect sales of both strawberry and neighboring raspberry jam, since consumers can substitute one for another.
The same can happen with plane tickets: When people visit a website such as Google Flights or Kayak and search for a ticket, a wide range of different flights from the same airline appear. Travelers tend to make selections that balance convenience and price: The price of one flight might push people to select a slightly less convenient but cheaper flight.
“But the systems airlines use don’t consider this kind of substitution,” Natan says. They set the prices of seats on each individual flight on a given route separately, “even though changing the price on one flight will affect the way people think about all their options.”
A small menu of pre-set prices
Perhaps most surprisingly, airlines also don’t directly incorporate the prices of their competitors in their automated price-setting. Typically, if one airline cut its prices, one would expect other firms to do the same. If they don’t, this dampens the benefits of a competitive market.
Setting prices of each product separately without considering substitution, Natan explains, is the result of a specific pricing heuristic—or decision-making shortcut—that airlines use called Expected Marginal Seat Revenue-b, or EMSRb. This shortcut is widely used because it is fast enough to set prices for hundreds of thousands of flights daily, and it allows airlines to reserve some seats to sell at higher prices.
The use of EMSRb, the researchers show, results in another outcome that consumers may not expect. Despite how it may appear when looking for flights, airlines have a fixed and relatively small number of prices that they assign to tickets on each flight. Unlike other consumer sectors, where pricing can be adjusted and targeted down to the penny, airlines operate with large gaps between each possible price—sometimes upwards of $100. They may sell the first 30 economy tickets at the lowest price, and then the next 30 tickets at the next possible price, and so on.
“Airline tickets are sold through global distribution systems that make sure a travel agent in Wichita or Miami sees the same price as you do on your computer at home,” Natan says. This system emerged from an industry alliance to facilitate inventory management across many channels. Other businesses in the travel sector, such as hotel rooms, cruises, trains, and car rentals do the same.
The downside is that airline ticket prices are relatively unresponsive to real-time changes in opportunity costs, as the next discrete fare is often a significant jump up. The researchers found that even if the airline would like to increase the price by $100—half the price of an average one-way ticket—they only do so about 20% of the time, since no fare is available at that price.
Today, airlines are starting to experiment with what’s known as “continuous revenue management,” which would, for instance, assign 100 different prices to a flight with 100 seats. “That would make pricing significantly more variable,” Natan says, “but even that would not be the kind of targeting that many consumers assume airlines use.”
Lack of coordination across departments
One of the strangest discoveries from the research relates to the process airlines use to set their prices. To an economist, Natan explained, there is never a reason that firms would not raise prices if the increase assures an increase in revenue. But the set of possible prices chosen by the pricing team nearly always includes an option which is too low, even by their internal estimates.
The pricing team’s work is made difficult by having to choose an entire menu of discrete prices, “but we found they could make more money today by selling fewer tickets at higher prices and not foreclose future opportunities. In practice, they choose the menu of prices without using their internal demand predictions,” Natan says.
Interestingly, the revenue management team corrects much of this underpricing before it ever reaches consumers. After prices are filed and before tickets go on sale, this team makes demand forecasts that determine final prices. These forecasts are routinely inflated, reducing the number of underpriced tickets shown to consumers by roughly 60%.
“We find that these prices are a consequence of teams from different departments choosing the best pricing inputs when they are unable to coordinate,” Natan says “This may result in lower revenue, but in practice our solution, which sells fewer tickets at higher prices, could not be implemented. ” Two other possibilities as to why airlines don’t only focus on short-term revenue, she speculated, are either to build customer loyalty or to avoid regulatory scrutiny.
Over the next several years, Natan says, airlines may start to adopt more dynamic pricing platforms, and non-business travelers may benefit from these changes. But for now, the hunt for an undiscovered trick to find lower fares is largely futile. What is clear is that it’s wise not to wait until the last minute. “What I can say is that prices do go up significantly 21, 14, and seven days before a flight,” Natan says. “Just buy your ticket before then.”
Note: This article has been updated from the original version published October 4, 2023.
Olaf Groth teaches Management of Emerging Technologies to Haas MBA students. (Photo: copyright Noah Berger)
Olaf Groth doesn’t like to make predictions, but he’s nonetheless become adept at helping executives see around corners and navigate the chaos of recent years.
A member of the Berkeley Haas professional faculty, a senior adviser at the school’s Institute for Business Innovation, and the faculty director for the Future of Tech program at Berkeley Executive Education, Groth is also chairman and CEO of thinktank Cambrian Futures.
In a new book, The Great Remobilization: Strategies and Designs for a Smarter World (MIT Press), Groth and coauthors Mark Esposito and Terence Tse (and editor Dan Zehr) focus on what they call the Five Cs—COVID; the cognitive economy, crypto and web 3; cybersecurity; climate change; and China. They show leaders how to power human and economic growth by replacing fragile global systems with smarter, more resilient ones.
In one chapter, the authors imagine a harrowing scenario they call “the Great Global Meltdown of 2028,” the economy collapses after a hack of a critical global payments system in the midst of recurring climate and COVID crises; nearly 1 billion people suffer mental health issues from the strain.
But with the help of the ideas in the book and a bit of practice, leaders can learn to “flip the chaos forward into opportunity,” the authors argue. The book, shortlisted for the Thinkers50 Strategy Award, launches on Oct. 17.
We talked with Groth about how executives can learn to turn turmoil into opportunity.
Your book talks about the need for humanity to make the leap from the “raging 2020s” to a “great remobilization” in the late 2020s and beyond. What will need to happen?
We’re looking at utter volatility over the next 10 years. We can’t afford to sit on the couch and just watch. We have an opportunity now to redesign how business leaders generate growth—both for their shareholders as well as for a broader set of stakeholders—that takes into account how the global economy has changed and continues to change. The book is all about: Where do we go from here? Are we going to accept that we’re going to have a fragmented world where people don’t trust each other, where trade and investment are inhibited, and where growth is limited?
“We can’t afford to sit on the couch and just watch. We have an opportunity now to redesign how business leaders generate growth—both for their shareholders as well as for a broader set of stakeholders—that takes into account how the global economy has changed and continues to change.”
What will the next era of globalization look like amid an era of rising economic nationalism, fragile supply chains, and recurring pandemics and natural disasters?
The globalization everyone keeps talking about is how many bananas get shipped from Argentina to China with the greatest degree of efficiency. And of course that’s important for jobs today. But what we really have to ask is, who controls the flow of any given thing from one place to another? At the end of the day, the people who have the power in what we call the “cognitive economy” can influence these flows. There are traditional flows—of capital, of intellectual property, of people, of goods and services. But there are also new flows of behavioral data, genetic material, and crypto currency. And that’s not even including the ecological flows of energy, air, water, and pathogens.
You define the cognitive economy as a world of algorithms automating or augmenting human decisions and tasks and spinning out valuable insights that can be used to improve businesses, governments, and society. What’s at stake for managers and leaders in a cognitive economic future?
In the cognitive economy, cybernetics functions are getting injected into everything we do—essentially intelligent, digital, command-and-control functions that are at the moment centrally steered by the likes of Google, Amazon, Baidu, and Alibaba. We need to see the new principles bubbling up now from the cognitive economy. Leaders need to be able to perceive the new operating logic in their domain and in their industry.
Take supply chains: 46% of supply chain managers are saying that investments in AI and automating human decisions is their top priority. That means we’ll see new types of integration of human and machine predictions and decisions. Or take the mobility industry. Electrified, autonomous cars use AI to create applications inside the car, and cars are now starting to be tied to a new charging infrastructure around the smart home. You need to design cars like they’re an iPhone: you start with a chip and you build everything around that. Essentially, the automotive industry is being scrambled, as is the energy industry, the home appliances industry, etc. Who controls these complex systems? How do you see those new patterns if you’re an executive any given industry?
Olaf Groth in the classroom teaching his Management of Emerging Technologies class to MBA students. (Photo: copyright Noah Berger)
Your book describes a FLP-IT (forces, logic, phenomena, impact, and triage) model for strategic leadership. How should executives approach the triage step, which requires them to make tough decisions about what to keep, discard, or build from scratch?
I was at the World Economic Forum meeting in Tianjin and a senior executive in the petrochemicals industry told me that after 30 years of building megafactories in China, his company now needed to throw that out the window and create chains of “nanofactories” across 12 different markets to create resiliency. To solve that kind of challenge you need to understand what to invest in and what to divest of. Because strategy is just as much about what you do as it is about what you no longer do.
Most executives are bad at dropping things that they hold dear. They need to make conscious choices.
“Most executives are bad at dropping things that they hold dear. They need to make conscious choices.”
Once you understand the new operating logic in your industry or domain, you need to redraw its boundaries and diagnose changed power relations. To address those changes, you have to then free up resources from activities that are not aligned with the new arena to create new ones that are. For instance, if you are a food conglomerate with most of its agricultural activities in the global south, how will you need to reconfigure that for massive labor migration coming in the face of climate change? Do you have the cognitive technologies, including AI, data science and automation in place to make your supply chains more intelligent, agile and resilient?
You argue that China is an indispensable partner for any global systems redesign. How will leaders in the rest of the world need to reimagine their engagement with a rapidly aging and slowing China while navigating rising economic and military tensions?
Not every investment in China is bad and we’re not advocating for a wholesale withdrawal from all activities in China. That total decoupling is not desirable for anybody. In fact, there are lots of new opportunities for joint solutions that serve both our economic and national security interests as Americans and Chinese. Let’s talk about the fact that China’s coastal cities are as screwed as America’s cities are with sea levels rising. Hundreds of millions of people will be migrating north. That could mean joint investment in climate change mitigation and adaptation solutions, in new approaches to education, in regulating and managing migration, in innovation for public health and early pandemic detection and warning systems.
At some level, we’re going to get down to technologies that are sensitive, like chips and AI. And granted, you can weaponize almost anything these days. But there are technical ways to govern that, design protocols for dispute resolution and risk mitigation. We can watermark and forensically trace content, technical components and code. We can figure out where specialized microchips are going. We can put protocols in place to jointly identify and settle infractions and misunderstandings. It’s not perfect, to be sure. Nothing ever is. But we have to try, because the alternatives are even worse.
So there’s some optimism in your message?
I have a conviction that a lot of these problems are truly solvable with the right design. Executives need to get out there and try things with risk-bounded experiments. This is a craft you can learn. We give leaders a roadmap to lead people into a new future that’s better than one of distrust, fear, and fragmentation.
Incentivizing online reviews may not give you the types of reviews you want.
Incentivized reviews don’t improve—and may even hurt—business.
Photo: AdobeStock
New research focused on whether incentivizing would-be online reviewers to post reviews can make a difference for the seller has shed light on the role those reviews play in sales and revenue outcomes.
Researchers worked with Airbnb to conduct a large-scale and costly experiment to find out just how those incentivized reviews impact day-to-day marketing. They found that although incentivizing reviews does lead to an increased number of reviews, the induced reviews tend to be more negative in nature, and have no impact on the number of nights sold or total revenue.
The study published in the INFORMS journal Marketing Science, “Do Incentives to Review Help the Market? Evidence from a Field Experiment on Airbnb,” is authored by Andrey Fradkin of Boston University and David Holtz of the Haas School of Business, UC Berkeley, and the MIT Initiative on the Digital Economy.
The Airbnb experiment divided participants into two groups: treatment and control. The treatment group was sent an email offering them $25 Airbnb coupons in exchange for a review if they had not already posted a review within the typical eight or nine days after checkout. The control group was sent an email simply reminding them to leave a review with no financial incentive offered.
“We found that reviews induced by incentives were more likely to contain information about lower quality experiences,” Fradkin says. “We also found that when compared to non-incentivized reviews with the same star ratings, incentivized reviews corresponded to worse guest experiences. In other words, a seller may have received the same star rating from both study groups, but the incentivized guest was more likely to post a negative review, while upholding an inflated star rating.”
“While the incentivized review program impacted the number and type of reviews left by guests, it had neutral-to-negative impacts on the actual business outcomes of the platform and its sellers,” says Holtz. “Incentivized reviews did not affect the total quantity (nights) sold for study listings, but they did cause a change in the composition of those transactions. More to the point, incentivized listings led to more transactions, but those transactions lasted fewer nights on average. Thus, a poorer match.”
The study found that as a consequence, the revenue impacts of incentivized reviews did not represent either a positive or negative material change. Incentivized reviews failed to improve transaction quality, and according to some measures, resulted in worse matches. At the same time, the incentivized treatment did not lead to higher complaint rates or reduced likelihood of repeat business.
“Ultimately, we found that while reviews do play a meaningful role in the overall marketing efforts for online sellers, incentivizing those reviews may not deliver the results many would assume,” Fradkin says.
Read the full study.
About INFORMS and Marketing Science
Marketing Science is a premier peer-reviewed scholarly marketing journal focused on research using quantitative approaches to study all aspects of the interface between consumers and firms. It is published by INFORMS, the leading international association for operations research and analytics professionals. More information is available atwww.informs.org or@informs.
Xepelin, co-founded by Sebastian Kreis, MBA 18, rocketed to No. 3 on Poets & Quants’ 2023 Top 100 MBA startups list this year. The company, based in Mexico City, has raised $567 million so far and will use the funds to invest in new markets in Latin America.
Haas News asked Kreis a few questions about his startup’s success.
Tell us about Xepelin. Xepelin was founded in 2019, the year after I graduated from Haas. Our aim is to be the “digital CFO” for small- to mid-size companies, offering an online platform that helps businesses organize their accounts and automate payments to suppliers and payment advances from customers.
You’ve had quite a lot of success getting the company funded over the past year. Tell me a little about that.
Last year, the company secured $150 million in equity and a $140 million credit line from Goldman Sachs that we are using to expand software, payment, and working capital services in Mexico. We are headquartered in Mexico City and now have 400 employees.
How big is the market for your services and what’s your expansion plan?
The companies we are targeting account for over 60% of Latin America’s GDP. These companies present an enormous opportunity for us, with more than $10 trillion in unmet needs. High acquisition and servicing costs have kept them underserved. Xepelin is committed to equipping these companies with efficient access to software tools, payments, and working capital.
The companies we are targeting account for over 60% of Latin America’s GDP.
What resources at Haas helped you become an entrepreneur?
There were two resources I tapped at Haas: learning from entrepreneurs and investors in the Bay Area who had already built successful startups, and working on fintech projects with Lecturer Greg La Blanc. I traveled to Mexico several times while I was at Berkeley because of the size of the market. I studied the metrics, such as credit and software market penetration, before committing to building a regional company, starting in Mexico and Chile.
Dean Ann Harrison was named 2023 Dean of the Year by Poets and Quants. Photo: Brittany Hosea-Small.
Berkeley Haas Dean Ann Harrison, a renowned economist lauded for keeping Berkeley Haas’ six business programs ranked among the world’s best and significantly expanding the breadth and depth of the faculty, has been named Dean of the Year by Poets & Quants.
Poets & Quants Editor-in-Chief John Byrne announced the news to a global audience at a Thinkers50 virtual conference today. Byrne engaged in a sweeping conversation with Harrison that covered the impact of globalization on workers, the responsibilities of government and business in fighting climate change, the critical role of diversity on campus, and the enduring importance of the MBA.
“The MBA is a wonderful degree,” Harrison said. “It combines the rigor of statistics, data analysis, hard-core quant skills with skills like how to work with people, marketing, and how to sell,” she said. “What’s wonderful about an MBA is that it allows you to combine these different skills. Other degrees don’t offer that combination.” Haas MBA students care about making an impact, she added, “not just a great paycheck.”
Harrison has amassed an unimaginable and nearly breathtaking record of achievement. —John Byrne, Poets & Quants
In his Poets & Quants article, published today, Byrne wrote that Harrison has amassed an “unimaginable and nearly breathtaking record of achievement” during her four-and-a-half years as dean. Harrison, who has led Haas since January 2019 and was reappointed to a second term last February, said she was “deeply humbled” by the honor.
“I am so lucky to be surrounded by a tremendous community at Haas—students, staff, faculty, and alumni who are always going beyond themselves,” she said. “It’s only together that we can seek solutions to climate change, build a more inclusive society, and fuel innovation in all its forms. This is a business school that embodies excellence. I feel great pride in our past and am thrilled to have the opportunity to create impact for the future.”
Harrison is the second woman to lead Haas; Professor Laura Tyson served previously. As the former director of development policy at the World Bank and a longtime professor, Harrison has focused her research on international trade and global labor markets.
Big Changes
Since joining Haas from Wharton, Harrison has made big changes, Byrne noted. She has led a major diversity, equity, inclusion, justice & belonging (DEIJB) effort, broadening the profile of the Haas faculty, school board, and the student body. The school’s entering full-time MBA class this year is 41% women, 47% U.S. minorities, and 13% U.S. underrepresented minorities overall.
Harrison has woven sustainability content deep into the curriculum while maintaining the school’s historical focus on entrepreneurship and innovation.
“The challenges of climate change permeate all aspects of business: supply chain, economics, management, and finance,” she said. “In the latter field, we have pioneered new ways of investing. We need to hire in all these dimensions. It is a big agenda and we are making a lot of progress in a lot of different ways.”
Harrison also oversaw the launch of the first Flex online MBA cohort at any top business school. Applying learnings from the pandemic, Haas used new technology to make the MBA available to expanded groups of international students and working parents who require flexible schedules.
Under Harrison’s leadership, Haas also stepped up fundraising, raising a record total of $227 million, including $56.1 million during the last fiscal year. The school also secured the largest single gift in the school’s history—$30 million from alumnus Ned Spieker, BS 66, and his wife, Carol, BS 66—to turn the undergraduate program into a four-year program.
Figuring it out together
In the Poets & Quants article, Courtney Chandler, Chief Strategy & Operating Officer and Senior Assistant Dean at Haas, noted that Harrison “hasn’t stayed in one lane as dean.”
“She’s ambitious, and she sees the full potential of Haas within UC Berkeley and is driven to realize that potential,” she said. “She has not been that one-dimensional dean and that is incredibly impressive.”
“She’s ambitious, and she sees the full potential of Haas within UC Berkeley and is driven to realize that potential.” —Senior Assistant Dean Courtney Chandler
Harrison’s record as a highly cited scholar has also helped her lead the school’s professors, Byrne said.
“It’s hard to get faculty to buy-in to a dean’s vision, but she has been able to do that effectively,” Erika Walker, senior assistant dean for instruction, who has been at Haas for nearly 20 years, told Byrne. “She relates so well to them…. Ann is very thoughtful about where we should be going. A lot of her success stems from her ability to get the buy-in and then enlist others to figure it out together.”
Professor Catherine Wolfram (left) talks with Dean Ann Harrison (center) and Professor Ulrike Malmendier (right) at a Dean’s Speaker Series event.
During her second term, Harrison said she will continue to work with her team to build upon the school’s academic excellence as well as the student experience at Haas. One important goal is to ensure that the school’s degree programs remain the best in the world, she said.
Harrison earned her BA from UC Berkeley in economics and history, and her PhD from Princeton University. She held previous professorships in UC Berkeley’s College of Agricultural and Resource Economics, as well as at Columbia University and the Wharton School, where she was the William H. Wurster Professor of Management.
An avid hiker, Harrison told Poets & Quants that returning to UC Berkeley and California has allowed her to use time off to explore the state’s cliff-lined beaches, redwood forest, and the Sierra Nevada Mountains, “a paradise for those who love the outdoors.”
Harrison is the 13th dean and the third woman to be named Dean of the Year by Poets & Quants, which covers business school education.
Khan Academy CEO and Founder Sal Khan traced his journey to provide free and accessible online education to kick off the fall Dean’s Speaker Series Sept. 19.
With more than 150 million users in more than 190 countries since its founding in 2008, nonprofit Khan Academy has become an educational staple for students and instructors alike.
Khan Academy CEO and Founder Sal Khan traced his journey to provide free and accessible online education during the first fallDean’s Speaker Series talk Sept. 19.
After graduating from Harvard Business School in 2003, Khan worked as a hedge fund analyst, having told his friends “I’m going to do this long enough so that I could become independently wealthy.” Around the same time—having always been passionate about education—he began tutoring his cousin over the phone using an interactive notepad. From there, Khan expanded his lessons to include family and friends, even writing software to create deep item banks—databases of questions—to help them practice.
With advice from a friend, he began incorporating video, uploading his lessons to YouTube—and by 2008, Khan Academy reached almost 100,000 users. Even with the nonprofit’s initial traction, however, Khan noted the risks that come with starting any new business.
“I think anytime you do anything entrepreneurial—for-profit or nonprofit—you have to start with that delusional optimism. You assume that surely the market, the investors, the philanthropists, whatever, whoever the stakeholders are, they’re going to recognize the value of what you’re creating. And oftentimes, it doesn’t happen as fast as you think,” Khan said.
I think anytime you do anything entrepreneurial—for-profit or nonprofit—you have to start with that delusional optimism.
Despite receiving offers from venture capitalists, Khan turned them down to maintain control and ensure his organization remained free and accessible. Nine months later, current Khan Academy Board Chair Ann Doerr made an initial donation, which led to a lunch meeting and eventual working relationship.
After Bill Gates talked about his own kids using the platform at the 2010 Aspen Ideas Festival, Khan had another meeting to discuss ways to support Khan Academy’s future. This, coupled with more funding from Google, allowed Khan Academy “to become a real organization,” Khan said.
Since then, Khan Academy has expanded to include more than 70,000 practice problems with translations into more than 50 languages. The nonprofit is also partnered with upward of 500 school districts, even having its own accredited schools: Khan Lab School in Silicon Valley and Khan World School, a partnership with Arizona State University. Now, Khan Academy is launching Khanmigo, an AI educational chatbot, to help instructors and students understand how to use AI to support learning.
In terms of running and scaling a nonprofit with an impact like Khan Academy, Khan cited the importance of teams maintaining a clear mission.
If you can grow and maintain your talent and your alignment, a 300-person team can outperform a 30,000-person team.
“The hardest thing is, as you grow, to maintain alignment. If you can grow and maintain your talent and your alignment, a 300-person team can outperform a 30,000-person team,” Khan explained. “Alignment is not just communication. And this is a muscle that I don’t think people in Silicon Valley exercise enough—being very clear with team members: ‘Look, this is what we’re doing. If you’re excited about that, awesome. If you’re not, this might not be the place for you.’ That is kind of strong language, but I have found it to be very powerful language because otherwise people are just trying to pull (the organization) in all sorts of different directions.”
Watch the full DSS talk below.
Read the full transcript:
– Good afternoon, welcome. I’m Don Moore, acting dean here at the Haas School of Business. I’m thrilled to welcome all of you to the first Dean Speaker Series of the year. Today’s guest, as you know, is Sal Khan, founder and CEO of Khan Academy. Sal holds two bachelor’s degrees of science and a master’s in electrical engineering from MIT and an MBA from a school called Harvard Business School, which you may have heard of. Sal became passionate about education while he was an undergraduate at MIT. He developed math software for children with ADHD and tutored fourth- and seventh-grade public school students. While working as a hedge fund analyst in 2004, Sal began tutoring his young cousin in math over the telephone with an interactive notepad. Within a couple years, he added more than a dozen relatives and family friends to his online classes. In 2008, he founded Khan Academy with a mission to provide free, world-class education for anyone, anywhere. Khan Academy now partners with more than 500 school districts and schools across the United States to help teachers tailor instruction to each student. Khan Academy is now used in more than 190 countries. There are more than 150 million registered users around the world and translations into more than 50 languages. Today, Khan Academy’s platforms include more than 70,000 interactive practice problems, as well as videos and articles that cover a range of subjects from pre-K to 12th grade. Khan Academy is piloting a new tool called Khanmigo, which gives students an entirely new way to use AI to learn. He also wrote a book, The One World Schoolhouse where he presents his radical vision for the future of education with a goal of leveling the playing field for access for all to world-class education. Thanks, Sal, for taking the time to join us today and to talk with us about your work. Now I’ll turn it over to today’s moderators, MBA students Anu Tej and Mrudula Vemuri.
– Thank you for that introduction. I’d like to just introduce myself one more time. My name is Mrudula Vemuri. I’m an EWMBA class of 2024. And I’d also love to share my first story using Khan Academy. I was desperately in need of help with my calc homework, and I had a calc final looming in college, and I turned to Khan Academy, and it hasn’t failed me ever since. Even through my macro and finance courses here at Haas School of Business, I’ve used Khan Academy, and it’s helped me all along the way, so I can’t wait to see what they have in the future.
– OK, I am Anu and I’m a full-time MBA class of 2024. I first learned about Khan Academy just after my undergrad. I had joined this early stage startup, and I was given a task to do data analysis, and I had no idea how to do it, and one of my teammates came to me and said, “Oh, you might want to check out Khan Academy for some of the concepts.” And I started with high school statistics that night, and I was like, “Oh my God, I needed this in college. I needed this in high school.” All this while I thought I suck at statistics, I did not. I was able to do that analysis that I did; and like Mrudula, I was able to pass my macro economics last year because of Khan Academy. Thank you so much for helping us. So I want to start,
– [Sal] Always good to hear the stories.
– I want to start with the origin story, Sal. I know you were tutoring students during your high school, and I also know that you will tell your friends when you are working in a hedge fund that ‘I want to work in hedge fund long enough to start my own school someday.’ So tell us more.
– Yeah, first of all, thanks for having me here, and as I said, always great to hear y’all’s stories. Yeah, I’ve always had this interest in education. I think going back to maybe even elementary school where we all had good moments in our education experience; and then we all had some not-so-good moments. And usually, some of those not-so-good moments are feeling a little bit bored or feeling disengaged or seeing some of your friends bored or disengaged. And so I’ve always been fascinated that there’s got to be a better way to do this. Anyway, you fast-forward to, I graduated from undergrad. My first career was in tech. Then after that, I go to business school. After business school, I find myself, I’m at a small hedge fund in Boston at the time, and I graduated from business school in 2003, and it’s now 2004. I had just gotten married. My family was visiting me up from New Orleans, which is where I was born and raised up in Boston. And even before that, actually, to answer your question directly, when I was working at the hedge fund—and this is what I did tell myself—but I would tell my friends who were like, “Well, what’s your lifelong goal?” I was like, “Well, I’m going to do this long enough so that I could become independently wealthy,” which I have not achieved, but I did the other, well, the thing I wanted to do after I wanted to be independently wealthy, so I can start a school on my own terms. I thought that would be the most fun thing to do, to be one day, be like a Dumbledore-type figure, but once again, in some ways have to convince anyone that, and we could talk more about, there’s some benefits of having to convince people things. But when it became clear that my cousin Nadia needed help, she didn’t even know that she needed help, but I knew that she needed help. She was in seventh grade, and she was put into a slower math track, which had all sorts of potential negative implications for her future. I offered to tutor her remotely when she went back to New Orleans. She agreed. And so, yeah, we got on the phone. We used to use instant messenger at the time to just communicate. We could see each other’s writing, et cetera. Slowly but surely, she got caught up with her class, even ahead of her class. At that point, I become what I call a tiger cousin. I called up her school, I said, “I really think Nadia Raman should be able to retake that placement exam from last year.” They said, “Who are you?” “I’m her cousin.” And they let her take it. And the same Nadia who was being tracked into a remedial track was now put into the advanced track.
So I was hooked. I started tutoring her younger brothers, Armand and Ali, and it was working with them, and word spreads in my family, free tutoring is going on. So before I know it, there’s 10, 15 cousins, family friends that I’m tutoring on a regular basis. My day job, I was at the hedge fund, but markets close. I’d get on conference calls with, for the most part, my cousins, a few family friends.
But that’s how I started, and I was enjoying it. I was connecting with family, but I was also, I believed that there’s a way of approaching a lot of this subject matter that could be very intuitive and even enjoyable for my family. I also saw that there was a common pattern, and I saw this pattern throughout my education. The reason why folks struggle isn’t because they aren’t bright. It isn’t because they don’t have good teachers or they’re not hardworking. It’s usually because they have gaps in their knowledge. And by the time you get to an algebra class, it’s really hard to address those gaps of, say, dividing decimals or negative numbers. And those gaps happen in a traditional system because kids just keep getting moved ahead, even if they get a 60 or 70 or 80%, or they forget stuff. So that’s when I started writing some software for my cousins, too, ’cause I honestly couldn’t find any good practice on the internet at the time. This was back in 2004, 2005. Because I wanted them to get as much practice as they need. So I was creating these very deep item banks that were partially computer generated, et cetera. And that was the first Khan Academy. It had nothing to do with videos at the time.
Then in 2006, by this point, we had moved out to the Bay Area. I was at a dinner party showing off the software. I’m a super cool dinner party guest, and the host of the dinner party, his name’s Zuli, I have to give him full credit. He said, “Well, this is cool, Sal, but how are you scaling your lessons?” And I said, “Yeah, it’s hard to do with 15 cousins what I was originally doing with just a handful.” And he said, “Well, why don’t you make some videos, upload them onto YouTube for your family?” I told him, “That’s a horrible idea. YouTube is for cats playing piano. It’s not for serious math.” But I got over the idea that it was not mine. And I gave that a shot. And that took on a life of its own, obviously. My cousins famously told me they liked me better on YouTube than in person. I think what they were saying was that they enjoyed having an on demand, not feeling shy, not feeling embarrassed. If it’s 11 p.m. and they had a question, they could access it. I believe they still enjoyed being able to get on the phone with me to go a little bit deeper. But then a lot of other people started discovering it. And then by 2008, there were about 50 to 100,000 folks using it on a monthly basis. And that’s where I incorporated it as a not-for-profit, a mission-free, world-class education for anyone anywhere. 2009, I frankly had trouble focusing on my day job. We had some savings at this point. My wife was a medical fellow, so she was still in training. Our first child had just been born, so our expenses were going up, but we were saving some money for a down payment out here in the Bay Area, which we all know is not a joke, even back in 2009. But we felt, well, maybe if we give it a year, hopefully someone will fund this philanthropically. So that’s when I took the plunge, and that first year wasn’t easy, but eventually, it worked out.
– Well, thank you. I know that I have personally shamelessly rewinded many, many videos. In my personal experience interacting with Khan Academy, I noticed that there’s a huge breadth of topics. Were you familiar with all of these topics before the beginning of Khan Academy? And if not, how did you teach yourself so many different things?
– Great question. The simple answer is no. I think that there, with that said, well, I can surprise folks with my breadth. Let me just say that I’m definitely above average. But there’s certain topics that I did know very, very well, math being one of them. I would say math and physics and some other parts of science I knew very, very well where I could kind of get into it. I didn’t need necessarily a reference or a textbook.
With that said, once Khan Academy became a more professionalized organization and we wanted to make sure we covered all of the standards, we now have a team who’s curating the questions and figuring out the scope and sequence and instead telling me, “Hey, they’re using a different term nowadays for,” I mean, the most famous example is when I was in school, it was called atomic weight. Now it’s average atomic mass. Or let’s redo a video. So there’s definitely support now, but in those early days, but even now, I take joy; and maybe there’s folks in this room who feel the same way that if you learn to learn something well, that that is actually a very transferable skill to almost anything. And it goes against what I think most people are indoctrinated into in school. My biggest pet peeve is when I hear someone say, “I’m a math person,” or “I’m a humanities person.” And unfortunately, you’ll sometimes hear it from educators. In fact, you’ll almost always hear it from, you want to give a math teacher anxiety, ask them to engage in the humanities, or you want to give humanities teacher anxiety, ask them to engage in math. And even at a high school level, like, you graduated from high school, right? Like, this shouldn’t be something… So unfortunately, I think sometimes that not optimal signaling happens to students.
But I think generally speaking, if you learn to understand something really well in any domain and you build that muscle of, like, asking the right questions: Wait, how does A lead to B? I’m not just going to take it as a leap of faith. I need to understand it, prove it to me. How does this make intuitive sense? You can take that to almost any domain. And I’ve always been, some of the subjects in Khan Academy are things like history, and I’ve always really been fascinated by history. I don’t have any formal training in it, but I’ve been fascinated both on just, like, these are real human stories that happened. How can it not be interesting? But then also I’ve always been sometimes frustrated on not being able to place things in time and space and seeing timelines and things like that. So I’ve tried to take that tack and when we try to cover some of that on Khan Academy, and we’ve gotten positive feedback. And then there’s certain even science topics that, when I took it in college, I took organic chemistry in college. It was not a class where I was like, “Oh, this makes intuitive sense.” I got through the class, I got a fine grade, I did what I needed to do, but I didn’t get out of that class thinking like, “Oh, I see the beauty in organic chemistry.” So when it was almost, as a challenge, some of those organic chemistry videos are still up there on Khan Academy. When I did this almost 10 years ago, I said, “Before I even press record on that first video, I’m going to just sit and immerse myself in organic chemistry for several weeks until I learn to love it, until I learned there must be an intuition behind it.” And there is and actually, it all boils down to electronegativity, actually. You don’t have to memorize as many mechanisms as you think. And I would call up friends and stuff like, “Explain this to me. Why is this happening?” And someone would say, “No, we don’t know. That’s an area of research.” I was like, “Well, someone should tell the students that too, so they don’t bang their head on a table and they’re like, I’m actually trying to figure out an area of open research.” So yeah, I don’t know it all, for sure. And I do use Khan Academy when I’ve already made 80 videos in a certain course and they want an 81st. I sometimes have to watch some of the videos that I already made in order to get up to speed.
– Oh my God, thank you so much for that. Can you tell us, I mean, you briefly took us through from your undergrad, to, like, starting of Khan Academy for your cousins and, like, the whole journey ’till today? But can you tell us what would you consider your first big break in your life? This can be before Khan Academy or after.
– Oh, big break. Oh, so many breaks. I’ll give you more than one. Probably the first is when I was in second grade, my older sister was three years older than me, and she was, like, the star student. She was in the gifted program. I thought I was in the gifted program because they take you out; they took me out of class every day into some kind of enrichment thing. It turned out it was speech therapy. So I was being in a different, I was put in a different category than my sister, but because I was Farah’s younger brother, I think the school said there must be something there. And so they kept testing me. And so the first break was I went from speech therapy into this kind of gifted enrichment program that they had. It was Louisiana Public Schools, not famous, but it was a really good program. I still remember that first day, 7 years old, I walked into a classroom, unlike any other classroom I had ever seen, there were just, like, five or six kids in the room. Some kids were playing chess, some kids were drawing, some kids were playing “Where in the World is Carmen Sandiego?” And there were two teachers. I mean, it was well-resourced if you really think about it in hindsight. And they’re like, “Oh, Sal, come on in.” And I walked up to both the teacher’s desks and they said, “Oh yeah, so what are you into?” And I said, “Well, I really like drawing. I really like puzzles.” And so they’re like, “OK, so we’ll draw and do puzzles.” And I didn’t even want to tell anyone about this thing ’cause I thought it was some type of, like, secret racket going on inside of my school that would be taken away from me if people knew what it was. But that I consider a break because one, that opened my eyes to what school could look like if everyone was really engaged and interested. And I think most of my memories from elementary school are that, so it had a real impact on me.
I mean, there’s a bunch of other breaks. I would say another, I’ll give my sister credit. When I was in, we didn’t have a lot of money, raised by a single mother. My mom was essentially making minimum wage eventually. And I remember when my sister was applying to college and I asked her, “What’s your first choice?” She said, “Brown University.” And I said, “You’re nuts. Brown University’s tuition is twice what our mom makes per year. Like, there’s no way that you’re going to Brown University.” And then my sister explained to me about financial aid and this and that. And my sister got in and was able to go. I mean, she still had some significant debt, and I think it’s gotten more generous since then, but it was very generous. It’s much better than what we thought it would be. So, that I considered a break of sorts ’cause that opened up my aperture to what’s possible.
You fast-forward, I mean, I’ll give a couple, well, I’ll fast-forward to the Khan Academy ’cause I think there was a big break there, which is, I talk about quitting my job. Me and my family are now living off of savings. I think anytime you do anything entrepreneurial for-profit or nonprofit, you have to start with that delusional optimism. You assume that surely the market, the investors, the philanthropists, whatever, whoever the stakeholders are, they’re going to recognize the value of what you’re creating. And oftentimes, it doesn’t happen as fast as you think. When I quit my day job, I was talking to some philanthropists, but very quickly, those pitted out. They really didn’t know what to make, how to make sense of what we’re trying to do at Khan Academy. And about nine months into that, it was stressful. I was waking up in the middle of the night. We were digging into our savings, about $5,000 a month. It was quickly not a down payment on a house anymore. And I was really questioning every, I’d given up a good career that I, for the most part, enjoyed. And then it was, yeah, about nine months in, all of a sudden, a $10,000 donation came in. I immediately emailed the person, her name is Ann Doerr. She’s now our chairperson. But at the time, I said, “Dear Ann, thank you so much for this incredibly generous donation. This is the largest donation Khan Academy has ever received. If we were a physical school, you would now have a building named after you.” And Ann responded back and said, “Well, that’s surprising to me. It’s actually surprising to me you’re a one person operation. I thought this was, like, a real thing. I see you’re in Mountain View. I’m in Palo Alto. I’d love to have lunch and learn more about what you’re doing.” So we have lunch, and Ann kind of asked me, “What’s your goal?” I said, “Look, I filled out free, world-class education for anyone anywhere.” And says, “Well, that’s ambitious. How do you see yourself doing that?” And I said, “Well, I’m building out all these exercises, videos, I’m going to go into all subjects, all grades, all and all this practice software to allow kids to learn at their own time and pace. I had already started building some teacher tools so the teachers could keep track of it. I had a notebook of testimonials from around the planet.” I said, “Eventually I want to localize this into languages of the world.” And Ann said, “Well, that’s ambitious, but how are you supporting yourself?” And I told her in as proud of a way as possible, I said, “I’m not.” And she kind of processes that. She pays the bill, and then we part ways. And about 10 minutes later, I’m driving into my driveway in Mountain View and I get a text message from Ann and it says, “You really need to be supporting yourself. I’ve just wired you a hundred thousand dollars.” So that was a good day a little bit. So that was great…
I’ll list one more ’cause the story only gets crazier. About two months after that, this is now about May of 2010, I want to say, or maybe June. I’m running a summer camp ’cause I never thought online is a replacement for physical. I always thought online could be a liberation of the physical, so you could do more games and simulations and things like that. So I was literally running a trading floor of seventh graders, and I start getting text messages from Ann, which you can imagine I now take very seriously. And there were five or six text messages that were coming. It was a little bit disjointed, but Ann was essentially writing, this is Ann writing, “I’m at the Aspen Ideas Festival. I’m in the main pavilion. Bill Gates on stage last five minutes talking about Khan Academy.” So I didn’t know what to make of this. And y’all can look, if you do a web search for like Bill Gates, Aspen Institute, Khan Academy 2010, you should find this video. But I found this video when I boot, I booted a seventh grader computer, and I found a delayed broadcast or a recording of it where Walter Isaacson asked Bill Gates, “What are you excited about?” Open question. And Bill Gates just, “There’s this guy Sal Khan. He’s got this website, Khan Academy. I use it with my kids. I use it myself.” And I was like, “Is this really happening?” And I remember that night I showed my wife that. I’m like, “What do I do now? Do I call him up? Like, what’s the protocol here?” And they left me in that state for about two weeks. Two weeks, I’m in actually this walk-in closet where I still am about to, worldwide headquarters of Khan Academy, about to record a video. And my cell phone rings, it’s a Seattle number, and I answer it. “Hello?” “Hi, this is Larry Cohen. I’m Bill Gates’ chief of staff. You might’ve heard that Bill’s a fan.” “Yeah, I heard that.” “And if you’re free in the next few weeks, we’d love to fly you up to Seattle and figure out ways that we might be able to collaborate.” And I was looking at my calendar for the month, completely blank. So I said, “Yeah, sure. I got to cut my nails next Wednesday, but I’m happy to meet with Bill.” But anyway, we had that meeting, and it was very similar to the meeting with Ann. And so by fall of 2010, and actually I was having similar meetings at around the exact same time with executives from Google, and I can talk more about that, who were trying to fund a, they were going to fund five projects that could change the world was their charter. And then fall of 2010, it all converged. So that was a major break. The Gates Foundation and Google, on top of the money the donors gave, allowed us to become a real organization.
– Wow, thank you for sharing all of those big breaks. And I hope that a lot of us in this room are hoping for our next one. Just to go back a little bit further down memory lane, who was a favorite teacher in your past and how would their role be transformed by Khan Academy and generative AI and ChatGPT that’s coming into the world now?
– Oh, good question. I’ve answered the favorite teacher, but not the combo one. I’ll give credit to, I talked about that first day in second grade, Ms. Kraus and Mr. Cell in that enrichment class. I think what would’ve happened if Khan Academy existed then, some of that “Where in the World is Carmen Sandiego” play would’ve been Khan Academy play. And you would’ve had some kids in Metairie, Louisiana, accelerating dramatically in math and science and other subjects.
Ms. Ellis was my fifth-grade social studies teacher. And I remember her because in hindsight I realized how brilliant she was. She ran her fifth-grade class like a humanity, a college graduate seminar. I remember she used to just peel an orange at the front of the classroom and just ask us questions all day about like, “So why do you think they did this? And why do you think this happens? And what would’ve had been different if they didn’t decide to declare war and et cetera, et cetera.” And I always found it really interesting, and later, I realized how special it is to have a teacher like that. If Khan Academy, Khanmigo and all of this existed, yeah, I think it would’ve provided even more space for that type of Socratic dialogue because students could get a lot of the core skills at their own time and pace. And if you have the AI to also be able to supplement and you can ask some questions that you might otherwise be embarrassed to ask in class, all the better.
Let’s see, I remember Ms. North was a really cool teacher in middle school who once again liked a very Socratic, very philosophical type of question. She was our English teacher.
And then in high school, Mr. Hernandez, who was a math teacher, but he was also the advisor. It might not surprise you, I was the president of the math club, and he was our advisor, and he kind of treated me as a colleague, which was really great. Like, we would plan things together. And I really appreciated that.
And then Ms. Kennedy, who ran journalism. She also ran it, like, we would just run, she ran it like a company, like, we ran the newspaper and she was like the chairman and she would advise us and help us.
And I should also, when I was in high school, I did dual enrollment at the University of New Orleans, and there was a professor there, Dr. Santania, who when he realized that I was taking some very advanced math courses with him. And when he realized that I didn’t have a computer at home, nor could I afford one, he got me a job at the university doing research with him. And that’s essentially how I first got access to a computer and I started learning how to program, et cetera. So yeah, I don’t think—I might not be here if any of those folks maybe weren’t there.
– Wow, thank you for sharing those inspirational stories and how those people have impacted your life. I know that you’ve told us that learning the ability to learn is of utmost importance, but what’s a skill that took a lot of time or attempts to master for you?
– Oh, that’s a good question. I mean, if I go refer back to what I’ve already told you, learning to speak properly. I still have to pause before I say hospital ’cause until I was about 8 years old, I would say hostable. And I still sometimes will if I’m not really, really careful. So I always say hospital, I pause. So I’ll say that.
Other things, well, I’ll say at the other end of my life, and this is one that unfortunately a lot of folks I think haven’t invested enough in is, in 2015, so eight years ago, like, weird things were happening. Like, I started getting claustrophobic on planes. Like, I would get on a plane, and I’d, like, want to get out as soon as possible. Like, it was not a pleasant experience. And I’m, like, something’s off, and I wasn’t sleeping well. And in hindsight it was really; I was probably too stressed, and t’s like a frog in boiling water. It just creeps up slowly, and it manifests itself in these really weird ways. But I started taking meditation very seriously at that point, really just so that, ’cause I had to travel, and I, just to see if it would make the planes more pleasant. But that practice of meditation I now take very, very, very seriously. And a lot of times, especially people who are ambitious, who want to do big things in the world, they try to kind of account for their own time as much as possible. And they try to be as on and as productive. And I still do that. I try to be as productive as I can when I’m on, but the off time is just as important, arguably more important. And I think it’s easy to overlook. So it’s taken me—it took me, when was that? I was 39 years old when that happened. It took me 39 years to realize that. And I think every year, I realized that more and more and every year, I take more permission with myself to, “I’m going to go to the sauna today,” or “I’m going to take this meeting walking around the park,” or “you know what, I’m just going to cancel these three meetings ’cause I don’t think I really need to be in that. And instead I’m just going to go meditate outside.” And I actually think that makes me more productive and it makes me a better entrepreneur, leader, whatever you want to call it.
– Wow, I think especially MBAs needed to hear that. We definitely need some off time and definitely some meditation to help us feel grounded. I did that just again before getting on stage. And maybe you answered this, but I’m going to ask this again. What is a topic or a skill that you think everyone should learn that is traditionally left out of K-12 education?
– Oh, so much. I mean, look, I’m talking to a bunch of MBAs. A lot of what I learned in the first year of business school, like accounting and the core of finance and how do you price something and present value? The mathematics of it are, like, middle school mathematics. It’s not super difficult. I don’t see any reason why that shouldn’t be taught in middle school or worst case, in high school. Anyone who just needs to think about one day starting a business or trying to decide whether they’re going to rent versus buy a house, they should learn that.
So I would say those types of skills and at minimum, personal finance skills, and this is something that schools are increasingly, at least states are passing legislation for schools to teach it. I think the schools without help are going to have difficulty ’cause like who’s going to teach it? Well, we’ve been working on this for many years, but we’ve recently launched a full financial literacy course. It’s targeted at the high school level, but honestly it’s useful for anybody where it’s like, people, well, I remember, I had an MBA and the first time I went to buy a house, I was just like, “Why do I need title insurance? What is title insurance?” You’re like, “What is this thing? And exactly how does escrow work, right?” So I think that type of thing is super, super useful.
There I had a brief period in undergrad where I wanted to be a, well, I’ll say there’s a class I took actually in, I was going to talk about, I briefly thought I wanted to be a lawyer, but then there was a class I took in business school called business law. But it was essentially, it was a crash course in law. And I remember thinking, “This could have been taught to me when I was in seventh or eighth grade.” Like, it should have been taught to me in seventh. Most people get through the whole education, including a college degree or including graduate school, and they don’t understand the legal system, which seems pretty darn important. So yeah, I would throw in financial literacy, legal system, basic finance, and accounting. Those are the big ones.
– Thank you. So our next question is: You’ve started Khan Academy and registered it as a nonprofit organization, and you’ve continued to run it as a nonprofit. How did you decide on this business model, and why not a private company?
– Hmm, yes. And there’s actually I think now three business school cases on this decision. How it’s evolved, it’s interesting. My day job at the time, I was a hedge fund analyst, which was arguably the most for-profit activity. It was only for-profit. Like, there’s no other benefit, or well, liquid capital markets and pricing efficiency, every hedge fund on the margin is helpful. But anyway, it’s primarily for-profit. But while in that job, I was able to be a really, it was interesting to be an observer of how organizations’ behavior was driven by capital structure. That honestly, you had annoying people like me, hedge fund analysts calling you and trying to figure out whether you’re going to meet or miss your earnings this coming quarter. And usually, the way that executives are compensated at these firms are based on stock performance and things like that. And I also saw when the debt holders are senior versus to when you have a strong founder versus when you’re in the second generation and the founder’s left, very few of these organizations are able to stay. There are for-profit organizations that start with a strong, mission-driven founder and are able to stay true to that until something happens to the founder or until the founder doesn’t have control of the organization anymore, right? Private equity just bought it or well, now the founder owns 20%, but 80%, it’s a public company now, or the founder dies, goes away, gets fired, and now someone else is taking over. There’s very few examples you can give.
And there is an irony that the only class in business school—I don’t know about Haas—but where I went to business school, they didn’t fail anybody, but they would give you these like internal grades and the only grade that I would’ve been, like, the equivalent of a fail if they actually failed anyone, was a class called social entrepreneurship. Because I was so skeptical of some of these nonprofits, I was just like, “Yeah, they tell a good story, they pull on your heartstrings, but is that really going to cure cancer? Is that really going to blah, blah, blah, blah.” And that’s what I wrote in the essay until I realized that that was the not-for-profit that the professor started, but not a good idea. But it didn’t fail anyone, so not a big deal either. It allowed me to speak my truth.
So I was cynical, but being in the hedge fund world, I said, “I want this Khan Academy project to just always be there.” I read a lot of science fiction. I was like, “what if this could be like a new type of institution, like the next Oxford or Smithsonian, but it could be on a completely different scale. It could be for billions one day and maybe it could last for hundreds of years.” And if you look at that, the only organizations that can be true to that type of a mission are not-for-profit organizations. And so, it’s sometimes delusional to say that you want to be the next Google or the next Facebook or the next Amazon. That’s hard enough. It’s arguably even a little bit more delusional to say that you want to be the next Oxford or you want to be the next thousand-year institution, but you live once, why not try?
– And what are some of the biggest challenges you faced while running and scaling a nonprofit organization? I know it’s very unique and different from running a for-profit company. A lot of our classmates who are in this room today are either building their own organizations like this or want to join one and lead one of these organizations.
– Yeah, if you asked me in 2000, if you asked me in late 2009, I would’ve said the biggest challenge is raising money and convincing people that you’re legitimate. I, at the time, took the most naive strategies and look, the advantage that Khan Academy had or that I had was that I was making something that was discoverable by people. I think it’s a very different thing if you’re starting a not-for-profit where you’re taking donations from one group and then you’re going someplace in the world and then you’re putting that money, I’m probably not the best person to ask on how do you start a not-for-profit like that. But here, not only was I making something that people could find and discover, but it was happening, it was growing, in those early days it was growing 15, 20% per month. So it was on this kind of exponential growth curve that people would associate with a high-growth tech company. And look, VCs were coming up to me and they were saying, “Hey, I’ll write a hundred thousand dollars check right now. You can quit your day job.” And it was tempting and I would take the second meeting with them, but the second meeting, they started talking about premium offerings and putting this behind a paywall and that and this is how you’re going to, and I’m like, no, and I was getting letters from kids that I knew and families that I knew would not be able to use the tool if this VC had their way. And so that’s why I kept turning that down, and luckily, my naive strategy kind of worked with people like Ann discovering it. And eventually, I mean, you can’t plan for Bill Gates just showing. I have a theory that I sometimes play with that benevolent aliens are using Khan Academy to prepare humanity for first contact. And some of what I told you of like, these things that seem a little bit more than just chance. Those are my data points. I have more. I actually think the benevolent aliens help set me up with my wife as well, so that I can prepare humanity for first contact through Khan Academy. But I have some good stories there.
But the real difficulties, I think, beyond that first validation, which is a hard one, and it was really hard. I mean, I kind of laugh about that nine months where I really didn’t know what I felt like; I kind of ruined our financial future. But once we started to have resources, I think it is, and we were a little bit, we are still unusual. We’ve been able to attract very good talent. Like, the main arguable advantages that a for-profit has over a nonprofit is access to capital, access to talent and potentially some notion of the incentives make you act in a different way, which could be good or bad. Access to capital, at least in those, in hindsight, I’m happy that the access to capital we’ve had, our budget is a real budget where it’s like 60, $70 million a year now. And it’s primarily philanthropic. So we have been able to do that.
Now, there have been other for-profit organizations, including in EdTech, that started a year or two after Khan Academy; and they get higher and higher valuations, and they’re able to do these really huge rounds and raise a hundred million, raise 500 million, raise a billion dollars. And as a 30-year-old, I would’ve been somewhat envious of that. Now, I don’t envy that at all because that creates all sorts of distortions and pressures. And the hardest thing is, as you grow, to maintain alignment. If you can grow and maintain your talent and your alignment, a 300-person team can outperform a 30,000-person team. I’m not saying this even as an exaggeration. I 100% believe this. Like, you see this playing out in the real world right now. I mean, OpenAI is a 300-person team. I think they’re outperforming many 80-, 90,000-person teams on what they’re doing ’cause they are aligned. And to some degree, probably the scarcity of capital in their early days helped drive that. And their group that I know very intimately now, and I know some of the 90,000-person organization, some of them were our original funders very well as well. But yeah, I think that alignment, when you get to like 20 people, you start realizing that people just don’t always assume the obvious, what you think is the obvious thing ’cause you’re not just having conversations on the way to the snack counter or whatever. And so you have these growing pains. There are like 20 people, 50 people, 100 people, 200 people. And so now I’m a big believer in like, you can’t overinvest in alignment and alignment is not just communication. And this is a muscle that I don’t think people in Silicon Valley exercise enough, is being very clear with team members of like: “Look, this is what we’re doing. If you’re excited about that, awesome. If you’re not, this might not be the place for you.” That is kind of strong language, but I have found it to be very powerful language because otherwise, people are just trying to pull it in all sorts of different directions.
– Thank you. And just to, like, double down on that last few statements: As your business grew, what were your main culture objectives, and how did you maintain or create that company culture? Did you have any hiring strategies or professional development plans that you leaned into? Tell us more about that.
– Yeah, no, it’s a good, I segued well into your next question. My early days, like, my first job out of college was at Oracle. I was a product manager there and I got very cynical, very fast. I was like this big company, I’m just sitting in meetings all day, well, anyway, I don’t want to name, well, there’s some fun stories there, but my manager at the time had some views about the efficiency and how it could be improved at the organization.
But that made me, like, there was a time where I said, “Khan Academy is always going to have, is always going to be a startup. It’s going to be a perpetual startup. I don’t want it to ever be a large company because large companies become bureaucratic and people just sit in meetings all day.” And so I had a view of “Let me just hire the smartest people I can and get out of their way and have as few meetings as possible.” That can sometimes be good, but it leads to issues at some point. Smart people, if they’re not aligned, can easily try to pull in different directions. And if you’re not communicating enough, you’re not going to get enough alignment. So over time, I’ve started to realize that it’s very important to, that we are a place where people can bring their full selves to. And because look, you’re spending a large chunk of your life doing work. And so, I try to—Khan Academy is a place where you shouldn’t be embarrassed to say that you’re going to take an hour to go meditate. You should be proud of that. You should even say that as an example. And look, a lot of that comes from me setting an example. It’s one thing to say it. It’s another thing to say, “Hey, I can’t attend that meeting because I’m going to be meditating,” and that sends a signal to other people. “Or I can’t do that evening meeting because I said that I’m going to do this event with my family that night.” So I think it’s very important to put those guardrails and all of that.
But I think culturally what I take a lot of, what I try to put a lot of energy into is making sure people are aligned. Like, disagreeing is great. Like, let’s disagree while we’re talking about the tactics of something, but we’ve got to make a decision and then we’ve got to commit to that decision, otherwise we’re just never going to go anywhere. And if you can’t truly commit to the decision and then that goes into the, like, this might not be the right place, right? Like, otherwise, you’re just going to be constantly trying to undermine it in passive-aggressive ways. So I think that’s another cultural thing.
I’ll say another cultural thing that I’ll say, which is, I think, obviously words like DEIB or letters like DEIB are, like, a big deal these days and they always have, and I’ve always been a very big believer in that. But I do believe that sometimes it is viewed as a signal for you are a left-leaning organization, or you say DEIB, but you really mean that you subscribe to these ideas that other people might not subscribe to, which is almost the opposite of DEIB, right? The whole argument behind diversity is that when you have different viewpoints, people who are different than you, now reasonable, and people who aren’t trying to shut each other down, et cetera. But if you have different reasonable viewpoints, that’s going to get your eventual decision to be a better place and people with very different experiences. And I remember a wake-up call for me was we had a senior executive at Khan Academy, this was about five or six years ago, and it was at an offsite, and all of a sudden, and he was an older gentleman and he said, “I was afraid to say this, but now I feel comfortable with y’all enough to say this. I’m an evangelical Christian and I’m a Republican.” And I remember when he said that, like half the team was like, “Wow, you’re so brave for saying that.” And then the other half of the team was saying things like, “Who did this guy vote for? And what’s his view on this and this?” And that was an eye-opener for me. It’s like, we can say DEIB, but we aren’t DEIB if that guy, who was really good at his job, who everyone really liked to work with, was afraid to even say what religion he is. And so what I try to do now is push everyone on DEIB where I say, “That’s not to hire more people that agree with you. That is to hire people who are truly from diverse perspectives that can push us to get to a better place.” So I think I take those words more seriously than a lot of other corporations that do it in a very performative way. I think there’s, most executives, if I’m honest, say I’m not going to get involved. I’m going to hire someone else to do that. And I’m just going to keep nodding and hope for the best versus actually trying to get true DEIB.
– Thank you for that. After hearing the inspiring story of the origin of Khan Academy, our ambitious and idealistic students here have questions for you. If you’re interested in asking a question, please go ahead and step to the mic right there in the aisle and tell us who you are and then go ahead with your question.
– Sal, thank you. Is this on? Probably not. OK, it is. Sal, thank you. My name is Doris. I’m a second year, and I lead our Haas Education Club here, which everyone should join. Aside from that pitch, oh, first off, I grew up hearing your videos, watching your videos, so I’m very familiar with that voice and today, I’m able to put that voice to the face and it’s great. My question for you is what is the largest gap that you see in American education today?
– Oh yeah, it’s a big one. I’ll say at least two. I’ll say the big one is, we fundamentally have a seat time based system, especially in high school, but even in college as well, right? In high school, we know the Carnegie units, which is like, you get a Carnegie unit when you sit in a year in math, a Carnegie unit, a year in English, et cetera. And even college, high school graduation requirements and college entrance requirements have stuff like you need to take three years of math. You need to take two years of foreign language. That, in no way, talks about whether you learn the math or learn the foreign language. So I would love to go more to an outcome-based or competency-based system that’s like, you should get to this level of math before graduating from high school. You should get to this level of reading, this level of writing. And if you don’t, you have the opportunity incentive to keep working on it. It’s not like it’s an all-or-nothing type of thing, but if you go there fast, great, you can get there fast and you can work on other things.
I think college should be the same, the fact that it’s all credit-hour based and that magically, regardless of what you major in, it’s all kind of like a four-year program. Yeah, there’s some arguments behind it from a stage of life point of view. So I would go to a competency-based, a corollary to that is being mastery-based, which is this idea that if students have a gap, a conceptual gap, that they have the opportunity, the incentive to continue to work on it. So if you’ve got a B in computer science, and two years later, you’ve mastered that material, you should be able to do something that allows you to show that you’re now at an A level, your transcript should be able to be upgraded, so to speak. And a correlator to that is, allow students to learn at their own time and pace.
– [Doris] Thank you.
– Hi, my name’s Patty Debenham. I run a center here for Haas students all about social impact. And I’ve been a big fan for a long time. I loved your book and at the end, you painted a terrific vision of a new kind of higher education that doesn’t cost $70,000 a year. I mean, even Berkeley, a public institution, is crazy expensive. I’d love to hear how you’re thinking about that now and how it can be possible that my 10-year-old doesn’t have to pay 70, well, I don’t have to pay $70,000?
– Yes. No, this is something I’ve been thinking a lot about and many of y’all know, I mean, I wrote that book, and then the year after I’m like, it’s one thing to write about a book. It’s a whole other thing to try to implement it. So I did start a school and now schools based on the principles in that book. We have Khan Lab school out down here in Mountain View and Palo Alto. And now there’s this Khan World school that we started with Arizona State University, which is a virtual high school. So the virtual middle and high school, and then Khan Lab school down here is K-12.
But the big question is: What about higher ed? And there’s a couple of tactics that we’re taking. One is we are actively exploring ways that Khan Academy, mastery on Khan Academy can result in college credit. So one method is if you can make essentially some version of dual enrollment via Khan Academy available to anyone on the planet, and it’s essentially it’s free or near free, and they can get a lot of these first- or second-year classes out of the way; that automatically lowers the cost of college education and actually increases the capacity for more people to get through the physical doors, so to speak. But the really cool thing would be if—and I don’t care if I’m involved—but someone, and there’s some universities, I think, that are already touching on that. There’s, like, Waterloo in Canada, which many of y’all know is an excellent, one of the world’s best engineering schools. Kids graduate from there, not with debt, but with savings because they use the co-op program so effectively, internships. And there’s another advantage where, while the Stanford and the Berkeley and the MIT kids are in classes in the fall and the spring, the Waterloo kids are getting, they get a monopoly in all the internships at all of the places, and they’re getting paid. And so that’s why not only are they getting better experience, they’re more employable. So I think a model like that would be really interesting. I would love to see it in the U.S. I know there’s a lot of talk these days about downtown San Francisco having emptied out. I’ve whispered to a few folks, “Hey, someone should start a university there, but make it so it’s free and not free through philanthropy.” Maybe you need some philanthropy to get it going, but in the long run, it’s free because the students are able to do a lot of real-world work that can be used to both pay the university and pay the students. So yeah, we’ll see. We’ll see. Yeah, your 10-year-old, I have about eight years to work on that. We’ll see what happens.
– Thanks for taking the time, Sal. My name is Asif. And a subject that talks about that learning to learn and critical thinking that you mention a lot is philosophy and looking at some of your books, maybe you’re a fan as well, but academic philosophy is famously dull, whereas the practical impacts of a philosophy educated population society is invaluable. So I’d love to get your thoughts on philosophy in the education system and maybe if you’re incorporating it into Khan World School or Labs.
– Yeah, so we definitely incorporate, like, when I think about what a great education looks like, I think you definitely should have Socratic dialogue about things like philosophy. I mean, there’s other types of things that you can get into really good, thoughtful debates and discussions about, but philosophy’s definitely like half of all the discussions that you could easily have. And at Khan World School, Socratic seminar is actually the anchor point for the whole school. And we’ve partnered with Steve Levitt’s group at the University of Chicago to create a whole list of topics. And half of them are literally philosophical, but they’re real-world philosophical to make it not just abstract. So there’s, like, literally a debate for middle school and high school students on things like should we leverage CRISPR to modify the human genome? Go. And all sorts of philosophical debates start to emerge from that. Will AI be a net positive or negative for humanity? Is GDP the best way to measure progress? You have to learn economics and history and math, but there’s a lot of philosophy in that. So yes, I’m a big fan of it. We’re definitely trying to, implementing the schools we’re doing.
The AI that we just, Khanmigo, which we built, we partnered with OpenAI over the last year. We launched with GPT-4 several months ago. One of the things that you can get into debates with the AI about frankly all of these topics and more, so that’s one way to hopefully scale it out a little bit more. And then we do have some content already on, and this is more academic philosophy. We’ve partnered with an organization, so that content is on Khan Academy. But yeah, I definitely think it is important. It is one of those things that I think you could have it in almost any subject. And I think that’s actually probably the way to make it, you tie it to real-world issues that is happening every day on the news. Like, reasonable people aren’t getting into debates about these things. Those are great fodder for philosophy.
– [Asif] Thank you.
– Hi, first off, a huge thank you. I was a public school teacher in New York City and Khan Academy liberated my classroom to be a standard space, like collaborative problem-solving space.
– [Sal] Oh, that’s good to hear. That’s the dream.
– My students in Brownsboro were, like, outperforming Westchester white, like, rich students because of work like yours.
– [Sal] If you have any information, I’m always looking of, anyway, but yeah, go.
– I’m looking for an internship this summer. But people looked at, and what made Khan different from me is that you trusted me as an educator to understand data and you gave it to me so I could use it. And you trusted my students to have the data too. That was better than ST, better than I learned, better than anything else. But people looked at me as an educator as unique because I knew how to use the data. My big fear going into AI and ChatGPT is how do we support and build a teacher force that is able to use that technology to liberate their classrooms?
– Yeah, that’s a great question. And what I’m hopeful about, obviously, it’s still very early days, we’re building this right now with all of our Khanmigo stuff. We realize it’s great; Khanmigo can act as a tutor, but it’s even more powerful that it can act as a teaching assistant. And some of that is to help with things like lesson plans and rubrics and give a first-pass grade and writing progress reports, which as you know, as a former teacher, take up a lot of your time.
– [Audience Member] A lot of time.
– So that’s good. But I also think that generative AI means the end of these, like, kind of spreadsheet looking dashboards that you’re familiar with because now it’s like you’re having an analyst. And we already have implemented some aspects of this where a teacher, instead of seeing the dashboard, right, and then having not every teacher can do what you did. And like looking at the dashboard and figuring out which kids could use what, they can just talk to the AI and say, “Hey, so what’s going on?” And the AI can say, “Well, for the most part, the students are doing all right. Here’s five kids who are, for the most part, struggling with these three concepts. Here’s an idea for a mini lesson that you could do with these three kids while the other, or these five kids while the other 20 or 25 students work at their own. Or if I were to group, or here’s an idea for a lesson, and by the way, I would group the kids this way for this reason. This third of the class is really struggling, this third is OK, this third is ready for some really enrichment. They’re getting bored. Let’s give them something else.” This isn’t science fiction anymore. And then if you want the data and the AI will say, “Oh, and by the way, if you don’t believe me, click here for the spreadsheet.” But I think that’s going to make it much more accessible to all teachers. You’re going to be able to text with the dashboard, so to speak, and say, “What should I do? What insights are here? What do you recommend I do next with the students?”
– [Audience Member] Thank you.
– Hey, Sal, my name’s Alex. Following up on that a little bit on the Khan World Academy, I was listening to the podcast with Steve Levitt, and initially I was skeptical, but then I heard the student talking about her experience and it started making a lot more sense. And so I was more curious if you can go in on, like, how you continue to design that curriculum and also what your vision is to scale that up so that more students can have an experience like that?
– Yeah, I mean the beauty of Khan World School and ASU, we did it ’cause ASU already runs ASU prep, which already has 40,000 students, so they know how to scale. And they already have a charter in the state of Arizona, which means ASU prep. And now Khan World School is free to any student in the state of Arizona. And hopefully, we can get charters in other states as well. So that free, world-class education for anyone, anywhere can be true beyond Arizona ’cause this is a full school.
A lot of the principles are what we’ve learned over 10 years running the physical Khan Lab School. And also what I wrote 10, 11 years ago in The One World Schoolhouse, students learning at their own pace, having more agency autonomy, the adults, the teachers, the guides, whatever you want to call them, their time focused more on unblocking students, motivating students, advising students, driving Socratic seminars versus lecture, et cetera, et cetera. And with a virtual environment, you’re not bound by time or space in the same way. And we also wanted to show that you could do virtual much better than people did it in the pandemic where it was soul-crushing. And so, we anchor it with a Socratic dialogue. Kids get an hour or two of synchronous community-building discussion advisory every day, but then they have a lot of autonomy to do other things.
And to your point, Steve Levitt, who everyone knows here from Freakonomics, famous economist, he’s not a sucker for bad data. But as far as we can tell, these kids are not learning 20, 30% faster. They’re learning two x, three x faster, and they’re enjoying it. And there’s stuff we’re not even measuring like the seminar ’cause there aren’t any standardized tests for how good someone is at seminar. But we feel confident that these are well-rounded students that are going to really thrive later on.
– [Alex] Thank you.
– Thanks, Sal. My name is Vinit, and I’m in the MBA program. So recently, I started getting excited and passionate about using technology and AI to perhaps achieve, like, educational parity. And so I was at this San Francisco tech meetup, and I started going around and just started pitching this idea to some folks. Look, we don’t need to teach children or kids how to use computers or even learn how to code anymore. Like, AI can generate code. We need to just provide them more creative ways to play with code or sort of just give them, like, the fundamental building blocks like Legos using AI and they can just sort of create apps on their own and feel empowered about themselves. And what ended up happening throughout the course of the evening was when I talked to the investor community, like the VCs and folks, they would just look at me like, no, no, like EdTech’s a very, very difficult business model. You’re either going to have to sell to the parents or schools, which is no joke. And then there happened to be folks there that were parents and many of them were technologists famously creating apps for adults. But then when it comes to their own kids, they would say like, “No, I just want my kids to play outside, be in the nature. Like, I personally believe that technology hasn’t really given us anything meaningful in form of, yeah, education other than Khan Academy, of course.” But I mean, it is valid concern giving tha the rates of stress, anxiety and focus attention for teenagers or preteens especially has gone up quite a lot recently. So, yeah. I don’t really have a question. I guess I would wonder how I really want to remain optimistic, and you’re one of the few people that I see that are optimistic still in this space. So when you’re not speaking to a room of full of people that have already bought into this idea, I guess, how do you convince those folks, whether it’s parents or investors or whoever it is?
– Yeah, well, I think a bunch of stuff you said resonates with, I think the VCs you’re talking to are right. EDTech is a hard thing especially if you’re trying to get VC-like returns. It’s a very hard thing. And if you want to get VC-like returns, you probably will have to sacrifice some of, maybe some of your principles around like, OK, let’s make this a thing for rich kids, and maybe it helps them cheat a little bit or something like that. Honestly, there are multi-billion dollar publicly traded EDTech companies, that is their business model. I won’t name names. So it is hard ’cause it’s, and it’s even harder if you’re trying to introduce a new paradigm around, in your case, engineering/coding.
My answer to the screen time is, it’s not about screen time is neutral. There’s some very good uses of screen time. Obviously, if a student is writing a paper, coding, doing some graphical art, I argue Khan Academy. But even that needs to be within reason. For my own children, I still want them to go outside, play with their friends. If someone visits Khan World School or especially Khan Lab School, if they visit, you’re going to see kids not on a computer all day. They do probably use a computer more than your average school, but they’re also getting more social interaction than your average school because they’re not sitting in lectures all day, either. So Khan Lab school kids are constantly working on projects and collaborating and walking around the space. They don’t have to sit in one chair all day. So I’m a big believer in that. So my advice would be, you might be trying to tackle too many degrees away from where people’s comfort zone is right now, but maybe there’s a way to take that same tool and show that you could use it to create, like, enterprise apps or things like that, and college students could use it to start companies. That could probably get, I could imagine, VCs wanting that even to start their own incubator. They create an app like that, and people could come in and just write apps really, really, really fast and get product market fit tests really, really fast. That could be an interesting model.
But maybe there’s other things if you find the right partner with Lego or something, I don’t know, it’s not easy to pull off. Who knows? I don’t want to discourage you either ’cause what I was doing in 2008 or 2009, people found far, far more ridiculous than what you are doing. So people thought not-for-profit, YouTube videos, personalized learning, all of these stuff, was sounded ridiculous.
– [Vinit] Thank you.
– Well sadly, we are out of time. Thank you for the time that you’ve spent with us today, Sal.
– [Sal] Thanks for having me.
– Thanks to our moderators, Anu and Mrudula. Thanks to all of you for joining us. This has been a wonderful and inspiring hour. I hope you’ll join us again for future Dean Speaker Series. Have a wonderful day.
Professor Emeritus Pablo Spiller (right) accepted the 2023 Elinor Ostrom Lifetime Award in August at the SIOE’s annual conference in Frankfurt, Germany.
The award is given biennially to a researcher who has made “sustained significant academic contributions to institutional and organizational economics.” It is named for Indiana University political scientist Elinor Ostrom, who shared the 2009 Nobel Prize in Economic Sciences with Berkeley Haas Professor Oliver Williamson.
Professor Emeritus Pablo Spiller
Spiller is the Jeffrey A. Jacobs Distinguished Professor Emeritus of Business & Technology at the Haas School, as well as a UC Berkeley professor of graduate studies.
Spiller’s research lies at the intersection of economics, politics, and the law, and spans political economy, industrial organization, the economics of regulation and antitrust, and regulatory issues in developing countries. One research stream analyzed the hazards inherent to public contracting, and how it differs from private contracting. Spiller applied the approach to such areas as utility regulation, the organization of bureaucracies, and the inner workings of public companies.
In addition to his academic work, Spiller has consulted for the World Bank, the InterAmerican Bank, the UNDP and multiple governments and private companies throughout the world on the design and implementation of appropriate regulatory policies, contract design and implementation. He has also testified in numerous international arbitrations involving contract, regulatory and investment disputes. He is the former President of the International Society for New Institutional Economics (now SIOE).
Spiller served as the Editor-in-Chief and Associate Editor of the Journal of Law, Economics, and Organization for 19 years, and held multiple Editorial appointments at a variety of academic journals. On special leave from Berkeley, he served as a Special Advisor to the Bureau of Economics of the US Federal Trade Commission.
It’s the second year in a row that Haas has tied with MIT for the #2 spot. Both schools had peer assessment scores of 4.6 on a scale of 1 to 5. Wharton held the #1 spot with a score of 4.7.
The business school ranking is based entirely on a peer assessment survey of deans and senior faculty members at peer institutions. They are invited to rate peer programs with which they are familiar, based on each program’s scholarship record, curriculum, and quality of faculty and graduates. Read more about the ranking methodology.
In U.S. News’ 2024 Best Colleges, UC Berkeley again ranked as the #1 public university in the country, tied with UCLA.
Berkeley Haas rose to #10, tied with MIT Sloan, in the 2023 Bloomberg Businessweek ranking of full-time MBA programs. Haas rose four spots this year, up from #14 last year.
Based on input from recent full-time MBA graduates, alumni, and employers, as well as school-provided data, Bloomberg rates MBA programs on what Businessweek considers to be the five fundamental elements of business school education: graduate compensation, learning, entrepreneurship, network, and diversity. Each area is weighted according to its importance among the alumni and employers who are surveyed for the ranking.
In these five areas, Haas ranked:
#6 in entrepreneurship
#6 in learning
#15 in compensation
#15 in networking
#16 in diversity
Among the top 25 schools in this ranking, only one school, Georgia Tech, beat Haas in the learning category. In diversity, Haas was rated third among the top 10 and fifth among the top 20 schools.
Bloomberg Businessweek is known as a volatile ranking — and this year was no different. Harvard Business School, Northwestern Kellogg, MIT Sloan, and Yale SOM dropped by multiple points, leaving room for other schools to rise into the top 10, including Virginia Darden and Michigan Ross, which moved up six spots. Stanford Graduate School of Business remained at #1.
Ruth Porat, president, CFO, and chief investment officer at Alphabet and Google, ranks among the most influential women in finance, having earned spots on both Forbes’ and Fortune’s 
In a new book, 
