Classified: Measuring the Effectiveness of Nonprofit Programs

By Charlie Cooper

In our ongoing series of “Classified” articles we spotlight some of the powerful lessons being taught in classrooms around Haas.

MBA students enrolled in the Social Impact Metrics course at Haas spent the fall semester wrestling with a challenge that’s puzzled management experts for decades: How do you go about measuring the effectiveness of a nonprofit program?

In an era of algorithms and analytics, where almost anything can be measured, there’s a growing push for metrics that will help donors know whether programs are effective—and whether they’re getting their money’s worth.

To get a better idea of what works and what doesn’t, the Center for Social Sector Leadership at Haas formed a multi-year partnership with Amgen. The partnership centers on an interdisciplinary, graduate-level Social Impact Metrics course on measuring outcomes of cancer patient advocacy education and support programs.

“I want our students to be savvy about the claims that organizations make, and to be able to separate what is really working within the nonprofit sector from activities that may feel good but aren’t having much real impact,” said Lecturer Colin Boyle, (pictured), who teaches the course and also serves as deputy director of UCSF Global Health Sciences.

Amgen awarded $35,000 each to four cancer patient advocacy groups participating in the project—Bladder Cancer Advocacy Network, Cancer Support Community, Chris4Life Colon Cancer Foundation and Critical Mass.

In the class, 28 students broke into four teams that worked with the nonprofits to study the specific metrics challenges they faced. The goal was to come up with a plan of action by the end of the semester, when the class presented recommendations to their clients: four cancer patient advocacy education and support programs.

During the semester, the students evaluated the methodologies that the nonprofits might use to vet a program’s effectiveness. They also worked to identify signs that a program was succeeding or veering off track.

Many of the nonprofits were offering the same services—peer counseling, help lines, and webinars to educate people. But as Boyle noted, they weren’t learning from each other about how to do this more effectively.  The course presented a new opportunity to foster greater sharing of best practices within the nonprofit community, he said.

“If everyone has a different help line, then let’s learn from each other so that we can do it as successfully as we can,” Boyle said.

Brigid Warmerdam, MBA 16, worked with the Cancer Support Community, a non-profit that operates a cancer support helpline focused on improving the livelihood of cancer patients and their families.

Her group was charged with making recommendations on how to best measure the helpline’s impact. At the end of the course, they recommended that the non-profit differentiate callers based on their reason for the call— for information, logistics, or emotional support —and to ask callers if their needs were met. The nonprofit could then use aggregate data to inform their program changes.

Warmerdam, who is a vice president at Wells Fargo, took the class hoping to learn more about measuring shifts in attitudes and behaviors, so she could apply this to both her job and her activist work. (One of her initiatives as a diversity leader was to launch a company-wide outreach program to engage LGBT team members with LGBT “allies” to promote diversity at the company).

“Measuring impact is a big challenge and the skills required are not obvious,” she says. “You need be a good listener, get comfortable with ambiguity, and work as a team. To me, this is the solution to doing the best job for your client.”

Chelsea Samuel, a master’s candidate at the Goldman School of Public Policy, who worked with Warmerdam on the project, said the course allowed them to apply principles from class in a real world setting, “with real constraints on their personnel and resources.”

“It was also fascinating to see how they intended to use the metrics,” she said.

Students who worked with the Bladder Cancer Advocacy Network came up with an entirely new approach for the non-profit, centered on the impact of bladder cancer treatments on sexuality.

The data the students gathered is also valuable for grant-pitching purposes, said Stephanie Chisolm, director of education and research for the Bethesda, Md organization.

“When we go to our partners in the pharmaceutical industry, they want a good estimate of whether what we’re doing will have an impact, and to know that we’re going to be able to report the results,” Chisolm said. “Ratcheting up our metrics is going be a huge benefit.”

Scott Heimlich, vice president of the Amgen Foundation, said many non-profits lack the resources to collect this sort of data.

“Our hope is that this work will go toward helping all patient advocacy organizations measure their effectiveness, and to help the larger nonprofit sector do what it does better,” he said.