Parallels in evidence crises: What can the field of impact evaluation learn from the field of social psychology?
Think back 15 years and you might remember the rise of TedTalks and a renewed cultural interest in positive self-affirmations, and maybe one type of affirmation in particular: the “power pose”. In 2010, Harvard and Columbia researchers studied how our posture could affect how you feel about yourself. In 2012, Amy Cuddy, one of those researchers, gave what would become an extremely popular TedTalk on the Power Pose – hands on hips, shoulders back, or other “confident” postures – and how it can make a person feel more powerful. From there, the simple message of the power pose burrowed into our collective consciousness, with politicians adopting it, people practicing their own power poses before interviews and speeches, and many a corporate Lunch and Learn espousing the benefits of a fake-it-til-you-make-it posture.
Jump to 2015, and the Power Pose came under close scrutiny and critique. The power pose research made three claims: that these postures increase psychological feelings of greater power, and control; that they create changes in dominance and stress hormones; and that these postures create behavioural changes with increased risk tolerance. In 2015, the latter two were now in question because other researchers couldn’t replicate the study’s results.
In recent years, impact evaluation in the nonprofit sector has come under increasing scrutiny for its failure to produce credible, actionable insights. Evaluations are often boiled down to a simple, positive, message, used more for fundraising (sales) than for genuinely learning about the effectiveness of programs. This "sales over science" tension can lead to nonprofits focusing on vanity metrics rather than rigorous evaluations that truly assess a program's impact.
The current state of impact evaluation closely mirrors the environment within social psychology that led to what is known as “the replication crisis”. Questionable research practices, driven by incentives for publication and media attention, were deteriorating social psychology’s credibility. It wasn't until researchers within the field acknowledged the crisis and sought reforms—such as pre-registration, transparent data sharing, and open peer review—that the discipline began to recover. The parallels between social psychology’s crisis and the challenges facing impact evaluation suggest that the nonprofit sector can learn from social psychology’s response. In a new paper by our own Michael Lenczner and Ben McNamee, along with our collaborator Patrick Forscher at Busara, we ask what the field of impact evaluation can learn from the replication crisis to take similar steps toward reform and renewal.
Our paper explores the parallels between social psychology and impact evaluation in-depth, drawing on the lessons from the replication crisis to propose solutions for the nonprofit sector’s current evaluation challenges. We need more appropriate, transparent, and rigorous evaluation practices—ones that are tailored to the specific needs and goals of nonprofit organizations. Solutions such as administrative data labs and other innovative tools could provide the sector with cost-effective ways to conduct higher-quality evaluations, benefiting both funders and nonprofits.
The power pose is just one high-profile example of the evolution of social psychology during the replication crisis. Debates on the merits of power pose science continued past 2015. By 2020, there had been questions about the power pose methodology and p-curving, as well as an important observation by Michael Crede on the lack of a “neutral” pose in the study. This all led to further and deeper research on the power pose and related phenomena, including a meta-analysis of whole body motor displays. Like many other examples from the replication crisis, the reassessment of research practices resulted in findings that can lead to a more rigorously researched and nuanced understanding of posture and behaviour.
While the challenges in creating more appropriate and rigorous impact evaluations are significant, meaningful change is possible. By recognizing the issues at play and taking action to address them—just as social psychology did—impact evaluation in the nonprofit sector can regain its credibility. This is an opportunity to ensure that evaluations do not just tell compelling stories, but provide valuable, accurate insights that drive real improvement in programs and services.