Measuring Impact in the Startup Newsroom
This article first appeared in the Winter 2018 issue of Change Agent.
We all invest in storytelling — we invest ourselves, our time, and our money in the production and consumption of stories. It’s a narrative education, a venerated format as old as time. But this may be the first time we’ve expected so much of it — a return on investment, measured as real-world impact.
Storytelling has always come with a basic theory of change: tell a good story and it will move the audience to feel a certain emotion, make certain decisions, broaden or shift their worldview. Th storyteller can set the goal.
For newsrooms, as platforms for fact-based storytelling, the most measurable goals were commercial. Media managers hoped that telling good stories would boost circulation or impressions delivered to an advertiser.
But a higher mission ran alongside the profit motive: a hope that journalistic storytelling would make a noticeable, positive difference in people’s lives. Raising awareness around the challenges of the day would bring a greater urgency and understanding to them. Stories would give readers the means and motive to produce better laws, better leaders and stronger societies.
For most of modern media history, however, the impact goal lacked a method of measurement — something to weigh against the dollarized data, as a sign that the stories met that higher mission. There were occasions when a single story could be directly credited with sparking new legislation or exposing a corrupt official. Usually, however, there was just a hope that good journalism would enable good outcomes.
Into this state of affairs came the philanthropic funding as an emergent resource base for newsrooms. By investing in news, grant-making foundations were betting on the impact of fact-based storytelling. Like any investors, they were keen to see the results. How could we prove that stories had a positive social impact?
Newsrooms working with philanthropic funders felt this pressure to somehow measure impact. But as digital communications surpassed print, many other newsrooms felt similar pressures. While foundations needed to measure impact to justify their investments, for-profit publishers needed new ways to understand the utility of their work.
In 2014 Elizabeth Green and her team at Chalkbeat developed a system called MORI, short for Measures of Our Reporting’s Impact. Her reporters would embed the tool into their daily workflow, recording evidence of a particular story’s impact. Developing MORI (now an open-source plug-in for WordPress) forced Chalkbeat to become clear and concrete about its impact goals.
“Our end goal is [to help] people who are going to make decisions and form debates on this issue,” Green said. “The more access to information they have to do that well, the better.” As an independent newsroom covering only public education, their target audience was educators, policymakers, parents and others who directly touch the public school system. They produce journalism – not direct advocacy for any particular outcome or side of an issue. But by informing the debate, Chalkbeat could help those stakeholders generate better outcomes for their communities.
To see the impact of their work, they looked at signs of uptake and activity.
“A parent group in Memphis called Memphis Lift meets regularly for a ‘book’ club, except instead of books, they read Chalkbeat articles,” Green shared, as one example. “They get together to debate the changes in the school system. Our reporting informs that debate.”
Lindsay Green-Barber helped shape Chalkbeat’s measurement methodology. As an early expert, she had no template for measuring or even defining news articles’ impact. She developed a trendsetting mixed-method approach.
“My mission became how we used quantitative [indicators] with qualitative data and other research methods to show what the real world change of media was or could be,” she said. She incorporated case studies, content analysis, and qualitative evidence of how journalism stirs discussion.
The Walton Family Foundation, which invests in storytelling that supports its programmatic focus areas, worked with Green-Barber to craft its impact measurement approach. It began with a need to “think through a basic definition of terms and a common approach to how we both set goals and measure progress against them,” said program officer Kristin Tracz.
Her team developed a systemic approach, orienting grants around targeted audiences and outcomes. “From there,” Tracz said, “we can have a conversation about the best ways to measure impact.”
As the Walton Family Foundation and Green-Barber were working on their methodology, and Chalkbeat was releasing MORI, a field of startup newsrooms were figuring out their own ways to measure journalistic impact. Many were single-subject publishers, like Chalkbeat, focused on a specific “beat” for a dedicated audience. Their focused approach made them a unique example of impact in storytelling.
We produced a study of the trend for Columbia University’s Tow Center for Digital Journalism.
We not only wrote the report, we were also part of the data set. My team at News Deeply built a network of single-subject news platforms on social, environmental and public health issues – each aimed at a community of key stakeholders and readers with a passionate interest in that issue. Our theory of change: stakeholders trying to solve the world’s biggest challenges need consistent, comprehensive information and better connectivity to each other. The broader public would benefit indirectly and directly as our articles were distributed across the media landscape.
That premise drove our strategies for measuring impact. Our quantitative metrics placed an emphasis on time on site, user return, content contribution and newsletter open rates. We closely analyzed our target readers by segment, which helped us identify user behaviors that signal concrete impact. On Ebola Deeply, a pop-up platform that covered the 2014-2015 outbreak, our story on pregnant girls who weren’t allowed back to school sparked an Amnesty International campaign to allow them back in. Our Refugees Deeply investigation of child prostitution among destitute refugees was cited in a US resettlement agency’s amicus brief to the Supreme Court. Our Women & Girls channel’s expert roundup on family planning sparked a new collaboration between two of the academics we covered. Oceans Deeply’s reports on seabed mining were used by UN diplomats working on the Law of the Sea, while Water Deeply reporting was cited and discussed among local water agencies across California. For our newsrooms, these were qualitative indicators that our stories made their marks.
Small newsrooms are often the petri dish for trying new formulas and methodologies. The incumbent newsrooms of our time haven’t traditionally needed measures of impact. For outlets like the New York Times or Washington Post, their impact was an ineffable but assumed fact. In contrast, the digital news upstarts had to prove their work was making a mark. Their innovations in measuring impact now benefit the entire digital news landscape.
What may be hardest to capture for any newsroom is storytelling’s impact far into the future, as a contribution to the public record. One of our proudest moments at News Deeply was when the Library of Congress asked our permission to archive all of our stories, citing their historical value. I imagined a Middle East expert in 2047 reading our articles about the early days of the Syria crisis, or a future water management professional looking for case studies from the 2015 California drought. I was grateful to consider that indicator of impact, knowing it would never be measured in our time.