“Today is perfect for checking your Pace Report!”

“Keep Zearning!”

“By opening this email, you’ve earned another 100 digital raffle tickets in the Zearn Math Giveaway!”

In partnership with Zearn Math, a nonprofit online math instruction platform used by roughly 25% of U.S. elementary school students, Duckworth and a team of researchers from the University of Pennsylvania’s Behavior Change for Good Initiative launched a megastudy that peppered 140,000 teachers with different types of email prompts to log into the platform’s dashboard each week and check their students’ progress.

Behavioral scientists like Duckworth, who popularized the “ about a decade ago, spend a lot of time trying to pinpoint what, exactly, it is that prompts an individual to sign a form, become an organ donor or click an ad that promises a secure and safe retirement now.

“In the case of education there’s the idea of nudging the students directly,” Duckworth said. “But there’s also the idea that’s less commonly studied, which is, what do you do to nudge the teachers, who are not in complete charge, but have a lot of authority about what is going to happen in the classroom that day? It was clear to us that if we could get the students onto the Zearn platform that their learning would progress. But are they actually going to log in?”

To that end, the team developed 15 different types of intervention emails featuring things like planning prompts, teaching tips, learning goals, digital swag and celebrity endorsements. The goal was to change behavior without mandates, bans or substantial financial incentives — though teachers were enrolled in a giveaway and earned digital raffle tickets every time they opened an email, increasing their chances of winning such prizes as autographed children’s books, stickers and gift cards. 

The researchers then compared the average number of lessons the teachers’ students completed on the Zearn Math platform over four weeks to a control group using Zearn that received only a simple weekly email.

So did it work? Did the emails prompt teachers to log in more regularly? And if so, did the number of lessons their students completed increase? To some degree, yes, it did work. But not at all to the extent that Duckworth and researchers had anticipated. 

The best-performing intervention, which encouraged teachers to log into Zearn Math for an updated report on how their students were doing that week, produced a 5% increase in students’ math progress. Emails that referenced data specific to a teacher’s students — versus those without that information — boosted students’ progress by 2.3%. And teachers who received any of the behaviorally informed email nudge saw their students’ math progress increase by an overall average of 1.9%

Duckworth was sure that the emails featuring famed astrophysicist Neil deGrasse Tyson and literary rockstar Judy Blume would move the needle more than anything else. But teachers were virtually unaffected. 

“We had sexier treatment conditions,” she said. “But no, it turns out, a simple message that says, ‘Hey, your students’ data are here, remember to log in,’ that is what worked the best.”

Notably, the intervention effects were consistent across school socioeconomic status and school type, both public and private. Moreover, they persisted for eight weeks after the email intervention period ended. Collectively, the reminders resulted in students completing an estimated 80,424 additional lessons during the four weeks their teachers received emails, and an estimated 156,117 additional lessons during the following eight weeks.

Yet the limited impact of the email reminders surprised virtually everyone involved with the study: Students whose teachers received any type of behaviorally-informed email reminder only marginally outperformed students whose teachers received a simple email reminder. In fact, the effect was at least 30 times smaller than forecasted by the behavioral scientists who designed interventions, by Zearn Math staff and by a sample of elementary school teachers. 

“It’s a sober reminder that big effects are very rare,” said Duckworth. “In general, we’re finding in our megastudies and what’s emerging across the social sciences is that intervention effects tend to be very small.”

“One of the things that this megastudy has reinforced is a kind of humility about how complicated human beings are and how challenging it is to durably change behavior. A kid is a complicated organism. Teachers are complicated. Schools are complicated,” she continued. “It would be naive to think that you could radically change behavior with these like light touch interventions.”

The findings not only underscore the difficulty of changing behavior, but also the need, Duckworth said, for large-scale, rigorous, empirical research on how to drive impact in math, which is a high-priority subject for education policy experts at the moment. 

Indeed, the findings come at an inflection point for math in the U.S. 

The most recent release of the National Assessment of Educational Progress showed that, nationally, average mathematics scores in 2024 were lower by 3 points among fourth-grade students and lower by 8 points among eighth-grade students compared to their scores in 2019 – the most significant drop since 1990. School districts have struggled to rebound after significant academic setbacks incurred by the COVID-19 pandemic. For math in particular, by the spring of 2022, the average public school student in grades three to eight had lost the equivalent of a half-year of learning.

Compared to students in other developed countries, Americans have ranked in the bottom 25% of students globally on standardized tests of mathematics for decades. U.S. students saw a 13-point drop in their 2022 math results when compared to the 2018 exam — “among the lowest ever measured by PISA in mathematics” for the U.S., according to the Organisation for Economic Co-operation and Development, which administers the exam. 

As a result, a contentious debate has erupted surrounding whether educators are effectively teaching the subject — and whether they themselves are being effectively taught how to teach it. 

“There was a dawning realization that there’s a real urgency around math achievement in the United States,” Duckworth said when her team decided to design the megastudy. “This very light touch nudge was helpful, but it does underscore how hard behavior is to change. And if there are bigger levers to influence teacher behavior, I think we would have found a bigger downstream effect on student achievement.”

Gov. Mike DeWine and the state Department of Education are paying more than $7.5 million in federal Covid relief funds to make the online lessons available for all Ohio middle school students for the next three school years. Students can log in to over 1,000 digital math lessons in class, with tutors and even at home tailored to their abilities and learning needs.

Along with adding support for in-person tutoring at schools, Zearn Math is the state’s biggest new strategy for pandemic recovery for this upcoming school year. 

“One priority…is to provide students with tools to accelerate their learning, both for literacy and numeracy,” said Chris Woolard, Ohio’s interim state education superintendent. “Zearn Math is providing Ohio students increased opportunities to access grade-level math concepts no matter where they are.”

Though student test scores in all subjects fell during the pandemic, math has been a particular concern in Ohio and nationally. In Ohio, the statewide proficiency rate in math fell 14 points from 66 percent to 52 percent during the pandemic, compared to eight in reading from 67 to 59 percent. Though scores for middle schoolers have rebounded some, proficiency rates for sixth, seventh and eighth grades were all still down by at least 10 points in 2022 results.

Score drops are higher for Black and low-income students, as well as for students learning English as a second language.

Data from Ohio’s spring 2023 tests will be released next month.

Nationally, the most recent National Assessment of Educational Progress scores from 2022 showed the biggest drop in math for 4th and 8th graders since 1990, sparking calls for better intervention to help students.

How much Ohio teachers will use Zearn is still unclear. It’s not required and has not been well-promoted. Officials of the Ohio Council of Teachers of Mathematics said DeWine and the state did not consult them before contracting with Zearn.

“The announcement was the first that we had heard about it,” said Bowling Green State University Professor Gabriel Matney, the council’s president.

Others said they were unfamiliar with the program, though the state has planned online webinars in September and October about how to use it and Zearn has since contacted the council.

Other Ohio educators are skeptical. Chad VanArnhem, superintendent of the Kirtland school district, an affluent rural district near Cleveland, said Zearn Math helped his students a few years ago — at first.

“We have found that online platforms and the gamification that comes with a lot of them, lose their luster after a few years,” VanArnhem said. “Students eventually became less interested and motivated so we moved to another program.”

A nonprofit company in New York City, Zearn claims to be used by one fourth of all elementary students in the U.S. and a million middle school students. It has gained prominence since the pandemic when the company made national news by providing some of the earliest national data about falling math scores and student recovery in the country.

The Bill and Melinda Gates Foundation is making Zearn part of its $1.1 billion push to improve national math skills over the next four years. And states like Colorado, Delaware, Louisiana, Massachusetts, Tennessee and Texas, along with Washington, D.C., are using Zearn in math tutoring and classes. And because the online program was readily available during pandemic school closures, Zearn’s sales grew 20 times larger from 2019 to 2021, according to one researcher.

Zearn has enthusiastic support from officials like Matthew Blomsted, Nebraska’s education commissioner, who told Congress last fall that it has been a big part of math recovery there since 2021, both during the school year and in summer programs.

The data on Zearn’s effectiveness is limited or not done by independent or accomplished researchers. Improvement results cited by Blomsted were partially prepared by Zearn. Boasts on its website of students in Washington, D.C.,and Louisiana making gains compared to non-Zearn users are backed by Zearn reports. A claim of students making 1.3 years of progress using Zearn in a single school year is attributed to a Harvard graduate student, not a professor.

A 2019 study by Johns Hopkins University researchers showed preliminary promising results, though cautioned that use of the program can be inconsistent, along with how well teachers are trained in using the program, so measuring gains can be hard.

Since the start of the current school year, the gap had been closing, but in December, the Omicron variant sent school systems back into disarray. Between Nov. 28 and Jan. 9, Zearn use among students in prosperous school districts fell 2 percent. In school systems with concentrations of poverty, however, it plunged 13 percent.

Dramatic, yes, but the next discovery was the bigger surprise. While students who were able to attend school in person have better weathered the pandemic academically, Sharma was stunned to see that Zearn’s new data did not correlate to places where schools had shifted to distance learning because of the variant. 

This time, the gaps appear to be biggest where COVID-19 infection rates are highest.

“While schools are open amidst Omicron surges, students from low-income communities are missing critical instructional time,” the report states. “While Omicron is everywhere, its inequitable effects are not.”

It’s a preliminary snapshot, Sharma says, but its implications merit immediate attention: Keeping schools open for in-person classes is not enough. Schools need better plans for preventing new disruptions from interrupting student learning. Zearn’s researchers aren’t certain what’s happening, but they have suggestions about what education leaders should consider.

“Districts that have remained open have experienced , either because they or someone in their family has contracted the virus or they have needed to quarantine for exposure,” the Zearn report notes. “In many districts, particularly those that serve students from low-income communities, the .”

has found that up to 12 million students still lack reliable internet connections. And while school districts have spent billions in federal relief funds on technology, it’s not clear to Sharma that students have adequate access to them this year.

“Do these kids have devices at home if they need to quarantine, if their parents need to quarantine, if their teacher is sick?” Sharma wonders. “Because it looks like they don’t. Why aren’t they logging in? My wondering is, have we spent money but not actually solved the digital divide? Are they letting the computers go home? I think they’re not.”

The data Zearn collects remains one of the nation’s only real-time indicators of children’s math participation and achievement. Economists at Opportunity Insights, jointly run by researchers at Harvard and Brown universities, as part of an economic tracker documenting the pandemic’s inequitable impacts on different socioeconomic groups. 

����Ӱ�� last year took a deep dive into how the Opportunity Insights data was showing up in schools. Academic assessments, surveys of student and educator mental health and other sources of information have since backed up many of the predictions researchers made in those stories.

When the gap yawned open again in December, Zearn researchers first compared locations where app use had plummeted against a closure tracker maintained by Burbio, which catalogues disruptions to in-person schooling nationwide. Surprised to see no correlation, they tried the same exercise using the New York Times’ interactive case-count tracker — to a very different result. 

In December, as Omicron’s disruptions were just beginning, McKinsey & Co. released an analysis that showed the achievement gap has widened by a third. Before the pandemic, students in majority-Black schools were nine months behind their peers. Now they are a full year behind.

The new disparity will compound existing gaps, Sharma predicts: “The kids who missed the most [at the start of the crisis] are now again missing the most.”

Disclosure: The Bill & Melinda Gates Foundation, Carnegie Corporation of New York, Chan Zuckerberg Initiative and Charles and Lynn Schusterman Family Foundation provide financial support to Zearn and ����Ӱ��. The Bill & Melinda Gates Foundation and Chan Zuckerberg Initiative provide financial support to Opportunity Insights and ����Ӱ��.