, released last week through the National Bureau of Economic Research, finds that charter students in the Bay State are significantly more likely to enroll in a four-year college and obtain a degree than their non-charter peers.

But an odd wrinkle emerged: Students in urban charters also experience a noticeable bump in their test scores, while those enrolled outside cities actually see their scores fall. 

The overall effects offer yet more evidence that the Massachusetts charter sector, as the highest-performing in the country, substantially improves the life outcomes of its charges. The state has long won praise for holding choice schools to high standards, shuttering programs that fall short of expectations and allowing only charter organizations with a proven record of success to open new campuses. By the 2000s, charter students in Boston had begun out-scoring children in much more affluent towns on math and English. 

But the new research seems to indicate a paradox. Unlike in Boston, charters in suburban and rural areas boost their students’ chances of attending and graduating from college while also dragging their test scores downwards. The divergent measures of educational achievement make it unclear exactly how the schools are working and what truly matters for kids.

Sarah Cohodes, a professor at the University of Michigan and the paper’s lead author, said her work reflects the simple reality that schools can change students’ lives in a multitude of ways.

“The whole premise of test-based accountability is that test-scores predict longer-term outcomes,” Cohodes wrote in an email. “And that is likely still the case, writ large. But this situation shows it is not always the case, and other things are going on in schools.” 

Cohodes’s analysis revisits the conclusions of by, among others, Nobel Prize-winning economist Joshua Angrist. That experiment showed Massachusetts’s urban charter schools significantly beating the results of nearby public schools, while non-urban charters lagged far behind local competition. 

The latest study makes use of the same sample of 15 urban charter schools and nine non-urban charter schools. It also relies on the same identifying data from the schools’ attendance lotteries, which include information on student race, class background, special education status, and previous scores on the state’s annual standardized test, the Massachusetts Comprehensive Assessment System (MCAS). 

Through the use of the school lotteries, which randomly assigned similar students to either receive a slot at a charter school or not, both studies are able to pinpoint the effects of enrollment. But Cohodes extended her observations further in time, capturing high school graduating classes between 2006 and 2018, and gathered further figures on college enrollment and completion from the . 

Importantly, she identified large differences between charter students based on whether or not they lived in a city. Black and Latino students made up 54 percent and 27 percent of applicants, respectively, at urban charters, while fully 90 percent of applicants to non-urban charters were white. Urban applicants were also much more likely to qualify for free or reduced-price lunch (a common proxy for poverty), and had previously scored considerably below the state average on MCAS; non-urban applicants tended to score above that average.

After two years of attending their charter school, urban students saw their scores in both math and English leap upwards compared with students in traditional public schools. By comparison, those in non-urban charters fell by somewhat smaller, though still significant, amounts.

That finding replicates both the results from the 2013 study in Massachusetts and those of , which have broadly pointed to a divide between urban charters and those in rural and suburban areas. The consistency of the result suggested to some observers that it could simply be easier to create a charter school that ; in more advantaged areas, however, newer alternatives must compete against schools where students already score fairly well.

Surprisingly, though, the same students whose scores fell in non-urban charters were also 11 percentage points more likely to enroll in a four-year college than their counterparts in traditional public schools. They were also 10 points more likely to attain a bachelor’s degree. Urban charter students also saw their college chances improve — 27 percent earned a bachelor’s degree within six years of graduating high school, compared with 23 percent of their peers in non-charters — but the effect was only about half that enjoyed by students outside of cities.

What could account for the difference? According to Jon Valant, a political scientist who leads the Brookings Institution’s Brown Center on Education Policy, charter-curious families in non-urban areas could be selecting for schools that don’t focus explicitly on raising test scores. Instead, their target schools might attempt to set themselves apart through a focus on the arts or social-emotional learning. Such an emphasis could boost chances of college completion while also leading to lower academic achievement in the short-run.

“In those areas, parents might not be looking for schools that are better at doing the same things as their local public schools,” Valant wrote in an email. “They might be looking for schools that do something different — even if that comes at the expense of their state test scores.”

That sentiment was echoed by Macke Raymond, the director of Stanford’s (CREDO), which conducts comprehensive reviews of charter school performance around the United States. While cautioning that the Cohodes study’s sample of just a few dozen schools made its findings difficult to generalize, Raymond argued in an email that suburban parents often strike a bargain when selecting charters: The alternative school model might provide academic and social resources that help their children excel in college, even while their explicit focus on core subjects falls somewhat behind that of local schools.

“Our team has seen that many non-urban charter schools across the country intentionally offer students an experience that does not focus on maximizing academic gains,” Raymond wrote. “Knowing that families are well resourced, suburban charters may offer a different experience, either with thematic focus or emphasize an environment that stresses non-academic development of their students.”

For her part, Cohodes said that while the correlation between test scores and later-life success is solidly established, it was important not to dismiss educational programs too hastily on the basis of setbacks on student assessments. She and her colleagues plan to conduct a follow-up study examining the practices in non-urban charters that might be contributing to their students’ post-secondary attainment, including smaller class sizes and college counseling.

“I think it’s important to find school models that work, and to define ‘work’ broadly such that it does not incorporate only test scores,” Cohodes said. “And I think we should replicate and expand school models that work, regardless of the sector.”

New research released last month has offered persuasive new evidence of the potential of summer learning opportunities, particularly in STEM subjects. One, a meta-analysis compiling the findings of dozens of prior studies over the last two years, shows consistent gains in math achievement resulting from student enrollment in summer coursework. Another showed participants in a summer STEM program enjoying significant later-life benefits, including greater success in college and higher earnings. 

The papers emerged just as national leaders made a concerted push to broaden access to summer instruction. In July, to spend more of their federal relief funds on tutoring, afterschool activities, and summer enrichment. Next, the Department of Education the Engage Every Student Initiative, a public-private partnership designed to guide local communities toward evidence-based programming. The administration to highlight the work of schools that have expanded their summer offerings.

The campaign demonstrates the promise that many experts see in summer learning — and the enormous academic challenges facing the nation’s schools after three school years disrupted by COVID-19. Along with extended school days and a stiff dose of high-quality tutoring, researchers and policymakers alike are turning to the traditionally vacant summer months as an untapped resource in the battle against academic erosion. 

Kathleen Lynch, an assistant professor at the University of Connecticut and coauthor of the meta-analysis, said the existing research shows not only that summer learning is an effective means of bolstering academic growth, but also a worthy recipient of finite COVID recovery dollars.

“Summer programs provide an opportunity for children to catch up on material they may have missed, or to enrich their learning on new topics aligned with their interests,” Lynch wrote in an email. “I would recommend an effort to replicate successful models over the next few years, as schools and districts continue to combat learning setbacks that children experienced due to the pandemic.”  

Lynch and her co-authors cast a wide net to gather relevant findings from existing research dating between 1998 and 2020, ultimately selecting 37 studies of summer math initiatives that included control groups against whom program effects could be assessed. Programs could be conducted in a school, a community site, or private homes, and while some of the experiments were exclusively math-focused, others provided instruction in other subjects as well.

Participation in the programs significantly lifted children’s math performance. The average effect size of .1 standard deviations (a common measure showing the difference in any group from the statistical mean) in improved standardized test scores compares favorably to other touted learning interventions, such as teacher merit pay and school choice. And the benefits were similar in scope regardless of whether a given program served primarily low-income or high-income children. 

That distinction is critical given the intense diversity of summer learning experiences. Many are operated by school districts on a remedial basis, recruiting (or requiring the participation of) students who struggled academically during the year. Historically, these forms of summer school with poor attendance and low engagement from participants.

By contrast, Lynch noted, “contemporary summer programs increasingly focus on enrichment, hands-on activities, and learning via projects and inquiry.” Such programs, offered electively, are more likely to attract high-achieving pupils from relatively advantaged families.

focused on a particular initiative that attempted to split the difference by signing up high-achieving students from racial or ethnic backgrounds that are historically underrepresented in STEM fields. The program, offered by an elite technical university located in the Northeast, draws a disproportionately nonwhite field of rising high school seniors with top test scores and an average GPA of 3.86. 

Researchers from Columbia Teachers College, the Harvard Kennedy School, and the consulting company Mathematica assessed the effects of three separate varieties of the program: two summer residential periods (one week and six weeks, respectively) on campus, complete with direct coursework in STEM subjects as well as workshops and visits to STEM-focused workplaces, as well as a six-month engagement that was primarily offered to participants online. 

In all, participants from the 2014, 2015, and 2016 cohorts of experiment gained impressive life advantages in the years to come. Across all three summer offerings, students were more likely than members of a demographically similar control group to enroll in college, as well as persist and finish with a degree. Perhaps most importantly, since the program’s top priority was to diversify the STEM pipeline, participants offered seats in the six-week residential experience were 33 percent more likely to graduate in four years with a STEM degree. 

Sarah Cohodes, an associate professor of economics and education at Teachers College, Columbia University, and a co-author of the study, said that the experiment provides evidence of a somewhat rarefied type of summer learning opportunity, tailored to students who were likely to enjoy its full benefits. That makes it a limited, though suggestive, window into what can be expected from summer school generally.

“Does it look like what we’re thinking about when we’re thinking about remediating learning loss? No, it doesn’t,” Cohodes said. “But I think you can see this as an existence proof that, yes, carefully designed programs targeted at the right level for students can make a huge difference for their life trajectories, and it is possible to create summer opportunities that change the lives of students.” 

Intriguingly, the study’s findings in terms of college outcomes aren’t clearly attributable to a particular facet of the college program; for instance, graduation rates after five years with a STEM degree were not significantly different in the one-week experience versus the six-week experience. This suggests that the benefits might be attributable to the simple influence of gathering students from traditionally underrepresented groups together on a prestigious campus, Cohodes argued.

“It’s not clear that the learning that made a difference here was standard, ‘I know more physics than I knew before’-type learning,” she observed. “A lot of it seemed to be around knowledge of the college application process, knowledge of what was out there, peer effects and social networks.”

The development of non-cognitive skills and traits was an explicit point of focus in Lynch’s compilation of summer learning studies. Across a range of 37 non-cognitive outcomes (including mindsets and attitudes, social skills, and academic behaviors like school attendance), summer math programs were associated with positive movement in 27; the average effect size for those outcomes was roughly equivalent to the programs’ effects on math test scores and course grades, with notable reductions to school-year absenteeism.

“The number of studies that measured noncognitive impacts is relatively small, but the evidence we found suggested that there’s unlikely to be a tradeoff between learning and noncognitive outcomes from attending summer programs,” Lynch said.

One example singled out in the meta-analysis was the Horizons National Summer Enrichment Program, an intensive summer intervention serving thousands of low-income pre-K–8 students across dozens of affiliates in 20 states. A commissioned by the organization found that its enrollees were less likely to be chronically absent or repeat a grade. A Horizons affiliate in New Haven, Connecticut, on the first lady’s July tour of summer learning and enrichment programs.

As policymakers at the state and federal levels search for tools to restore the academic growth forfeited during the pandemic, they will have access to thousands of existing summer schools, camps, and enrichment activities targeted toward K-12 students of different ages and achievement levels. National Summer Learning Association CEO Aaron Dworkin, who accompanied First Lady Biden on her visit to Horizons, said in an interview that this panoply of approaches — wedded to ample government support — could make a significant impact in the next few years.

“We have a lot of people who are doing what they think is best, but we can support and train them and invest in them so that they don’t have to reinvent the wheel. A lot of people have tried already and learned the hard way. What’s different is that we have a lot of training, data, intermediaries, and infrastructure to support all kinds of people who are trying to be helpful right now.”

The study, circulated as by the National Bureau of Economic Research, represents the latest evidence pointing to some charters as institutions that strengthen civic engagement. A that focused on North Carolina schools found lasting benefits to traditionally underserved students, including more frequent voting and reduced criminality, who attended a charter secondary school rather than a traditional public school.

And both echo the findings of of the civics-focused Democracy Prep charter schools. Graduates of the network, which operates over 20 schools across five states, were 12 percentage points more likely to vote in the 2016 presidential election than similar students, according to that study, and substantially more likely to be registered as voters. 

Sarah Cohodes, an economist at Columbia University’s Teachers College and a co-author of the Boston paper, noted that the voting effects she found were about half as large as those generated by Democracy Prep — six percentage points of increased voting likelihood, from a status quo of 35 percent — and that she measured no impact on registration. But a network like Democracy Prep, which persistently emphasizes civic participation and demands that students demonstrate mastery over multiple democratic skills, might be expected to lift voter participation, Cohodes added.

“This [research] is showing that even if you have a school where civics isn’t the mission, but you are still instilling more general skills — executive function, conscientiousness — alongside academic skills, that spills over into voting,” she said.

Cohodes and co-author James Feigenbaum, a professor of economics at Boston University, gathered student records from the Massachusetts Department of Elementary and Secondary Education, along with lottery reports and voter records. The academic data included a battery of student demographic information, as well as performance metrics on state standardized tests, Advanced Placement course enrollment, SAT-taking, and college enrollment and persistence. Their sample included 12 Boston charters that enrolled students who were at least 18 years old at the time of the 2016 U.S. elections.

They then matched those records with Massachusetts voter files drawn from 2012, 2015, and 2018 (as well as files from nearby states New York, Connecticut, New Hampshire, Vermont, Rhode Island, and Maine, to account for out-of-state moves). 

Like many other studies of charter school effectiveness, the analysis relies on the lottery mechanism that randomly assigns admissions to Boston’s heavily oversubscribed charter sector. Lottery “winners” (students who are ultimately enrolled in the charters) are broadly similar to lottery “losers” in terms of racial and ethnic background, socioeconomic status, and prior academic performance.

After comparing the two sets of data, the authors found that charter attendance increased students’ incidence of voting in their first presidential election after turning 18 by about 17 percent. That effect is particularly noteworthy because the study found that charter attendance did not seem to increase voter registration, as one of the biggest procedural barriers preventing people from turning out on Election Day.

But within those results, an even more striking pattern emerged: The average increase in voting is the result of an especially large boost to female charter students — 12.5 percentage points — and no corresponding rise among males. That outcome generally mirrors , which have increasingly shown females outvoting males in recent years. 

To isolate a possible explanation for the gender split, the authors studied the various ways in which Boston charters affected their pupils compared with traditional public schools, including academic aptitude (measured through test scores), civic skills (measured through enrollment in an AP government or U.S. history class), and non-cognitive abilities (measured through school attendance and a student’s decision to take the SAT). Ultimately, the third category was the only realm in which a similar gender disparity existed, showing significant increases for girls compared with boys.

That detail is reminiscent of research conducted by political scientist John Holbein, who has found that high school students who are more likely to describe themselves as gritty are also more frequent voters. The link between non-cognitive skills like grit and persistence and voting propensity could be due to the obstacles that often stand in the way of filling out a ballot, Cohodes argued.

“You have to register, you have to find your polling place, you have to make your plan for getting there and getting off of work,” she said. “And then you actually have to show up and do it all. That involves persistence and follow-through, and…that’s where I see those schools coming in.”

It’s unclear whether charter schools in Boston are aiding the cultivation of such follow-through in female, but not male, students — or, perhaps, that they are burnishing those qualities in equal measure, but that boys begin school already far behind their female classmates. In either instance, Cohodes concluded, the findings provide more reason to think that the civic byproducts of charter schooling could be as consistent as their academic effects, which have largely been shown to be replicable across different settings and charter models.

“I do think it’s a different dimension of skill from academics, so it’s not necessarily the case that the schools that are bringing up test scores the most are also bringing up voting the most. But the things that Boston charters do are also things that KIPP schools do, that STRIVE charters and others do. So it’s not like it’s something that’s totally out of left field.”