The Fact-Check – ��Ӱ��

��Ӱ�� Fact-Check: Are Teachers Really Burning Out Because of Tougher Tests and Evaluations?

Matt Barnum — Sat, 01 Jan 2000 00:00:00 +0000

The Fact-Check is ��Ӱ��’s ongoing series that examines the ways in which journalists, politicians and leaders misuse or misrepresent schools, education data and research. See our complete Fact-Check archive.

“Many educators are leaving the field because of what is being done to schools in the name of ‘accountability’ and ‘tougher standards,’” wrote Alfie Kohn in . “I have no hard numbers here, but there is more than enough anecdotal evidence — corroborated by administrators, teacher-educators, and other observers across the country, and supported by several state surveys that quantify the extent of disenchantment with testing — to warrant classifying this as a fact.”

Kohn’s essay appeared all the way back in 2000, but evidently, little has changed since to challenge this commonly accepted narrative: Op-ed pages and education blogs are still quick to spotlight that high-stakes testing, accountability and education reform — ahem, “reform” — have piled stress on top of teachers, driving an increasing number of educators from the profession and stopping others from entering it altogether.

The latest in this genre comes from Timothy Walker, this month in The Atlantic, who quotes a teacher saying, “[T]he system that we have right now in America, which is focusing on test scores and accountability, and has teachers being pulled in so many different directions at once, has got so many different pressures coming from so many different places.”

Walker proceeds to cite an unscientific (“circulated via email and social media”) of teachers conducted by the American Federation of Teachers and Badass Teachers Association and a single Colorado teacher who after 25 years because of too much paperwork.

In another recent example, Education Week blogger Marc Tucker much-discussed (and debated) teacher shortages in part to “a growing number of teachers … getting sacked on the basis of the test performance of their students.”

A earlier this year in The Washington Post proclaimed, “In an era of fierce debate about public education, morale among teachers has taken a nosedive, according to national polls.”

Such dire coverage sparks two important questions: Is the teaching profession becoming less appealing — as measured by turnover, job satisfaction and interest in the profession? And if so, are testing and accountability policies part of the explanation?

On both questions, the evidence fails to support the sweeping declarations about the desirability of a teaching career and the impact of reform policies on educators.

Interest in becoming a teacher is down — but so is teacher turnover

Only one poll to my knowledge shows that teacher morale has taken a “nosedive.” This was a widely cited conducted in 2012 that found that only 39 percent of teachers reported being satisfied with their jobs, compared with a high-water mark of 62 percent in 2008. It’s not clear what caused this decline, but the survey finds that declining school funding — a common phenomenon in the aftermath of the Great Recession — was associated with lower morale.

I am not aware of other scientific surveys that have shown drops in teachers’ job satisfaction. earlier this year found that about 60 percent of teachers reported that they and their colleagues were generally satisfied, though a similar number said they didn’t have as much enthusiasm for the job as when they began teaching.

But, setting satisfaction aside, what about teacher retention? The numbers show that in recent years teachers have been more likely to stay in the classroom. A 2015 issued by the federal government looked at teachers who began their career in the 2007–08 school year and found that only 17 percent were not in the classroom five years later — much than previous estimates of 50 percent attrition over five years. Again, a possible cause of this phenomenon is the economic downturn, as many teachers may have been unlikely to quit while few jobs were available.

Let’s turn next to the steep decline in new students signing up for teacher training programs, which some on accountability measures. The evidence for this view is scarce. One careful analysis found three state policies strongly associated with drops in enrollment: the number of teachers laid off in recent years, whether layoffs were conducted based on seniority, and teacher salary. However, there was no statistically significant relationship between enrollment declines and whether a state linked test scores to teacher evaluations — a high-profile and controversial policy backed by the Obama administration.

This study is not definitive for determining cause and effect, but it does suggest that bread-and-butter issues like pay and layoffs — rather than accountability, standards and testing policies — are the main factors driving away potential educators.

Finally, there is research, both and in , suggesting that more academically qualified individuals have entered teaching starting in the early 21st century, though there doesn’t appear to be more recent evidence available. The national study notes, “We find that new teachers in high-stakes classrooms tend to have higher SAT scores than those in other classrooms. … Test-based accountability greatly increased after the 2001 passage of NCLB, but we see no evidence that more academically proficient teachers entering the workforce in the year immediately following graduation are shying away from (or at least are not being assigned to) high-stakes classrooms."

Research findings conflict on how testing and evaluations are affecting teacher turnover

Despite the rhetoric, there is little evidence suggesting that testing and evaluation policies have led to across-the-board reductions in teacher retention or job satisfaction. (I asked two researchers — Dan Goldhaber of the University of Washington and Matt Kraft of Brown University — and both agreed there isn’t a great deal of research on the question.)

But what studies exist suggest that accountability policies have not had large impacts on teacher retention or satisfaction.

For instance, a trio of researchers the now-infamous No Child Left Behind (NCLB), using national data, found that “perhaps surprisingly” there were “positive trends in many work environment measures, job satisfaction, and commitment across the time period coinciding with the implementation of NCLB.” In trying to isolate the impact of the law itself, the research estimated little effect one way or another.

“Simply stated, our results do not support media accounts, academic reports, or policy rhetoric more generally that portray NCLB as undermining teacher morale and intent to remain in the profession,” the authors concluded.

looked at the teacher attrition after New York introduced testing for fourth-grade students in the 1990s: Again, the researchers note surprise that turnover among fourth-grade teachers actually dropped.

A recent paper showed that Chicago’s pilot teacher evaluation system had no effect on teacher turnover writ large — except for the lowest-performing teachers. Another pair of studies found that, unsurprisingly, teachers who receive higher evaluation scores are and in the classroom than colleagues who receive lower performance ratings.

This suggests that the rise of evaluation systems may not have had negative across-the-board effects on teachers but could induce turnover among low-rated teachers — arguably a desirable effect of the policy.

On the other hand, there is evidence that teachers in specific schools facing stronger accountability pressure experience greater stress and higher turnover.

of NCLB found that in schools trying to avoid sanctions under the law, teachers felt less job security and untenured teachers worked longer hours. Notably, the study found that students seemed to benefit from the accountability in the form of slightly higher scores on low-stakes exams.

A in Florida, which uses letter grades to judge schools, found that F-rated schools saw an uptick in attrition among high-performing teachers. Similarly, North Carolina’s old accountability system an increase in teacher turnover. Still, in Florida, those schools saw performance gains.

One argued that test-based accountability policies are associated with increases in teacher stress, though it’s not clear whether the research was able to isolate cause and effect.

Those concerned about testing and accountability policies driving teachers from the classroom often cite surveys of teachers who depart from the classroom. Indeed, about a quarter of non-retiring teachers who left the profession they quit was that they were “dissatisfied with student assessments and school accountability measures.” Seventeen percent cited dissatisfaction “with support preparing students for assessments.”

However, social scientists, including those in , have long cautioned against reading too much into self-reported surveys. This survey allowed teachers to select multiple responses, so it is unclear how many teachers identified testing policies as the primary reason for leaving.

One concerning note, however, is that teachers of color to cite dissatisfaction with testing and accountability as a reason for leaving. This may help explain other evidence showing that non-white teachers have higher turnover and dissatisfaction rates than white teachers.

To summarize, as of the published evidence put it: “Research to date suggests that accountability has not dramatically changed the career choices of teachers overall, but that it has likely increased attrition in schools classified as failing relative to other schools.” There is less research on teacher evaluation policies, but what exists suggests that turnover and dissatisfaction may be particularly acute for teachers who receive poor ratings.

The research doesn’t support the sweeping claims about why teachers quit

There are many potential explanations for why research is not aligning with conventional wisdom on this issue. Maybe the existing studies aren’t sophisticated enough. Maybe there isn’t enough research out yet on the new breed of evaluation systems. Maybe accountability systems — including evaluations that continue to rate the vast majority of teachers as effective — simply do not directly affect most teachers.

And although a four-year-old study shows a drop in teacher satisfaction, and enrollment in teacher prep programs has fallen, other indicators, like retention and teachers’ academic credentials, paint a more encouraging picture.

It is certainly legitimate about the effect of new education policies on the teaching workforce, and it is clear that working conditions matter a great deal for teacher retention.

However, panicked headlines and broad assertions about testing or standards driving our teachers out of the classroom are simply not supported by the existing evidence. If we want to solve the problems of teacher dissatisfaction and turnover — as well as declining interest in the profession — the first step is a correct diagnosis.

Footnotes:

1. Some suggest that because of non-response bias, this survey may underestimate the number of teachers who left the classroom. Adjusting for this, another suggested that the turnover rate was 19 percent over five years.

��Ӱ�� Fact-Check: All the Ways Donald Trump and Ted Cruz Flubbed on Education at Thursday’s Debate

Carolyn Phenicie — Sat, 01 Jan 2000 00:00:00 +0000

The Fact-Check is The Seventy Four’s ongoing series that examines the ways in which journalists, politicians and leaders misuse or misrepresent schools, education data and research. See our complete Fact-Check archive.

As the 2016 presidential primaries seem like they’re maybe, hopefully, beginning to wind to a close, the American people still haven’t heard much from the remaining contenders on the issue of K-12 education.

We’ve learned that Hillary Clinton thinks unions need to “eliminate” the criticism that they protect bad teachers. Florida Sen. Marco Rubio wants American kids to be welders instead of philosophers. And last night, GOP frontrunner Donald Trump, who’s been largely mum on grade schools beyond his hatred of Common Core, said he’d have one-time rival Ben Carson “very involved with education, something that’s an expertise of his.” (Read Carson’s exclusive interview with ��Ӱ�� on K-12 education issues.)

Through it all, facts have often been in short supply, replaced instead by sweeping rhetoric about our kids falling behind those of other countries, or how the feds have imposed their will on our neighborhood schools.

But in Thursday night’s Republican debate, both GOP frontrunner Donald Trump and his closest rival, Sen. Ted Cruz, uttered a series of eye-popping proclamations so removed from reality they demand a stiff antidote of truth to bring them back to Earth:

TRUMP: “It [the Common Core] has been taken over by the federal government. It was originally supposed to be that way [developed by states and voluntarily implemented by states and districts]. And certainly sounds better that way. But it has all been taken over now by the bureaucrats in Washington.”

FACT: It was the states — more specifically the Council of Chief State School Officers and National Governors Association — that developed the standards. During the Obama administration, the Education Department has played no specific role in the implementation of those standards, and the classroom curriculum used to meet the broad goals set out in Common Core is created by districts and states, as it always has been. Further, states have made tweaks to the Common Core standards since their initial adoption and, in some cases, have decided to drop the standards entirely. (Check out our Common Core flash cards.)

CRUZ: “The Obama administration has abused executive power in forcing Common Core on the states. It has used Race to the Top funds to effectively blackmail and force the states to adopt Common Core.”

FACT: States competing for Race to the Top funds in 2009 got more points on their application for the adoption of “internationally benchmarked standards and assessments that prepare students for success in college and the workplace.” Adopting those standards won a state 40 points out of 500 possible, .

Congress has not funded Race to the Top grants in the annual appropriations process for several years, and several states – notably Oklahoma and Indiana – have dropped the Common Core.

Former Louisiana Gov. and GOP presidential candidate Bobby Jindal sued former Secretary Arne Duncan and the department, alleging that they offered an illegal “quid pro quo” to states to adopt the standards in exchange for Race to the Top money. A federal judge in September.

CRUZ: “The one silver lining of Obama abusing executive power is that everything done with executive power can be undone with executive power, and I intend to do that.”

FACT: Federal law already prohibits the government from forcing states to adopt Common Core.

The Every Student Succeeds Act, which Obama signed into law in December, includes 13 references to the Common Core – all limitations on federal power to meddle in curriculum.

Specifically from the law: “No officer or employee of the federal government shall, through grants, contracts, or other cooperative agreements, mandate, direct, or control a state, local education agency, or school’s specific instructional content, academic standards and assessments, curricula, or other program of instruction…including any requirement, direction, or mandate to adopt the Common Core State Standards.”

To the contrary, ESSA specifically protects states’ rights to “enter into a voluntary partnership with another state to develop and implement” challenging academic standards.

CRUZ: “I intend to work to abolish the federal Department of Education and send education back to the states and back to the local governments.”

FACT: The federal government already has a limited role in K-12 education. Particularly in the wake of the passage of the Every Student Succeeds Act, the primary federal roles are providing supplemental funds for the education of children in poverty (the Title I program), setting standards for the education of children with disabilities and helping fund those services (the Individuals with Disabilities Education Act), and ensuring children don’t go hungry (the school lunch program, which is run through the Agriculture Department.)

The monetary role is small, too. , between 1980 and 2011, between 7 and 13 percent of total annual education funding came from federal sources. And only about half of that funding in 2011. Another quarter of that funding came from the Department of Agriculture for the school lunch program. The Defense Department (junior reserve officers’ training program and their own school system for students of military members), Health and Human Services (Head Start pre-school) and about a half-dozen other departments for smaller programs made up the rest.

CRUZ: “The most important reform we can do in education after getting the federal government out of it, is expand school choice; expand charter schools and home schools and private schools and vouchers, and scholarships.”

FACT: Abolishing the federal Education Department would also wipe out the Office of Innovation and Improvement, which oversees the very initiatives Cruz wants to promote: federal efforts to spur more charter schools and magnet schools; the DC Opportunity Scholarship Program, the only federal school voucher program; and the Office of Non-Public Education.

Opinion: Don’t Humiliate Teachers… But Fire the Worst: Does David Denby Realize He’s Agreeing With Reformers?

Matt Barnum — Sat, 01 Jan 2000 00:00:00 +0000

The Fact-Check is The Seventy Four’s ongoing series that examines the ways in which journalists, politicians and leaders misuse or misinterpret education data and research. See our complete Fact-Check archive.

The New Yorker’s David Denby is tired of the war on teachers. In a titled, “Stop Humiliating Teachers,” Denby declares, “We have to stop blaming teachers for all of the ills and injustices of American society.”

Denby doesn’t really say who’s doing this specifically, except for vaguely defined “reformers.” But his point is clear: testing, tough teacher evaluation, and charter schools are bad.

Denby’s essay is flush with talking points but thin on facts, concluding with a head-spinning twist ending where he concedes that reformers are right about a fundamental point.

What Denby gets wrong (a lot)

Denby’s piece is riddled with contradictions and backed by confident assertions offered often without the slightest evidence.

Denby declares, “In December, the Obama Administration pulled the plug on No Child Left Behind, deputizing the states to administer tests and to reward and punish — a de-facto admission that the program wasn’t working well.”

Not really. It may have been a failure as a political matter, the media has declared it a disaster, and it certainly had problems that needed to be addressed. But researchers have generally found that the law improved student achievement in math.

Denby: “As recent surveys have shown, the high-stakes testing mania has demoralized the profession as whole.”

I don’t know what Denby is referring to because he does not cite any source. I suspect he may be talking about an — which was sent to respondents “by email and social media” — of teachers conducted jointly by the American Federation of Teachers and the Badass Teachers Association. There is certainly evidence that some teachers are frustrated by accountability reforms, but research has actually found that contrary to conventional wisdom No Child Left Behind voluntary teacher turnover and teachers’ job satisfaction. Maybe that’s changed with recent reforms — it certainly wouldn’t shock me — but Denby offers no evidence to support this claim.

Denby: “Using … tests to evaluate teachers themselves has been questioned again and again by statistical experts as well as by critics of these programs. The heart of the criticism: the tests measure demographics (the class and wealth level of the students) more than teachers’ abilities.”

Denby does not appear to have even a passing familiarity with the way many teachers are evaluated via test scores. It’s often through a statistical formula, known as value-added, which statistically controls for past achievement and demographic characteristics. (I encourage Denby to read ��Ӱ��’s thoroughly researched flashcards on this issue.) This method is controversial and researchers disagree on what role, if any, it should play in teacher evaluation. Yes, we can discuss whether the method controls enough for student characteristics or whether it’s sufficient to control just for past achievement, as some states do. Yet other measures of teacher performance, such as principal observations, may be even more biased against teachers of disadvantaged students.

Denby complains about what he sees as an obsessive focus on bad teachers, quoting journalist Dana Goldstein who describes it as a “moral panic”; later he points to “linked promotion or dismissal to teachers’ ability to get kids to score well on tests.”

He doesn’t mention that in most places recent reforms have not appeared to lead many teachers to actually be dismissed. New evaluation systems have not generally labeled many teachers ineffective. Denby highlights New York, but as I reported last year, under the supposedly tough new evaluation system just one tenured teacher has been fired because of it, according to the State Education Department. Teachers may have many legitimate frustrations with new evaluation systems, but the new moral panic, it seems to me, is about imaginary mass firings of teachers

Denby says reformers have put too much pressure on teachers to fix societal problems — this after he writes, “At their best, teachers are everyday gods, standing at the entryway to the world.”

Denby says reformers should be focused more on expanding “health services” to poor families. Somehow he doesn’t mention that at the same time the Obama administration was pushing education reform, its signature domestic policy initiative was something we now call Obamacare.

My favorite Denby-ism though is this breezy declaration, offered without a shred of measurable evidence: “Corporate thinking, mostly inappropriate to education, has turned teachers into individual operators potentially at war with one another.”

The war on teachers is silly enough — but now it’s a civil war that’s broken out?

Denby’s twist ending

By far the oddest moment in Denby’s piece is when he flatly agrees with a crucial aspects of the school reform project: “My own feeling is that it should be easier than it is now for principals to fire bad teachers, but that tenure should not be abolished.”

That sounds suspiciously similar to, say, former U.S. Secretary of Education or the reform group , neither of whom supports eliminating due process for teachers, but do want to ensure ineffective teachers can be swiftly dismissed if they do not improve.

Denby also advocates significant increases in teacher pay, something many (though not all) reformers have argued for as well. Little does he seem to know that this utopian vision exists in the arch-reform capital of Washington, D.C. Perhaps Denby would disagree with some specifics, but teachers in D.C. are , and then held accountable for performance based on multiple measures. Research suggests that D.C.’s system of rewarding the best teachers and pushing out the worst improved student achievement, particularly in high-poverty schools.

What Denby gets right

Like Denby, I also have a twist ending. Even though, in my view, he gets a lot wrong — really wrong — Denby also gets some important things right. I think he is correct to argue that reform movement, such as it is, ought to advance a coherent anti-poverty agenda, put more political capital towards raising teacher pay, improve teacher evaluation systems, and do more to cut back on unnecessary testing.

Indeed, some of what Denby recommends — higher teacher salaries, greater efforts to address poverty — are not at odds with the reform agenda. They actually complement it, and many reformers recognize as much. Unfortunately, these areas of potential agreement are lost in Denby’s storm of sweeping claims and broad condemnation.

In many respects the school reform movement is right now. And the fact that the pages of the New Yorker are now filled with fact-free anti-reform talking points ought to be concerning. Yes, reformers should point out where Denby is wrong, but they should also consider where he’s right.

The Fact-Check: Not Every State Takeover of a Troubled School District Deserves an F

Matt Barnum — Sat, 01 Jan 2000 00:00:00 +0000

Are state takeovers of schools good for students?

That’s a question elected officials are wrestling with across the country. Recently, Illinois Gov. Bruce Rauner has proposed taking over Chicago Public Schools, the country’s third-largest district that is facing a huge deficit and the threat of its second teachers strike in four years. A ballot initiative in Georgia would allow for a state-run district of more than 100 schools that are deemed low-performing.

Many don’t think that’s a good idea. A Jan. 27 in EdWeek declared that state takeovers deserve a “failing grade.” A Feb. 1 in The Washington Post said, “Takeovers in Newark, Detroit and Memphis have not improved test scores — in fact, some schools have gone backward.”

The Post story does not point to a single place where takeovers have improved student achievement — but there are some.

And contrary to the sweeping conclusion, the research on state takeovers is more complicated. Policymakers considering this route should be focusing on what’s driven some takeovers to succeed and others to fail — not whether state takeovers are simply “good” or “bad.”

First, the positives: Virtually all New Orleans schools became charters after a state takeover there following the destruction of Hurricane Katrina. Initial by Tulane economist Doug Harris found large positive gains in achievement after the state stepped into a district where student performance had been abysmal for decades. These changes also coincided with a large increase in per pupil spending.

Harris previously told The Seventy Four, “I've never seen an effect of this size before,” comparing the charter reforms to efforts such as class size reduction.

The EdWeek op-ed claims, “New Orleans still reports some of the nation's lowest achievement scores and graduation rates.” But this is a completely unserious response. Achievement of students in New Orleans started at a very low level and efforts to reform the system should be judged on the gains that the state takeover produced.

There are still many questions about student outcomes in New Orleans and how to measure them — for instance, Harris hasn’t published his full study, so fully examining his methods isn’t possible yet — but right now the New Orleans evidence is clearly positive.

More good news comes from Lawrence, Massachusetts, which has been under state control since 2012. A new , released as a working paper from the National Bureau of Economic Research, found that the takeover led to moderately large gains in math achievement and very small gains in reading.

Interestingly, though the turnaround involved a package of reforms — including efforts to improve teacher quality — the researchers found that a large fraction of the improvement could be attributed to extra learning time over vacation breaks for struggling students.

Research beyond these two cities is much less encouraging. A recent Vanderbilt found no gains from Tennessee's state-run district of nearly two dozen schools in Memphis. The state takeover of Detroit has failed to stop the district’s extreme financial distress. A separate takeover effort within the already state-run Detroit schools also , though pinning down the effects on student achievement is difficult.

A 2007 found middling results from a state takeover of Philadelphia schools. They made gains but generally not much more than similar schools in districts that weren’t taken over.

To my knowledge, there are no recent studies that show a state takeover having a negative effect on student achievement.

In sum, state takeovers have a mixed track record. Unfortunately, there’s little evidence why some turnaround efforts seem to have produced results and others haven’t.

There may be reasons to oppose takeovers beyond their effects on student outcomes — such as concerns about — but condemning such efforts writ large based on student achievement simply isn’t justified by the existing evidence.

New York Times Gives Credence to Conservatives’ Common Core Delusion

Matt Barnum — Sat, 01 Jan 2000 00:00:00 +0000

In a of the book “Beyond Measure” for The New York Times, prominent UCLA professor Pedro Noguera takes some swipes at the school reform movement.

Not only have reformers, in his telling, alienated many parents, but, “A number of conservatives now regard No Child Left Behind and the Common Core learning standards as intrusive federal mandates.” In his haste to criticize reform, Noguera has given legitimacy to a falsehood that has long percolated in right-of-center circles: that the Common Core is a federal requirement.

This is unequivocally false. How do we know? Because several states — such as Texas, Virginia, and Alaska — the standards in the first place, while other states have repealed or altered the Common Core, which set out what K–12 students should know and be able to do in math and English. Texas even a law making it illegal for public schools to teach the standards. That doesn’t sound very mandatory to me.

To be clear: yes, people are actually arguing — and the Times has sympathetically printed this without clarification — that the Common Core is mandated despite that many states are not using the Common Core. Welcome to our national education debate.

The truth is that the Common Core was the brainchild of the (non-federal) National Governors Association. The Obama administration liked the standards and provided incentives for states to adopt them through Race to the Top. The Department of Education also conditioned waivers from No Child Left Behind on the adoption of college-and-career-ready standards, which included, but was not limited to the Common Core. Some states that do not use the Common Core have received federal waivers. In other words, the feds weren’t neutral bystanders, but they certainly weren’t enforcers of a mandated set of standards either.

This has not stopped many conservatives and libertarians from misstating the federal role.

Reason, a libertarian magazine, the standards a “massive federal takeover of K–12 education.” The conservative National Review the standards “amount to a national curriculum.” The Washington Examiner that “President Obama essentially forced states to agree to adopt Common Core.” Breitbart’s “Common Core 101” that it is “federally-led education.” Republican U.S. Senator Lamar Alexander, chair of the education committee, that there new federal education law “repeals the federal Common Core mandate.”

(And this doesn’t even touch the more insane Common Core — #Corespiracy — bouncing around social media, such as the Arizona Tea Party’s that it is “part of a United Nations plan to have complete control of our educational system.")

The mainstream media has done a much better job describing Common Core, but slip-ups, such as the New York Times piece, are still disturbingly common. For instance, the LA Times the standards are a “federal curriculum”; PBS that Common Core is responsible for all the standardized tests students take in school; The Atlantic it as “standard national curricula.” As far as I can tell none of these pieces have been corrected for the record.

When the media can’t consistently keep the facts straight, no wonder the public remains so misinformed about the Common Core.

The Fact-Check: Why John Merrow’s Misleading DC Rant Ignores Facts, Research and the Full Truth

Matt Barnum — Sat, 01 Jan 2000 00:00:00 +0000

John Merrow, the retired PBS education reporter, does not like Michelle Rhee, Kaya Henderson, or Washington, D.C.’s education reforms.

Rhee was the controversial D.C. schools chancellor from 2007–2010, and Henderson is her successor, having led the district for the last five years. Both Rhee and Henderson implemented policies involving tougher teacher evaluations, expansion of charter schools, and test-based accountability.

Merrow was last seen engaged in a public feud with Success Academy founder Eva Moskowitz over his reporting in a PBS on the New York City charter school’s disciplinary practices.

In a recent directed at D.C., he argues, “The approach to ‘education reform’ begun by Michelle Rhee in 2007 and continuing under Kaya Henderson to this day is a failure and a fraud. Washington’s students and teachers deserve better.”

With such strong language Merrow must have some pretty compelling evidence.

As it turns out, not really.

Merrow misuses test data (Part I)

Merrow acknowledges that D.C.’s scores on the National Assessment of Educational Progress (NAEP) “have improved faster than any other urban district’s” but they are “at best, mixed” because such gains were “likely the result of more affluent families moving into Washington and enrolling their children in public schools.”¹

It’s difficult, even impossible, to know for sure, but two recent examinations of this question suggest that D.C.’s test score gains were not largely a result of changing student demographics. One comprehensive study that “overall, even when accounting for changes in student demographics, test scores in the District have improved substantially, especially in math.”

A more at the data reported that only a fraction of the NAEP gains were likely explained by demographic changes: “NAEP results have increased not just across race, but across all subgroups such as gender, disability status, and ELL status.”

More importantly, though, reading the NAEP tea leaves to try to determine the efficacy of certain policies is entirely misguided, referred to by some as “misNAEPery” — in which sweeping declarations are made without being able to tie the test scores back to specific policies.

To be clear: Merrow is not the only one to do this. Many supporters of D.C. reforms have in this unfortunate practice. But that is no justification for making unsupported claims about D.C. NAEP scores, as Merrow has done.

Merrow misuses test data (Part II)

Merrow next points to D.C.’s proficiency rates from the recently released PARCC standardized tests: “Barely 10 percent of District students who took the PARCC geometry test, and only 25 percent of those taking the English test, achieved ‘college and career ready’ status.”

To put it bluntly: using raw proficiency rates is entirely inappropriate for judging the impact of specific education policies. We simply don’t know what the scores would have been in the absence of Henderson’s reforms.

Like most places that have administered new Common Core-aligned tests, D.C. saw proficiency rates — not because students are learning less, but because the bar for proficiency was made . Since the test is new, there’s no baseline for comparison.

Once again, it must be said that Merrow is not alone in abusing data in this way. Some supporters of school reform regularly use raw proficiency data to definitively judge schools or policies. Anyone who does so is wrong. So is Merrow.

Research says “test and punish” improved student achievement

Merrow broadens his critique: “Nationally, many in education people [sic] are waking up to the failures of ‘test and punish,’ and the new [K-12 federal education law signed by President Obama Thursday] pulls back on testing.” He presumably is referring in part to the sanctions levied under the former law, No Child Left Behind, to schools that scored poorly on state exams.

Merrow is certainly entitled to this view, but the empirical research on No Child Left Behind shows that it improved student achievement in math. As Tom Dee, a Stanford professor who has studied the law, told me a while back, “The available research evidence suggests it led to meaningful — but not transformational — changes in school performance.”

The law also produced many unintended consequences, such as teaching to the testing, which may be why Merrow has deemed it a failure. But he ought to acknowledge the real academic gains that NCLB produced as well.

The cheating conundrum

Merrow refers to the evidence that cheating occurred in D.C. under Rhee, which he has reported on extensively. This is an important topic. But Merrow offers no discussion of whether there is reason to believe that such cheating is ongoing or whether test security policies have improved in the D.C. Public Schools.

A National Research Council report that “from what we could determine … the alleged [cheating] violations were likely not widespread enough to have affected citywide scoring levels” and that “new test security measures that have been introduced in D.C. schools.”

I would welcome a thoughtful discussion of this issue, but the notion that several instances of cheating years ago invalidate the entire D.C. reform project is absurd on its face.

Rigorous research on D.C. reforms paints a nuanced picture

Although Merrow doesn’t cite any of it, there is a good deal of careful research on D.C. that can contribute to a debate on the pros and cons of the policies pursued by Rhee and Henderson.

I would start with the city’s teacher evaluation program, an aggressive system that has led to performance bonuses and firings. on the early implementation of the policy shows it pushed many teachers to improve and led less-effective educators to leave the classroom. On the other hand, there is that the evaluation system unfairly penalizes teachers with the hardest-to-serve kids. High-poverty schools in DC experience teacher turnover (around 40 percent in the 2010–11 school year), which, although not necessarily bad, does on average student achievement.

Another found that the city’s policy of replacing principals deemed ineffective boosted test scores.

On school choice, D.C. charters appear to be relatively at improving student achievement.

D.C.’s aggressive school closures temporary declines in achievement for students directly affected, though they rebounded fairly quickly. There is no evidence on whether students who would have attended closed schools benefited, though in New York City they did.

However, the number of African-American teachers in D.C. has in recent years, potentially because of closure and teacher evaluation policies. that students of color benefit from teacher diversity.

The National Research Council raised important concerns about how D.C. collects, disseminates, and uses data. It’s also worth noting that enrollment in D.C. Public Schools has in recent years, perhaps a sign that families are more confident in the quality of that schools, though we can’t assume cause and effect.

All of this is to say that there is a serious, nuanced discussion to be had on the impact of D.C. education policies. But John Merrow’s tirade is not it.

(Disclosures: I worked as a summer intern at StudentsFirst, an organization founded by Michelle Rhee, a few years ago. The Seventy Four’s Editor-in-Chief Campbell Brown sits on Success Academy’s board of directors.)

Footnotes:

1. As his source, Merrow says, “Earlier this year pointed out that most of the academic gain was likely the result of more affluent families moving into Washington and enrolling their children in public schools.” I took a look at this report and I’m not sure what Merrow is referring to. The study concludes: “The percentage of all students scoring proficient or above in reading and math on the DC CAS increased between 2007 and 2014. The increase is larger for math than it is for reading. The positive trends are also apparent on NAEP.” However the report warns, “Although we can document some of the changes that occurred over the past 7 years, we cannot determine the independent effects of [DC reform efforts] on achievement and attainment. Changes in the demographic composition of D.C.’s public school students, the growth of the charter sector, differences in the programmatic choices made in DCPS and the individual charter schools, and many other changes that have occurred are intertwined with the changes brought by [DC education policies] …. The signs of improvement are positive, but a more complete picture of student outcomes is needed.“ In other words, as far as I can tell, the National Research Council suggests that demographic changes may be an important factor, but certainly does not definitively claim that they led to the observed test score gains. (Return to story)

Fact-Checking Clinton on Charter Students: She Misses the Big Picture — but Also Has a Point

Matt Barnum — Sat, 01 Jan 2000 00:00:00 +0000

The Fact-Check is The Seventy Four’s ongoing series that examines the ways in which journalists, politicians and leaders misuse or misrepresent education data and research. See our complete Fact-Check archive.

Last weekend Hillary Clinton took a shot at charter schools heard round the education world. In an with journalist Roland Martin, she said:

“Most charter schools — I don’t want to say everyone — but most charter schools, they don’t take the hardest-to-teach kids, or, if they do, they don’t keep them. And so the public schools are often in a no-win situation, because they do, thankfully, take everybody, and then they don’t get the resources or the help and support that they need to be able to take care of every child’s education.” (Read our exclusive interview with Roland Martin about Clinton’s charter comments)

Clinton’s claim, about charters not taking “the hardest-to-teach kids” is a common one. But does the data bear it out?

In a word, kinda. It depends on how “hardest-to-teach kids” is defined and whether charters are compared to all traditional public schools or just those that are in the same neighborhood as charters.

Clinton clearly overgeneralized, ignoring the key fact that charters serve large numbers of low-income and low-achieving students; there’s also little evidence that charters harm traditional public schools. Still, she’s raised a genuine issue that charter advocates should take seriously.

Here’s what the data say about her argument:

Broad generalizations about charter schools nationally are very difficult.

It’s important to start with this point. After all, charters operate under different laws in different states, and individual schools have diverse missions, catering to different students. Some charter schools teach, for instance, students at risk of dropping out, while others may be as a method of white flight from more integrated traditional public schools. Compounding this problem, is the limited research we have on different states. Most rigorous evidence comes from just a handful of higher-profile districts. That said, we’ll do the best with what we’ve got.

Compared to traditional public schools as a whole, charter schools serve more students of color, more low-income students, fewer special education students, and about the same number of English-language learners.

In a simple head-to-head comparison, national data finds charters as a whole are much more likely to serve students of color, particularly black students, than all traditional public schools. (This data is from the 2011–12 school year.)

Again based on national data, charters serve slightly more students who qualify for free or reduced-price lunch, a common, though , proxy for low-income status. This may understate the disadvantage of charter students, because about 18 percent of charters don’t participate in the federal lunch program, compared to just 3 percent of traditional public schools.

National data show charter schools serving a slightly lower number of students who are classified as special education and virtually the same number of English-language learners. Keep in mind that schools make decisions regarding the classification of students in each of these categories, so differences between sectors may reflect those subjective choices.

Compared to neighboring traditional public schools, charters serve less disadvantaged students by some measures but not others.

A more appropriate comparison may be between charter schools and district schools in the same area, since charters are not located evenly across the country. Here we see that charters serve about the same number of low-income students, a higher number of black students, and a lower number of special education students and English-language learners. There is also some that in certain areas, district schools are more likely to serve very low-income students, the ones who qualify for free lunch, not just reduced priced.

This supports Clinton’s claim.

Another way of looking at this question is not through socio-economics but prior achievement levels. One of the most sophisticated of this question examined whether students moving from traditional public schools to charters had higher academic achievement than students who stayed put. The study looked at several states and cities including Ohio, Texas, Chicago and Philadelphia. “Students transferring to charter schools had slightly higher test scores in two of seven locations, while, in the other five locations, the scores of the transferring students were identical to or lower than those of their (traditional public school) peers,” the report concludes.

Based on prior achievement, there is much less backing for the idea that charters take the easiest-to-educate kids.

There is limited empirical evidence that charter schools are systematically pushing out hard-to-serve kids — but only a handful of studies exist.

Arguably, Clinton implied that some charter schools push out difficult-to-educate students. The most, recent high-profile example of this comes from the New York Times’ on Success Academy’s “got-to-go” list, and similar exist with many other charters.

(Disclosure: The Seventy Four’s editor-in-chief Campbell Brown serves on the board of Success Academy. She was not involved in the editing of this article.)

Empirical studies on this question are limited, but recent research has found no evidence that charter schools as a sector systematically push out disadvantaged students — at least not any more so than traditional public schools. That doesn’t mean that illegal push out never happens — obviously it does — just that research to date has not found consistent, sector-wide evidence of it.

Research from Marcus Winters, a professor at the University of Colorado–Colorado Springs and senior fellow at the pro-charter Manhattan Institute, on and show that in both cities’ there was a gap in the number of special education students and between charters and district. The primary driver of this is that such students are less likely to apply to charters in the first place — not because those students are more likely to leave charters.¹

In fact, in some cases these students are less likely to leave charters than traditional public schools. Winters that low-performing students were no more likely to leave charters than district schools. This evidence squares with a separate from an anonymous urban school district.

In sum, this research further supports the notion that neighboring district schools on average are serving more special education students and English-language learners, but not that charters as a whole are pushing out those students. However, the evidence here is limited to just a handful of cities and in some cases just a handful of grades within those cities.

Charter schools’ backfilling policies may exacerbate this issue.

There is significant about whether charter schools should be required to “backfill” — that is replace the students who leave with new ones. Traditional public schools generally take all students in an enrollment zone even if they arrive in the middle of the school year. Some charter schools follow this policy, but many others don’t.

Since disadvantaged students often change schools frequently, charters that don’t backfill may have an easier-to-educate population of students, particularly over multiple years. In contrast, district schools may have a revolving door of mobile, at-risk students. A of 19 KIPP charter middle schools from several different states showed the network’s student population becoming more advantaged over the course of multiple grades, in part because, unlike neighboring district schools, KIPP replaced fewer students who left. Notably though, the researchers found that this was unlikely to explain the large achievement gains of KIPP students.

Charter schools generally have no effect or a small positive effect on the test scores of nearby traditional public schools.

There is evidence that traditional public schools serve a more disadvantaged student population than neighboring charters, at least as measured by special education status. Because of this, it’s reasonable to expect that the district schools would suffer as charter schools enter their neighborhoods.

In fact, however, a large body of research suggests that charters generally have no effect or even a small positive effect on the achievement gains of nearby district schools.

It’s not clear why, but it rebuts the notion that charters are harming traditional public schools by leaving them with the hardest-to-serve students. Clinton described district schools as in a “no-win situation,” but in some cases the situation may be more like win–win.

Beyond the charter vs. traditional public schools wars

In her original comments, Clinton was speaking off the cuff and her brief remarks certainly contained a degree of truth.

When asked for comment, the Clinton campaign reiterated a previous statement from spokesperson Jesse Ferguson: “As Hillary Clinton has said, she wants to be sure that public charter schools, like traditional public schools, serve all students and do not discriminate against students with disabilities or behavioral challenges. She wants to be sure that public charter schools are open to all students.”

Yet the evidence is much more nuanced than Clinton let on. By some comparisons charters serve a more disadvantaged student population than district schools, though by others, they don’t.

Some districts have common enrollment systems, designed to ensure a level playing field between charters and district schools. Others have encouraged to ensure that the most disadvantaged — low-income students, students with disabilities, English-language learners, and students who are migrant, homeless, or delinquent — have a better chance of getting into charter schools. It remains to be seen how effective such systems are.

Charter schools leaders also say they lack the necessary resources and economies of scale to provide services to severely disabled students

Discussing how to address these issues might be more fruitful than rhetoric, on both sides, that pits charters against traditional public schools.

Footnotes:

1. Some of the special education gap also results from the fact that charter schools are less likely to classify students as requiring special education services, and more likely to declassify those already labeled as such. (Return to story)

Fact-Check: Hey Education Pundits, Stop Writing Early Obituaries for School Improvement Grants

Matt Barnum — Sat, 01 Jan 2000 00:00:00 +0000

Conventional wisdom among journalists and education wonks is that School Improvement Grants (SIGs) simply did not work.

The program, which was part of the federal economic stimulus and designed to help turn around struggling schools, has had many none-too-flattering post-mortems written about it. The comes from Politico, which declares that the $7 billion initiative “didn’t help America’s worst schools.” This is one in a long line of such proclamations.

SIG’s skeptic-in-chief is Andy Smarick, a partner at Bellwether Education Partners. (Smarick has written for the Seventy Four and Bellwether’s co-founder Andy Rotherham sits on The Seventy Four’s board of directors.) Smarick has waged a relentless one-man crusade against the program, calling it “,” “,” a “,” a “,” and .”

His skepticism appears widely held. In 2013 when the U.S. Department of Education released test results from schools receiving a grant, news coverage called the results “” and the minority of schools that had seen achievement fall.

With such dire and wide-spread proclamations of failure, the evidence against SIG must be overwhelming, right?

Not so much.

In fact, two of the program have shown positive results — which unfortunately rarely make it into news stories.

So how did it become so commonplace to assume that SIG did not work? Mostly because people are misusing the data. Take this graph, showing only small proficiency rate increases for SIG recipients, which is part of the supposedly damning evidence that many rely on.

Photo: Courtesy the

But Stanford professor Tom Dee says using data in this way is just wrong. “I don’t think anyone who takes policy and program evaluation seriously would consider that type of descriptive evidence meaningful in terms of understanding the true impact of the program,” he said in an interview.

Or as Shanker Institute senior research fellow Matthew DiCarlo in a blog post on the misuse of SIG evidence, “This breaks almost every rule of testing data interpretation and policy analysis.”

The reason is that this sort of eye-ball test cannot isolate the impact of SIGs. We simply don’t know what scores would have looked like in the absence of the program.

That’s why rigorous analyses that focus exclusively on the impact of SIGs are necessary. When Dee ran one such careful of SIG schools in California — comparing schools that were eligible for a grant to similarly low-performing ones that weren’t — he found positive results overall. And despite headlines emphasizing the costs of program — $7 billion!!! — Dee showed that SIG was relatively cost effective compared to class-size reductions.

Another of SIG recipients in Massachusetts by the American Institute of Research found statistically significant achievement gains in all three years of the program across both math and English. The effect sizes were fairly large — on the lower end of of the impacts of New Orleans’ charter school reforms. (On the other hand, the research found that SIG had no impact on student attendance.)

Photo: Courtesy the

None of this is to say that SIG has clearly been a success. We don’t have much evidence from states other than Massachusetts and California, and it’s certainly possible — even likely — that the program was successful in some places and less successful in others, as the recent Politico story argued. Indeed, that implementation of the program has been difficult. More rigorous research is in the , so time will tell.

For now, though, confident declarations are getting far ahead of existing evidence.

It’s also plausible that the large amount of money dedicated to SIG could have been used more efficiently for other purposes; it’s difficult to know.

To some extent, the misinterpretation of SIG evidence is the Department of Education being foisted by its own overheated rhetoric. In announcing the program in 2009, Secretary of Education Arne Duncan , “We want transformation, not tinkering.” Although the gains in the Massachusetts and California studies are real and important, it would be hard to call them transformational.

But maybe a program that — at least in some places — resulted in steady, meaningful progress is worth celebrating, not condemning.

Fact-Check: Mainstream Media Picks Up Anti-Charter Report, But Fails to Disclose Union Funding

Eric Owens — Sat, 01 Jan 2000 00:00:00 +0000

Updated October 22

This week, the Center for Media and Democracy, a reliably pro-union advocacy organization based in Madison, Wis. that is funded in part by union donations, issued a report on America's charter schools.

In a nutshell, that the American public "does not have ready access to key information about how their federal and state taxes are being spent to fuel the charter school industry." There's "a horrifying lack of accountability." There's "fraud, waste, and abuse in many states" because of too much "flexibility." Also, some charter schools have failed.

Several big-name newspapers dutifully printed the report’s findings and arguments for local readers, but none of them seemingly scrutinized the center’s bolder assertions, which later required corrections, and not one acknowledged the group's union funders — misrepresenting the report as an unbiased set of facts about America's charter schools.

"Report: Charter school budgets are a 'black hole' of public information," proclaimed. "Report: $1.7M paid to Mich. charters that never opened," . “$2 million of Indiana charter school funds squandered,” decisively declared .

The Center for Media and Democracy — a small, five-person operation with a research director — claims that the charter school report is the result of a "year-long investigation" into charter funding.

Coincidentally, the pro-union group's "year-long investigation" corresponds almost to the day with an Oct. 24, 2014 disbursement of $30,000 from the American Federation of Teachers (for "member related services").

The $30,000 contribution isn't the only endowment the Center for Media and Democracy has received from a union. A cursory search of its contributions over the last five years shows a stream of generous union-linked money.

Since 2010, the organization has received $237,730 from national unions. Cash suppliers include the Service Employees International Union (SEIU), the American Federation of State, County and Municipal Employees (AFSCME) and the AFL-CIO. There has also been an unspecified grant from the Open Society Institute, a funding network established by progressive billionaire George Soros.

Beyond these issues of funding bias, the group's report is also beset with a host of factual problems.

"The report shows a total misunderstanding of how federal charter schools run," says Gina Mahoney, senior vice president for government relations at the National Alliance for Public Charter Schools.

While some charter schools across America have closed their doors for various reasons including financial improprieties and poor academic performance, Mahoney said, many of the charter schools which the Center for Media and Democracy identifies as closed simply have not closed.

Mahoney says the union-funded group based its closure findings on a sloppy and incorrect data set. In fact, some of the schools declared closed in the report have actually expanded while altering their names. Other schools now have new names because they are the result of charter school mergers. Still others reorganized and were folded into local school districts.

The Center for Media and Democracy has also already issued a correction to their earlier media advisory about charter school spending because it falsely claimed that with charter school supporters.

"The original advisory suggested that Governor Ducey was meeting secretly with charter school advocates. The governor’s office refutes this and we are changing our report to reflect this information," the pro-union advocacy group said in a follow-up statement. "We regret the error."

More notable, perhaps, than the correction is what was left out of the report.

Mahoney says the report avoids the fact that charter schools are now routinely surpassing public schools in academic performance — often for around the same per-student cost as public schools.

And the real success story in charter schools is not in wealthy suburbs where comfortably middle-class kids and rich kids attend public schools with abundant resources, Mahoney stressed, but in urban areas where poor kids live and where public schools have been failing students and families for decades.

"You are beginning to see the academic outcomes the teachers unions have feared. Kids attending charter schools are learning much more, as measured by the same math and reading tests kids take in traditional public schools," Mahoney said.

"Poverty is not a final destination for these kids. Charter schools have shown that kids from impoverished backgrounds cannot be written off."

Mahoney cited data from Stanford University’s Center for Research on Education Outcomes, a leading independent analyst of charter school effectiveness. — which is absent from the Center for Media and Democracy report — shows that charter schools are facilitating real academic improvement for poor kids in urban areas.

"This research shows that many urban charter schools are providing superior academic learning for their students, in many cases quite dramatically better," Stanford's Margaret Raymond said in a statement. "These findings offer important examples of school organization and operation that can serve as models to other schools, including both public charter schools and traditional public schools."

Across 41 regions of the United States, "more than twice as many urban regions show their charter schools outpacing their district school counterparts."

In math, urban charter schools in 63 percent of the 41 regions currently outperform their public school counterparts. In reading, urban charters in 56 percent of the 41 regions outpace public schools in the same areas.

"The data shows that students are better served in charter schools, hands down," Mahoney said.

Will the Salt Lake Tribune, Detroit News and various other newspapers around the country now follow up the Center for Media and Democracy’s report with a wider analysis of the available research and statistics? Or at the very least inform their readers about the funders and agendas behind a charged report that so many rushed to cover without context?

What if No Child Left Behind Worked and Nobody Realized it? Blame the Media.

Matt Barnum — Sat, 01 Jan 2000 00:00:00 +0000

If the public knows one thing about No Child Left Behind (NCLB), it’s that it probably didn’t work. It’s a widely held belief; the problem is it’s not true.

“The public perception,” says Stanford professor Tom Dee who has the law, “seems to be that No Child Left Behind has failed, but the available research evidence suggests it led to meaningful — but not transformational — changes in school performance.”

Unfortunately, the media has largely not addressed the crucial question of NCLB’s impact on students, favoring instead traditional he-said, she-said style reporting or offering crude statistical analysis of its legacy.

Take a recent Washington Post news on NCLB. Reporter Lyndsey Layton explains the law’s impact: “[NCLB’s] goals were later seen as unrealistic, and the law had unintended consequences: Many schools squeezed out art, science and other subjects to focus on math and reading; cheating scandals erupted; and some states lowered standards so their students would appear more proficient.”

There were in fact unintended consequences of the law, but the Post story doesn’t get around to how NCLB did when it came to intended consequence, namely, improving student learning. No research is cited; no researchers are quoted.

When I this on Twitter, Layton pointed to the National Assessment of Educational Progress (NAEP), a widely regarded, low-stakes national assessment, saying student progress has been “jagged” and “incremental.” From this, she implied that NCLB hadn’t improved academic achievement.

Education researchers Morgan Polikoff, a USC professor, and Steve Glazerman, a senior fellow at the research group Mathematica, quickly in to explain that it’s inappropriate to use data that way.

In an interview, Polikoff said that simply looking at trends in test score data does not allow for a counterfactual — meaning there’s no way of knowing what the scores would have looked like in the absence of NCLB. That’s why sophisticated analyses are necessary to isolate NCLB’s impact.

Dee also said that NCLB can’t be judged by looking at overall NAEP scores , in part because some states already had NCLB-style accountability before the federal law required it, while others didn’t.

Similarly Matt Di Carlo, a senior research fellow for the Albert Shanker Institute, has about why using simplistic test score data is misguided for judging individual policies.

Instead, researchers must carefully construct control and treatment groups to determine a policy’s impact. In the case of NCLB that means, for example, comparing Catholic schools (not subject to the law) to public schools; states with accountability policies pre-NCLB to those that didn’t; schools just below a cutoff for NCLB sanctions to those just above the cutoff.

Polikoff and Dee agree that tends to find small but real gains in student achievement, particularly in math. Dee says the law produced “meaningful, important, and cost-effective improvements.”

On Twitter, Layton expressed that anyone would think that NCLB had led to improved student achievement.

She and the Washington Post are not alone in ignoring the evidence. When the Senate passed a revision to NCLB in July among , , , , , and , not a single outlet cited research about its impacts or spoke to an academic expert.

Instead they all covered the conflict. Teachers unions said this; civil rights groups said that. Republicans want less federal involvement; Democrats want more.

There were a couple exceptions: The Washington Post’s Wonkblog cited and Stanford professor Sean Reardon. had a solid write-up. NPR put together a decent of the research literature in 2014.

This is not a new tendency for our press corps, where false balance and conflict often trump substance. Matt Yglesias recently in Vox about how the media is paying more attention to Hillary Clinton’s email debacle than Jeb Bush’s tax plan.

In education the same problem exists: the boxing match is covered, the research is not. It’s charters vs. traditional public schools, reformers vs. unions, opt-out-ers vs. accountability hawks. Empirical evidence rarely sees the light of day.

Polikoff recently sent around a new, particularly rigorous research , which found positive outcomes from NCLB, to a group of education journalists. Not one of them covered it, he says.

“Part of the problem is it’s not a sexy thing,” Polikoff explains.

He’s right: Getting into the weeds of different ways to evaluate NCLB — ya’ know, interrupted time series, regression discontinuities — makes readers’ eyes glaze over. But that is part of a reporter’s job: to render complex subject matter understandable so the public can make informed decisions

After all, taking research, and all its complexity, seriously is necessary if we have any hope of expanding effective education policies.

Resisting School Integration: A Brooklyn Case Study in Scared Parents and Bad Data

Matt Barnum — Sat, 01 Jan 2000 00:00:00 +0000

Controversy is brewing in Brooklyn, where a city plan to rezone local elementary schools may accidentally lead to school integration — and parents aren’t happy.

They say it’s due to fear of sending their kids to a worse-performing school, but school-performance data tell a different story, suggesting that the real hurdles to integration include a misinterpretation of testing data as well as deep-seated prejudice.

The controversy centers around two public schools: P.S. 8 and P.S. 307. A Sept. 23 New York Times shares some parents’ concerns: “At a meeting at P.S. 8 on Monday, Dumbo residents pointed to P.S. 307’s low test scores and asked what kind of training and extra resources the school’s teachers would receive to make the education there comparable to that at P.S. 8.”

Parents in P.S. 307, meanwhile, are also skeptical of the plan. “We fought hard to build this school, and we’re not just going to let people come from outside when we worked so hard and dedicated ourselves,” one parent explains.

The Times then points to the differences in school proficiency rates: “P.S. 307’s population is 90 percent black and Hispanic, and 90 percent of the students’ families receive some form of public assistance. Its state test scores, while below the citywide averages, are closer to average for black and Hispanic students, with 20 percent of its students passing the math tests and 12 percent passing the reading tests this past year. At P.S. 8, whose population is 59 percent white, with only 15 percent receiving assistance, scores are considerably above the city averages. Almost two-thirds of its students passed each test.” A Wall Street Journal highlights the same stats.

Without quite saying so, we’re led to the obvious conclusion: P.S. 307 is a failing school while P.S. 8 is a good one.

But this is a deeply misleading picture. Although these scores accurately show disturbing and well-documented achievement gaps in student performance they are incredibly poor measures of school quality, the key reason being that many factors other than schools — such as poverty, racism, and home environments — affect student proficiency rates.

By examining New York City’s — which look at a variety of factors including test scores, parent and teacher surveys, and school inspections — there doesn’t appear to be a huge difference between the quality of P.S. 307 and 8. These performance reports are not the be all and end all, but they are much better indicators of school quality than raw proficiency rates.

School Quality Guide summary, P.S. 8 (Source: )

School Quality Guide summary, P.S. 307 (Source: )

But wouldn’t you know it, parents don’t seem to care about school performance being properly measured. (This squares with in Los Angeles.) The Times story reports that one parent calls P.S. 307 “severely underperforming.” Another acknowledges that P.S. 307 is “kind of on the upswing, all things considered,” but asks, “is it at the level of a P.S. 8 yet? It’s not really clear.”

That’s probably because, as Halley Potter, a fellow with the pro-integration Century Foundation, said in an interview, “Families use demographics of schools as a proxy for school quality.” This means that parents assume any school with a lot poor kids or kids of color is a bad one.

Potter says that this assumption is about deep-seated personal prejudices, yes, but it’s also about structural inequity. It is true that in many places schools with lots of poor students get the , experience significant , and .

Though according to state reports teachers at and have comparable qualifications, experience levels, and teacher attrition rates.

Clearly, it’s inaccurate to assume that the only good schools are in wealthy neighborhoods. Such a presumption leads parents to believe that attending a school like P.S. 307 will harm their child — that’s probably not the case, though, as the performance data show.

In fact, if the school becomes more integrated, all students win, Potter argues. Not only is there integration leads to academic gains for low-income students, but middle class students may also , particularly in terms of tolerance and social skills.

One would hope that getting parents more accurate and accessible data about school quality will help them realize as much. After all, it is not surprising that parents pay little attention to data that is complex and opaque, often baffling to even experienced education writers.

But there’s also a more disturbing, but equally plausible explanation for the resistance to integration: Some parents opposed don’t really care about school performance. They may say that their problem with rezoning is based on low test scores at P.S. 307, but in fact, it is possible that no amount of test score data would convince them that sending their kid to P.S. 307 is a good idea.

What they may really oppose, then, is sending their kids to school alongside a large number of poor kids of color. And based on that attitude, it’s hardly surprising that families in P.S. 307 are not welcoming integration either.

Fact-Check: Are Washington’s Charters Harming the State’s Traditional Public Schools?

Matt Barnum — Sat, 01 Jan 2000 00:00:00 +0000

Charter schools versus traditional district schools — this is the legal fight now playing out in Washington state, after the state Supreme Court ruled charter schools unconstitutional. (See The Seventy Four’s complete coverage of the Washington court decision, and the fallout affecting parents and educators.)

In the wake of the verdict, the president of the state teachers union cheered the decision, , “charter schools steal money from our existing classrooms.” And similar funding arguments are now fueling charter showdowns in other cities across the country — including a to change the charter funding formula in Baltimore.

But is it true that charter schools harm traditional public schools? Or, as some charter advocates claim, does the introduction of charters spur competition and innovation in neighborhood schools, leading to improvement across-the-board?

Based on evidence from districts across the country and interviews with several researchers, the conclusion defies a simple us vs. them characterization: Charter schools are unlikely to have significant negative effects on student achievement in traditional public schools — and may, in fact, have small positive effects on nearby schools. At the same time, there is research indicating that charters may in fact harm school district finances.

Marcus Winters, an assistant professor at the University of Colorado – Colorado Springs, has studied charters extensively. In an interview with The Seventy Four, he explained, “My are really consistent with the literature overall, which is that as competition from charter schools — but also from — increases, we see small improvements in student performance in traditional public schools.” Emphasis on small, Winters said, and not in line with expectations from some advocates that competition¹ would be a “” for the system as a whole.

Similarly, research in , , , , and shows small benefits with the introduction of charters on nearby traditional public schools, as measured by student test scores.

On the other hand, Ron Zimmer, an associate professor at Vanderbilt University, found basically no effect, positive or negative, on achievement in two across several states. A of — including authored by Winters — have shown similar results.

The overall research is “mixed,” Zimmer said, but “the vast majority [of studies] don’t show negative effects.”

A small of studies have produced negative results, though, by Scott Imberman, an economist at Michigan State University. Imberman used a different approach than most other research, which may better isolate the impact of charter schools on neighborhood schools. He finds small to moderate decline in student test scores, but a positive impact on student discipline².

In an interview, Imberman acknowledged that his study — which looks at a single school district — was an “outlier,” saying that most other research had effects closer to zero. “My study throws a bit of caution to the general finding,” he said.

On the other hand, a September 2014 of North Carolina suggests that past research may have actually underestimated the positive effects of charter competition. That study looked at the impact of charter schools in grades that overlapped with traditional public schools as well as in non-overlapping grades. In the case of overlapping grades, there were significant positive results on student test scores, which may have been downplayed in past research.

The effects of charter competition almost surely varies from place to place, perhaps because of size and quality differences in the sector along with differences in underlying issues, such as how they are funded. However, research has, by and large, not looked at which set of charter characteristics are most likely to create competitive benefits.³

There has generally been a dearth of research on the long-run effects of charter competition. Most studies look at impacts over the course of one or two years based exclusively on English and math test scores. (Though some, like Imberman’s study and this New York City , look at outcomes beyond test scores; others, like this Texas , examine many years of data.)

It is certainly possible that the longer-term implications of charter growth may be different than the short-run ones. This is particularly true since there is charter expansion can have negative impacts on school districts’ finances, which raises legitimate concerns regarding how the two types of public schools coexist.

That said, as Zimmer points out, it cannot be assumed that harmful effects on district finances will translate to harmful effects on students. In fact, it is theoretically possible that it is this very threat to districts that will spur the competitive effects suggested by charter supporters.

Taken together, the research tells a story that’s far more nuanced that what you’ll hear in the us vs. them debates now rippling across the country: We can’t assume that charter schools have either negative or positive effects on traditional public schools, and generally whatever impacts exist will be small, as measured by test scores. Many studies find either positive or zero impacts, while a small number have shown negative effects. Questions about what methods researchers have used exist for all studies, and the long-run impacts of charter competition remain unclear. (Finally, effects will likely vary from place to place)

But ultimately, the argument that charter school expansion will inevitably harm the students who remain in traditional public schools is simply not supported by the evidence.

Footnotes:

1. It is important to note that the effects – positive or negative — that charter schools have on traditional public schools would not exclusively come through “competition.” For example, changes in student population may drive peer effects; funding changes as a result of charters may impact schools; traditional public schools may take innovation from certain charters (and vice versa); charters and district schools may even collaborate to try to improve performance across both sectors. For brevity, though, this piece refers to “competitive effects” as inclusive of all impacts that charters have on traditional public schools. (Return to story)

2. However, it’s unclear whether that was a result of genuinely improved student behavior or changes in how discipline incidents were reported. (Return to story)

3. One exception is a working that studied Washington, D.C. and found potential positive effects of competition — but only on schools where the nearby charter school was of high quality. This may also explain why much of the research comes to differing conclusions. (Return to story)

Fact-Check: Just How Many Tenured Teachers Are Fired Each Year Anyway? (Hint: Not Many)

Matt Barnum — Sat, 01 Jan 2000 00:00:00 +0000

“Tenure,” American Federation of Teachers (AFT) President Randi Weingarten has , “is not a job for life. It’s ensuring fairness and due process before someone can be fired, plain and simple.”

National Education Association President Lily Eskelsen Garcia when someone says “it’s nearly impossible to fire a bad teacher.”

Echoing these arguments is Century Foundation senior fellow Rick Kahlenberg who recently in an article for the “American Educator” — a magazine published by the AFT — that “2.1 percent of American public school teachers, including tenured teachers, were fired for cause.” This is based on from the National Center for Education Statistics (NCES), which conducts surveys of school districts across the country.

Kahlenberg concludes that perhaps teacher dismissals in a few high-profile districts could be streamlined, but that overall the data suggest Garcia and Weingarten are right, and that a reasonable number of tenured teachers are fired every year.

This data — that the average district fires about three tenured teachers per year for poor performance — has long been to argue against pushing for more teacher dismissals.

But are these numbers accurate?

More recent , also from NCES, suggest that in districts across the country, it is rare for a tenured teacher to be formally dismissed due to poor performance. In fact, the two data sets find wildly divergent numbers of tenured teachers fired for cause.

According to 2007–08 data, roughly one in 40 tenured teachers is fired for poor performance. But the more recent data suggests that it’s more like one in 500 tenured teachers.¹

What can explain such a massive disparity in the rate of dismissal for tenured teachers?

I put the question to NCES staff member Chelsea Owen and she explained that the survey questions had undergone revision between years. How the question was asked in 2007–08 seemed to produce irregularities, she said in an email.

Specifically some “respondents reported numbers of dismissed tenured and non-tenured teachers which summed (added up) to more than the number of teachers in the district.” There were data checks for this Owen said, but the question was revised for the 2011–12 data to get results that could be better checked for irregularities.

Owen added, “While both the 2007-08 and 2011-12 surveys collect the counts of tenured and non-tenured teachers who were dismissed due to poor performance, it is likely that a change in estimates observed over time may be (at least partially) due to the changes in how the question was asked.“

What this means is that any reference to the 2007–08 data is outdated and potentially inaccurate. The more recent data suggests that it is relatively rare for a tenured teacher to be formally dismissed for poor performance. And this number might be even lower if it weren’t for districts such as Washington, D.C., which has teachers deemed low-performing and thus has a high rate of teacher dismissals in the most recent data. (About one in twenty Washington, D.C. teachers were dismissed for poor performance in the 2010–11 school year.)

To be clear, these numbers do not answer the fundamental question of whether more teachers should be dismissed. They also don’t include ineffective teachers who of their own volition or who are by administrators without going through the formal dismissal process. And they do not include tenured teachers dismissed for reasons unrelated to performance.

Finally, such stats can’t show whether or not the processes for dismissing tenured teachers across the country are overly cumbersome.

Some economists , using data simulations, that dismissing about 5 percent of the teaching force — approximately 180,00 teachers — would improve student achievement. But others have argued that this would be and even .

These debates will surely continue. But in the meantime, we should make sure to have them with accurate and up-to-date data, which find that most districts in the country dismiss very few tenured teacher for poor performance.

Footnotes:

1. These are rough estimates since neither years of data show the precise total number of tenured versus non-tenured teachers. I got these numbers by finding the number of teachers who have tenure nationally ( according to 2011–2012 ), which is about 55 percent. That means that in 2011–12, the average district had about 103 tenured teachers, of whom .2 were dismissed for poor performance. In 2007–08, if the proportion of tenured teachers is same, that means 116 teachers per district had tenure, of whom 3 were dismissed for poor performance on average. (back to story)

A NYT Contributor Attacks the ‘Myth’ of New Orleans’ School Progress After Katrina. She’s Wrong

Matt Barnum — Sat, 01 Jan 2000 00:00:00 +0000

Here we go again.

The misinformation, a-contextual data on New Orleans’ school reform will not stop. The latest edition comes from Andrea Gabor, that “it is more important than ever to accurately assess the results, the costs and the continuing challenges” of the New Orleans reforms.

That’s reasonable enough, and Gabor makes a few important points. In particular, she is right to urge caution to those trying to replicate New Orleans’ virtually all-charter model in other cities. But much of the piece — far too much — is a stream of ill-supported arguments designed to discredit the significant progress made in New Orleans schools.

Strap in, as we correct the record:

If not progress, then what?

One of the chief problems with commentaries that seek to downplay the success of New Orleans’ school reforms is a refusal to focus on measurable progress. Gabor’s piece is a case study in precisely that. She briefly acknowledges test score gains — based on Tulane economist Doug Harris’ rigorous — before quickly downplaying them, pointing to the city’s low ACT scores and the alarming number of young adults in the New Orleans metro area who are neither in school nor employed (“disconnected youth”).

But these data tell us absolutely nothing about the success or failure of school reform in New Orleans — it’s possible that these numbers would be even worse in the absence of the reforms. Gabor makes no attempt to consider the data over time.

In fact, the average ACT scores in the city have gone up significantly — although without more sophisticated analyses it would be inappropriate to credit the reforms for these gains. The city also to other urban districts that require all students to take the test. (Gabor does not mention that Louisiana all high school juniors to take the ACT, even those who do not intend to go to college.)

On the question of disconnected youth, there is no data available showing whether the problem has gotten worse or has improved in the New Orleans metro area since Hurricane Katrina. I confirmed as much with with Kristen Lewis, of Measures of America, which put out the that Gabor cites. In an interview Monday, Lewis also emphasized the fact that New Orleans should be viewed in the context of the entire state — Louisiana is last in the country among all fifty states, and Baton Rouge actually has a slightly higher number of disconnected youths than New Orleans, particularly among black residents. Again, though, any inferences about school quality based on these numbers are entirely unsupported.

To say the city still has problems or that the schools could improve further is a truism. But as an indictment of the New Orleans reform, such claims fall short. And if the progress that New Orleans has made — Harris says he has “never seen an effect of this size before” — is not good enough, then what would be?

No evidence that data is skewed

Gabor suggests the data showing school success in not to be trusted, in part because of the disconnected youth numbers. She claims that students “often disappear from school entirely and, thus, are no longer included in the data.”

But when looking at cohort graduation rates — which measure the percentage of students starting ninth grade who graduate, and is thus unlikely to be affected by the number of disconnected youth — New Orleans has made . Graduations rates have increased from 54 percent in the 2003-04 school year to 73 percent in 2013-14.

Moreover, insofar as the disconnected youth create an upward bias, this would have also occurred in earlier data as well. Without evidence showing increases in disconnected youth — evidence which Gabor fails to cite, and which doesn’t appear to exist — attempts to debunk New Orleans’ progress are simply conjecture.

Teacher diversity: reformers agree it’s a problem

Gabor writes: “A key part of the New Orleans narrative is that firing the unionized, mostly black teachers after Katrina cleared the way for young, idealistic (mostly white) educators who are willing to work 12- to 14-hour days.”

It is that after Katrina, the number of teachers in New Orleans who are black dropped precipitously, but the teaching force remains half black. Moreover, contrary to Gabor’s suggestion, supporters of reform are not celebrating the decline in black teachers, but trying to fix it. For example, a recent from the pro-charter group New Schools for New Orleans acknowledges a “major teacher diversity problem.” To be fair, talk is cheap, but the data suggest that although newly hired teachers remain disproportionately white, more black teachers have been hired in recent years.

School takeovers – problem or solution?

Gabor says: “Consider Joseph S. Clark Preparatory High School, one of the city’s last traditional public schools to be ‘taken over.’ Most of its 366 students declined to re-enroll when it reopened under new management in the fall of 2011. During its first year under FirstLine, a charter management organization, Clark had only 117 ‘persisters,’ or returning students, according to a study by Stanford University’s Center for Research on Education Outcomes, known as Credo.”

Here, Gabor simply assumes her premise — that the disruption inherent in closing a community school — is a bad thing. But when looked at schools in New Orleans taken over by charters, they found large gains in achievement. That does not preclude the possibility that some students were made worse off by closing of traditional public schools — research on this question in other parts of the — but Gabor shares precisely zero evidence to make this case.

The reforms failed and they only worked because of the money

Gabor says that “one of the most important post-Katrina reforms” was “a big increase in both government and philanthropic funding.” She is quite right on this front, but it’s a bizarre point for Gabor to underscore. After all, she has just gotten done explaining how the system has been a failure to a large extent, but now we’re told that the infusion of money was a big success. How does Gabor know this? No evidence on this point is offered.

Least advantaged students suffer?

To close her piece Gabor writes, “Privatization may improve outcomes for some students, but it has hurt the most disadvantaged pupils.” Previously she claims, “There is also growing evidence that the reforms have come at the expense of the city’s most disadvantaged children.”

But Gabor’s “growing evidence” is fabricated. Harris broad gains in achievement across New Orleans, and the writ large gains for the largely low-income, largely black students are a huge victory for anyone who cares about educational equity. There is simply no empirical evidence on offer suggesting that the New Orleans reforms harmed disadvantaged students in terms of achievement or any other educational outcome. None.

Gabor does cite showing that disadvantaged students are less likely to enroll in the highest performing schools. This is a real problem and squares with from other cities, but it falls far short of showing that disadvantaged students are actually worse off. Paradoxically some of the same research finds that disadvantaged students who do enroll in charters benefit the most. If Gabor’s concern is that too few of these disadvantaged students have access to charters, well, she should be praising their massive expansion in New Orleans. (For a more extensive discussion on this question including research from other parts of the country, see .)

You say miracle, everyone else says progress

The heart of Gabor’s piece is this: “The New Orleans miracle is not all it seems.” Yet purveyors of the city’s reform that no miracles have occurred — just real progress.

Perhaps this framing explains why Gabor thinks the New Orleans experiment has been such a disaster. If she is looking for miracles, any reform effort will inevitably come up short.

Photo by Getty Images: A student attends a dance class at the Encore Academy charter school in May in New Orleans.

Separating Fact from Fiction at the Education Summit: What Research Confirms (and Disproves) About the Rhetoric

Matt Barnum — Sat, 01 Jan 2000 00:00:00 +0000

The Fact-Check is The Seventy Four’s ongoing series that examines the ways in which journalists, politicians and leaders misuse or misinterpret education data and research. See our complete Fact-Check archive, and be sure to check out all our coverage of the 2015 New Hampshire Education Summit.

Several themes emerged last week, as six Republicass convened to talk K-12 education at the 2015 New Hampshire Education Summit. (Watch all the highlights.) Three themes of note were shared by several candidates, including opposition to teachers unions, spending more money on classrooms, and federal involvement in education. But what does the research say? Let’s dive in:

1. Claim: The federal government stifles innovation.

Republicans were generally consistent in saying that when it comes to education, the federal government should get out of the way and empower states, districts, and schools. Wisconsin Governor Scott Walker said, “The federal government is the last place in the world I want holding state and local school districts accountable.” Carly Fiorina claimed, “What doesn’t work are big bureaucratic programs from Washington, D.C. What doesn’t work are people spending money on mandated programs, either at the state or federal level.”

These comments are driven in part by philosophical views about the role of government, but from an empirical perspective, there is little to no evidence that federal involvement has harmed student achievement. To the contrary, research strongly suggests that federal intervention through No Child Left Behind increased student learning in math (though did lead to some unintended consequences). And even the federal school turnaround program had some .

Several candidates also complained about the federal government’s role in pushing the Common Core. While it’s true that the feds incentivized the adoption of the standards, it is flat-out wrong to say that they are “federally mandated” — after all, several states have not adopted the Common Core. Fiorina also raised concerns about the federal government’s role in driving the over testing that some parents and teachers have complained about. She’s on firmer ground here, as it is clear that the Department of Education has contributed to a large increase in standardized tests in many states.

2. Claim: Teachers unions are holding back public education.

Almost all the Republican candidates shared a dislike for teachers unions. Chris Christie, for example, suggested that part of the solution to improving schools is “the guts to challenge [the] unions.”

Does the empirical evidence support this? The short answer is not really, but the research is mixed. Though one can make the case that unions have modest negative effects on student outcomes, the large-scale harm that many of the candidates suggested are unlikely.

Determining unions’ causal impact on students is extremely difficult for a variety of methodological reasons, but some have tried. An oft-cited from Harvard economist Caroline Hoxby found negative effects of teachers unions on student outcomes, but there have been significant raised about it. A recent found that strong collective bargaining for teachers had harmful long-term effects on students.

However, a review of by Robert Carini suggested modest positive effects of unionization, whereas a more recent analysis found the “mixed, but suggestive of insignificant or modestly negative union effects” on student outcomes.

In sum, the research is ongoing, but at this point there is certainly no clear evidence that unions have had large negative (or positive) effects on schools.

3. Claim: Throwing more money at public education is not the answer.

Many of the candidates suggested that there is little relationship between spending more money and getting better results. Carly Fiorina claimed, “We know that how much money you spend doesn’t always have much to do with the outcome… California spends more per pupil, K–12, than 49 out of 50 states, and the results are 49 out of 50.” Jeb Bush said, “It’s not about money. We spend more per student than any other country in the world other than Belgium, Luxembourg, and one other rounding error.”

It is true that the United States on K–12 education than most other industrialized countries, but U.S. spending as a fraction of gross domestic product is relatively average. (Contrary to Fiorina’s claim, most suggest that California is in the bottom of the pack nationally in spending per pupil.)

Moreover, the evidence that money has no relationship to results is weak. The basis for this conclusion is likely research from Stanford economist Eric Hanushek, who has that money doesn’t matter. Yet his results have faced , and more strongly suggests that additional resources do in fact improve student outcomes.

More food for thought: Even reforms favored by many of the candidates — such as New Orleans’ virtually all-charter school district — necessitated a of resources.

Fact-Checking All Those Charter Critics Who Snatched Defeat from Jaws of New Orleans’ Victory

Matt Barnum — Sat, 01 Jan 2000 00:00:00 +0000

No matter who you talk to, the sweeping reconfiguration of the New Orleans schools is a model for something.

To proponents, it’s a successful model of reform that should be expanded to other struggling public school districts.

To critics, it’s a top-down model of exactly what not to do when facing the daunting challenge of a long-troubled school district decimated by a natural disaster.

In the aftermath of Hurricane Katrina, New Orleans dramatically expanded the number of charter schools in the city, fired existing staff, and held schools accountable for student test scores, closing those that didn’t hit certain targets. Today, schools in New Orleans are independently operated, publicly funded charter schools.

Kindergartners smile on their first day of school at Dr. Martin Luther King Jr. Charter School for Science and Technology in the Lower 9th Ward August 20, 2007 in New Orleans, Louisiana. (Photo by Getty Images)

The best study of what those ambitious changes brought to New Orleans schools, students and their families 10 years after Katrina — much-anticipated by Tulane University economist Doug Harris — was published this week in Education Next.

Harris found that the combination of reforms produced large gains on almost every measurable outcome: from test scores to graduation rates, so much so that he declared in an interview with ��Ӱ��: “I have never seen an effect of this size before.”

But such bold statements won’t mollify the New Orleans skeptics. Here’s a rundown of the criticism leveled at New Orleans’ reforms and some facts debunking each one:

It’s apples and oranges

Examining the effects of New Orleans’ reforms is challenging since the pre- and post-Katrina student populations changed as a result of the hurricane. Harris addresses this in two ways: First by looking only at the growth of students who returned to New Orleans after Katrina and second by comparing all New Orleans students to students in other districts who were affected by Hurricane Katrina, which hit between Aug. 23 and Aug. 31, 2005, or Hurricane Rita less than a month later.

Neither method is perfect, but both are serious efforts to ensure apples-to-apples comparisons. The first method finds small improvements in student learning in test scores in math, English, science, and social studies. The second approach produces much larger estimates, finding that the city, which initially was massively behind the comparison group, caught up in just five years.

NOTES: Separate analyses show that New Orleans students returning after the storm, who could be studied only through 2009, also made gains relative to their own prior performance, but these differences were often not statistically significant. Scale scores are averaged across grades 3 through 8 and across English language arts, math, science and social studies. Scale scores are standardized so that zero refers to the statewide average. (Source: Education Next)

Combining both methods, Harris still finds significant positive results of at least eight percentage points for the average student. The upper end of Harris’ estimate is a massive 15 percentage point increase per student.

Who cares about test scores?

Research consistently finds that once high-stakes — such as the prospect of a school being closed or a teacher receiving a poor evaluation — are attached to tests, , and other forms of gaming significantly increase.

As John Thompson in the Hechinger Report, “If [Harris] was studying a school system that was not totally focused on raising test scores during an era of data-driven competition, there would have been nothing wrong with Harris’s conclusion that New Orleans reforms increased student performance, as measured by test scores, eight to 13 percentile points, which compares favorably with expanding early education. Since , however, districts across the nation have reported miraculous increases on bubble-in tests, but test score growth on the reliable NAEP (National Assessment of Educational Progress) test has mostly slowed.”

Since New Orleans has used tests as a measuring stick of success for schools, it’s appropriate to be concerned that scores may be rising due to factors other than improved learning. If this were the case, one would expect that scores on high-stakes reading and math exams to have gone up much more than lower-stakes science and social studies tests. One would also predict that ACT scores would not have improved significantly since the college entrance exam is and externally administered, making cheating harder. Yet Harris found the same gains on both science and social studies tests, and that ACT scores by two points (on a 36-point test) in New Orleans since 2005.

Some may still argue that using any form of assessment to measure school quality is inappropriate because of long-standing skepticism of the value of standardized tests. But even ignoring test score growth as a metric of success doesn’t help reform skeptics: Harris finds big jumps in both rates of high school graduation and college entry of 10 and 14 percentage points, respectively.

Nothing to see here

Duke Professor Helen Ladd that gains have been made but pointed out that the progress was “in the performance of students at a basic level — and it’s a very basic level.” A Politico highlights that the average ACT score in New Orleans’ Recovery School District of 16.4 is “considerably below the minimum score [of 20] required for admission to a four-year public college in Louisiana.”

Similarly an in ThinkProgress claims “New Orleans’ all-charter school system is struggling” pointing to data that finds “ improvement” in aggregate ACT scores.

Yet an average two point increase on a 36-point exam is actually statistically speaking. In an interview with The Seventy Four, Harris says the gains he observes on state tests are also massive: “Put differently, I’ve never seen an effect of this size before.”

The argument that the schools still have room to improve is one wholeheartedly to by the supporters of the New Orleans reforms. But it does nothing to diminish the real growth in the schools based on multiple measures of success.

It’s all about the money

Ladd, the Duke professor, also that the main reason for the improvement was the large increases in per pupil expenditures, not the systemic reforms.

Ladd is right to say that money matters and that funding increases likely played a role in New Orleans’ improved school performance. After the storm, schools initially received large increases in resources. Harris estimates that “from 2004‒05 to 2011‒12, the same years covered by our achievement analysis, total public schooling expenditures per student increased by $1,000 in New Orleans relative to other districts in the state.” The average per pupil expenditure for Louisiana districts outside New Orleans is about $10,500.

Harris agrees that the resources were a contributing factor, but says, “There’s not much reason to think that it was all about the money.” He points out that many other urban school districts have seen serious infusions of funding without comparable gains.

They could have done it differently

Related to the more money argument is the more-money-and-different reforms argument. Teachers’ union President Randi Weingarten rhetorically, “Is there a way to actually have devoted the same kind of funding, the same kind of thought, the same kind of talent to do democratic, publicly accountable public schools?”

Harris rejects the suggestion that traditional investments would have produced the same results after looking at improvements made in schools that went that route. “We compared the effects that we generated to things like early childhood education and small class sizes and there just isn’t evidence generally to support the idea that that would generate effects of this size,” he says.

Less discipline, not more

Jennifer Berkshire (aka ), writing in , points to the “strict disciplinary codes that are now the norm at many of the city’s charter schools.” A in the Atlantic says that in New Orleans “alarming numbers have begun to prompt challenges to and reassessments of charters’ no-excuses regimens.”

The harms of aggressive disciplinary practices are , and no doubt there are many schools in New Orleans that could improve on this front. Overall, though, Harris’ research finds that student expulsions and suspensions have decreased for all students since the reforms were instituted. Certainly there is continued improvement that should be made — and there is evidence that New Orleans charter schools are and in addressing these problems to some extent — but the trend is moving in the right direction.

Segregating New Orleans

Writer and school reform critic Jeff Bryant that charter schools have an “unquestionable link to increased segregation of students based on race and income.” In fact, the research is mixed on this front, but Bryant is certainly right that in some charters may have exacerbated segregation. Did this happen in New Orleans?

Nope, finds Harris, who reports that segregation across a number of demographics has remained virtually unchanged since the reforms. That’s not to say that segregation is no longer an issue in New Orleans — — but there’s no evidence that school choice made the problem worse.

Against popular opinion

The central thesis of Berkshire’s Salon , which features many New Orleans residents, parents, and activists is that the New Orleans community hasn’t bought into the city’s school reforms. Berkshire also suggests that a large number of people who were initially supportive of the turnaround reversed their position upon seeing its implementation.

She on Twitter that it’s been hard for her to find anyone who doesn’t criticize New Orleans’ reforms privately. That may be true, but the people who will speak on the record to a well-known charter schools critic may not be representative of the community as a whole.

A scientific of New Orleans residents finds that far more believe that the schools have improved since the storm than believe they’ve gotten worse; 59 percent agree that charter schools have improved education in the city; and 73 percent supported an open enrollment school choice system over a traditional geographic school assignment model.

The polling data is not all good news for proponents of the New Orleans model — white residents are more supportive than African-Americans, which is a major red flag — but there is no evidence that more people are skeptical of the system now compared to past surveys.

Another compared views of public school parents across several different cities — Baltimore, Cleveland, Denver, Detroit, Indianapolis, New Orleans, Philadelphia, and Washington, D.C. — and found that 56 percent of NOLA parents believe the schools there are improving. This was significantly better than most other cities, though slightly behind the reform-minded districts of Denver and Washington, D.C. Again, there is an important nuance here: just 42 percent of New Orleans parents expressed trust and confidence in the schools. But the overall picture is mixed to positive — not the one of widespread rebellion against the reforms Berkshire attempts to paint.

The schools haven’t cured cancer

Berkshire also says that New Orleans residents still face many obstacles: “The challenge for architects and advocates of the reform effort here is that, expanded even slightly beyond these narrow metrics [test scores, graduation rates and ACT scores], the case that life is improving for the children of New Orleans gets much harder to make. stands at 39%, a figure that’s unchanged since Katrina, even though the city is now home to tens of thousands fewer children. Inequality is the in the country, on par with Zambia. And remains a persistent plague here.”

To education writer and activist, Chris Stewart, this can be called the “Well, it doesn’t cure cancer” argument against education reform. To expect successful school reform to cause immediate drops in child poverty or violent crime is laughable on its face — a bright red herring that if applied consistently would doom virtually every school reform effort ever tried.

—-

This is not an exhaustive list of the critiques of New Orleans school reform, but it addresses some of the main concerns that can be answered with the existing evidence.

None of this is to say that there aren’t valid questions about the success of New Orleans school reform experiment. For example, school choice advocate Howard Fuller has spoken eloquently about the process through which the New Orleans reforms were implemented on the community rather than with it.

Harris and have noted the potential difficulties in trying to implement New Orleans reforms in totally different cities and contexts. The large-scale firing of African-American teachers left a in the community and its schools.

The case is not, as , closed. Policymakers and community members need to continue to look at what’s working and what can be improved; researchers need to continue studying the district, using multiple measures of success, particularly long-run outcomes.

But let’s retire the arguments against New Orleans that, to date, aren’t based in fact.

The Fact-Check: No Vox, All the Good Schools Are NOT in Rich Neighborhoods

Matt Barnum — Sat, 01 Jan 2000 00:00:00 +0000

Are all the good schools in rich neighborhoods? Yes, says Matt Yglesias at , citing a new graph released by :

Source: D.C. Office of Revenue Analysis

The simple scatterplot shows a clear correlation between the cost of homes in D.C.’s neighborhoods and the test scores of nearby elementary schools. (For the record, this is not remotely surprising; we know that for a variety of reasons wealth is highly associated with student test scores.)

But Yglesias’ analytical reach seems to exceed his grasp when he goes on to say that this graph proves that virtually all of D.C.’s bad schools are found in cheaper neighborhoods — and vice versa. That’s just wrong.

If you’re setting out to measure the quality of a school, there are far better stats than proficiency test scores to look at. In fact, a much better indicator of school quality is “median growth percentile.” Yes, it’s a mouthful, but it measures something really important: how much progress students at a certain school made relative to other students in the district who started at the same level.

The basic principle behind median growth percentile is this: Students come into their schools at vastly different performance levels, which are often totally outside schools’ control. What schools do have significant control over is how much progress their students make over the course of a year.

While median growth percentile is still an imperfect stat (more on that in a moment), it does give us a much better idea of how schools are serving their students.

I re-ran the graph that Yglesias analyzed, this time using publicly available stats about reading growth and math growth in the same schools.¹

Source: D.C. Office of the State Superintendent of Education and the D.C. Office of Revenue Analysis

As we can see, there is still a general correlation between housing price and student growth, but particularly for math scores, these graphs paint a much different picture of D.C.’s schools.

You will now find many schools in poorer neighborhoods demonstrating strong student growth – and in some cases outscoring wealthy neighbors. Take Mann Elementary School, where 90% of students were proficient in reading in 2014 and where housing prices are among the highest in the region; yet the school was actually a bit below average in math growth scores and about average in reading growth.

Or there’s Plummer Elementary where just a third of students were proficient in reading in 2014; yet the school ranked as 9th best in the city in reading growth scores. Different stats, much different story. Dare I say more accurate story.

Measuring school quality is extremely tricky because doing so requires isolating the school’s influence on student test scores when, in fact, most of the variation in scores can be traced back to. Using measures of growth instead is a good start, but it as out-of-school factors also influence how much progress students achieve in a given year.² (That’s why most attempts to evaluate teachers on student test scores — usually known as — measure growth while also controlling for student demographic factors such as poverty and special education status.)

Yglesias is quite right to point out the shameful fact that low-income students are often zoned into the and taught by the . He’s also right to support D.C.’s charter school sector. But the between school performance and student performance continues to lead many policymakers and journalists astray.

In fact, it also leads many parents astray. Early in his piece, Yglesias bemoans the fact the school where he lives seems to be an outlier. “Garrison, where the reading scores are terrible and the houses are expensive anyway is my neighborhood public school,” he writes.

But I crunched the numbers, Matt, and Garrison is hardly terrible. If you had used better metrics, you would have discovered it’s just about average.

Footnotes:

1. All median growth percentile numbers came from. The scores are based on D.C.’s standardized test, the CAS, using 2014 data. The home price data comes from the original put together by D.C.’s Office of Revenue Analysis. (Back to story)

2. Another common concern about growth scores (and value-added measures) is that they tend to bounce around a lot from year to year. (Back to story)

The Fact-Check – ����Ӱ��

����Ӱ�� Fact-Check: Are Teachers Really Burning Out Because of Tougher Tests and Evaluations?

����Ӱ�� Fact-Check: All the Ways Donald Trump and Ted Cruz Flubbed on Education at Thursday’s Debate

Opinion: Don’t Humiliate Teachers… But Fire the Worst: Does David Denby Realize He’s Agreeing With Reformers?

The Fact-Check: Not Every State Takeover of a Troubled School District Deserves an F

New York Times Gives Credence to Conservatives’ Common Core Delusion

The Fact-Check: Why John Merrow’s Misleading DC Rant Ignores Facts, Research and the Full Truth

Fact-Checking Clinton on Charter Students: She Misses the Big Picture — but Also Has a Point

Fact-Check: Hey Education Pundits, Stop Writing Early Obituaries for School Improvement Grants

Fact-Check: Mainstream Media Picks Up Anti-Charter Report, But Fails to Disclose Union Funding

What if No Child Left Behind Worked and Nobody Realized it? Blame the Media.

Resisting School Integration: A Brooklyn Case Study in Scared Parents and Bad Data

Fact-Check: Are Washington’s Charters Harming the State’s Traditional Public Schools?

Fact-Check: Just How Many Tenured Teachers Are Fired Each Year Anyway? (Hint: Not Many)

A NYT Contributor Attacks the ‘Myth’ of New Orleans’ School Progress After Katrina. She’s Wrong

Separating Fact from Fiction at the Education Summit: What Research Confirms (and Disproves) About the Rhetoric

Fact-Checking All Those Charter Critics Who Snatched Defeat from Jaws of New Orleans’ Victory

The Fact-Check: No Vox, All the Good Schools Are NOT in Rich Neighborhoods

The Fact-Check – ��Ӱ��

��Ӱ�� Fact-Check: Are Teachers Really Burning Out Because of Tougher Tests and Evaluations?

��Ӱ�� Fact-Check: All the Ways Donald Trump and Ted Cruz Flubbed on Education at Thursday’s Debate