;)
On Wednesday, the latest results of the National Assessment of Educational Progress (NAEP) 鈥� the gold standard, low-stakes test given to a representative cross section of 4th and 8th graders across the country 鈥� will be released. Virtually every news outlets, including The Seventy Four, will cover the numbers, which makes sense since the scores are important and have some limited value for policymakers. But if history is any guide, for one day the educational world will collectively lose its mind over NAEP, engaging in what Mathematica researcher Steve Glazerman 鈥渕颈蝉狈础贰笔别谤测鈥�.
Here are four reasons why NAEP results should be interpreted very cautiously:
1
Raw NAEP data can tell us NOTHING about which education policies are effective and which aren鈥檛.
Raw NAEP data can tell us NOTHING about which education policies are effective and which aren鈥檛.
When the 2013 NAEP scores were released, Secretary of Education Arne Duncan pointed to the relatively large student 鈥済ains鈥� in Washington, D.C. NAEP scores, , 鈥淟eaders in D.C. have shown tremendous courage and taken bold steps that are resulting in strong growth.鈥� Commentators across the country, including the editorial boards of the , , and , picked up on this to suggest that D.C.鈥檚 NAEP gains (as well as Tennessee’s) were showing that school reform was working.
Unfortunately, such claims are entirely unsupported. it is inappropriate to use raw NAEP scores to judge the success or failure of specific policies.
The basic reason is a wonky, boring but crucially important one: NAEP scores, on their own, offer no comparison (or 鈥渃ontrol鈥�) group by which to judge specific policies or even packages of policies. Remember eighth-grade science class? To make causal inferences, there must be both a treatment group and a control group.
As an example, let鈥檚 say Wednesday brings good news in the form of higher NAEP scores. Reformers will claim their policies are working 鈥� but how do we know? Maybe scores would have been even higher if a different set of policies were pursued. Maybe scores went up for reasons entirely unrelated to reform policies. We simply can鈥檛 say.
2
Lots of things besides schools and education policies affect NAEP scores.
Lots of things besides schools and education policies affect NAEP scores.
Student achievement is based on everything that has happened in a student鈥檚 life before taking the test.
We tend to think of schools as driving test scores because students take tests and formally learn academic content in schools. Indeed, schools have an extremely important impact on student learning, but out-of-school factors have an even on student test scores. This is yet another reason we can鈥檛 use NAEP to judge school policies. The many out-of-school factors driving achievement 鈥� the economy, access to healthcare, etc. 鈥� mean we can鈥檛 even be sure that changes in NAEP scores had anything to do with changes in schools.
3
Changes in NAEP scores are not actually 鈥済rowth.鈥�
Changes in NAEP scores are not actually 鈥済rowth.鈥�
In the coverage of NAEP scores, we will almost surely hear about some state whose students 鈥渟howed the most growth.鈥� For example, in 2013, the Washington Post that 鈥渢he District [of Columbia]鈥檚 fourth- and eighth-graders made significant gains on national math and reading tests this year, posting increases that were among the city鈥檚 largest in the history of the exam.鈥� This is , because the fourth- graders who took the test in 2013 are not the same fourth-graders who took the last NAEP years earlier. In other words, all we can say is that one group of students has a higher average score than a completely different group of students from a couple years ago.
This may seem like an academic point, but it raises yet another problem with trying to make inferences about policy based on NAEP: demographic changes among students tested to changes in average test scores. What look like 鈥榞ains鈥� may just be differences in which students were tested.
4
Most people will use NAEP data to reiterate what they already believe 鈥� no matter what the data say.
Most people will use NAEP data to reiterate what they already believe 鈥� no matter what the data say.
I can guarantee that the NAEP results 鈥� regardless of what the actual data are 鈥� will be used by commentators to reinforce their previously held policies positions. That people will use the same data to reach opposite conclusion is an indication that we shouldn鈥檛 read too much into said data.
Advocates will surely declare 鈥淸State X, which had 鈥榞ood鈥� results] did [Policy Y, which I already like]; therefore everyone should do [Policy Y].鈥� If scores show improvement reformers will say, 鈥淭his shows our policies are working 鈥� full speed ahead!鈥� If there aren鈥檛 improvement reformers will say, 鈥淭his shows why our schools are in desperate need of reform 鈥� full speed ahead!鈥�
Similarly reform skeptics will gleefully point to disappointing results as evidence that reform policies are failing. But if scores rise, they will that NAEP scores shouldn鈥檛 be taken seriously and that tests don鈥檛 much matter.
People believe what they believe; NAEP scores won鈥檛 鈥� and frankly shouldn鈥檛 鈥� change this. But can we just drop the charade?
This is not to say that NAEP scores are useless. They are genuinely important indicators about whether students across the country are learning more math and reading than past students. And although raw data cannot be used to judge specific policies or policymakers, it is absolutely reasonable to make hypotheses about policy that can then be tested rigorously.
In turn, NAEP scores have been used by researchers with careful, statistically rigorous designs to test the efficacy of certain policies. (For example, much of the research on No Child Left Behind uses NAEP data, but does so by creating controls and applying careful statistical analyses.) The key words here are statistically rigorous 鈥� an eyeball test does not count.
So, yes, although some rumors that they鈥檒l be lower, I hope NAEP scores go up on Wednesday. It will be nice to see and a hopeful sign for education reform and our country. But no, I won鈥檛 be using raw NAEP scores to judge the success of policies or politicians or to support the things I already believe 鈥� however tempting it might be.
