The University of Arkansas today released what it calls a "first ever" study exploring the relationship between charter school funding and student achievement. Here at NPR Ed we get a lot of press releases for studies related to education — teacher turnover, financial aid access, social and emotional learning in preschool and more. But not all studies are created equal. It's important to understand not only what the study says but who the researchers are and how they arrived at their conclusions.
For today's study, researchers relied heavily on one standardized test, the NAEP (aka the "Nation's Report Card"). They took NAEP scores in reading and math from 28 states, then broke them down by schools' funding per student. The report found, as other research has shown, that student performance at charter schools is roughly on par with public school performance.
But, the researchers argue, because charter schools tend to have smaller budgets (according to previously published research from this same University of Arkansas department), "these differences amount to charter schools overall being 40 percent more cost-effective in math and 41 percent more cost effective in reading, compared to traditional public schools."
Patrick J. Wolf is the study's lead author and a professor of education policy at the University of Arkansas. "The headline of this report is that the charter school sector in states across the country is more productive in generating desirable student outcomes at a lower cost than the traditional public schools," Wolf said.
That is, indeed, the headline. But the math behind it — and the conclusions Wolf and his team draw from it — may not be that simple.
Ted Kolderie is a senior associate with Education Evolving, an education policy nonprofit. Some call Kolderie the "godfather" of the charter school movement because of his work dating back to the late 1980s in Minnesota. He says he takes the findings of this new report with a grain of salt.
"This is the kind of quote-unquote 'study' we've been seeing for years that falls into the category of 'advocacy research,' " he says. "Pretty soon you'll have another study showing just the opposite."
Kolderie takes issue with the report's dependence on NAEP scores alone to make determinations about school effectiveness. The National Assessment Governing Board, which produces NAEP, was not involved in the study.
"This is a simple-minded notion, that performance is how students score on assessments," whether NAEP, the international PISA test, or state accountability tests, Kolderie says. "Is achievement, performance, success, quality, really one-dimensional?"
The finding that charter schools are more cost-effective rests on the University of Arkansas group's earlier research claiming that charter schools have less money to spend per student than traditional public schools. According to a review by the National Education Policy Center, that report failed to take into account that charters sometimes depend on districts to help pay for school lunches, transportation, special education and other services, and that they often serve a less needy population.
Wolf defends the design of the new report, saying "test scores are an important metric of effective education" and that, currently, NAEP scores are the only way of measuring student performance across state lines. He insists that the study provides important evidence that "the money invested in charter schools is more productive than the money invested in traditional public schools."
The Department Of Education Reform
Both the NAEP study and the earlier budget study came from the Department of Education Reform at the University of Arkansas. The department's work often focuses on return on investment in public schools and promotes alternatives to the traditional public system. It was established in 2005 with help from outside donors, including the Walton Family Foundation (which is also a donor to NPR).
The late philanthropist John Walton, son of Wal-Mart founder Sam Walton, was an outspoken champion of charter schools and gave generously to the charter and voucher movements.
Patrick Wolf, the author of the new study, says today's findings aren't meant to be used as an argument to reduce funding to traditional schools. Still, he said: "Public policy in education can't ignore cost. Money is scarce, so it's a service to policymakers for them to know which education sectors are most productive."
Joe Nathan is the director of the Center for School Change in Minneapolis. Nathan helped write the nation's first charter school law in Minnesota in 1991. Like Kolderie, he too is wary of studies like this one: "I have been very dubious about research that has tried to compare charter schools and district schools," he says. "I think the charter idea is a brilliant idea, but we need to handle it responsibly. ... Trying to make sweeping statements about charter schools or district schools does not advance the overall cause of improving American public education."
The ultimate goal, Nathan says, should be to help all young people to be successful.
ROBERT SIEGEL, HOST:
From NPR News, this is ALL THINGS CONSIDERED. I'm Robert Siegel.
AUDIE CORNISH, HOST:
And I'm Audie Cornish. The University of Arkansas today released what it calls a first ever study comparing charter schools to traditional public schools. The study's headline - charter schools keep pace with public schools in terms of student achievement and they do it with a lot less money. Claudio Sanchez with the NPR Ed team reports that both the study's methodology and its conclusions are controversial.
CLAUDIO SANCHEZ, BYLINE: The study focused on eighth-graders in 28 states and is based on two sets of data. First, the amount of money charter schools receive per student compared to public schools. Researchers then compared how kids in both kinds of schools did in reading and math based on one big test - the federal NAEP test. The study's conclusion, according to the lead author, Patrick Wolf.
PATRICK WOLF: The charter school sectors in states across the country are more productive in generating desirable student outcomes at a lower cost than the traditional public schools.
SANCHEZ: On NAEP, charter student scores are roughly on par with those of traditional public school kids. Wolf, however, argues that charters are 40 percent more productive because they get less money - about $3,800 less per student on average.
WOLF: Charter schools have to be very lean. We know that they tend to employ fewer administrators, fewer support personnel. They tend to channel more of their scarce resources directly into the classroom. And so that may be the specific source of their greater productivity.
SANCHEZ: But one NAEP official told NPR the study's correlation between NAEP scores and school funding is not reliable. And Ted Kolderie, who many consider the godfather of the charter school movement, says he's not convinced by the report.
TED KOLDERIE: This is the kind of (quote) "study" we've been seeing for years and years, that falls into the category of advocacy research. Pretty soon you'll have another study showing just the opposite.
SANCHEZ: Another big flaw, says Kolderie, is that the researcher's definition of productivity and effectiveness is based exclusively on standardized test scores.
KOLDERIE: Is achievement, performance, success, quality really one-dimensional?
SANCHEZ: Some critics also say that the $3,800 funding gap is overstated. The National Education Policy Center has argued that charters often depend on districts to pay for big budget items like school lunches and transportation. Charters also don't enroll nearly as many special education students, who are more expensive to teach. Still, Patrick Wolf stands by his study.
WOLF: What our research is suggesting is when funds are scarce in education as they are now, we're getting more bang for our buck with our investments in public charter schools compared to district-run schools.
SANCHEZ: But in the long-running debate between supporters of charters and traditional public schools, this study raises more questions than it answers. Claudio Sanchez, NPR News. Transcript provided by NPR, Copyright NPR.