Key Findings

1. High-quality studies find that well-run charter schools consistently deliver a better public education to students than do typical traditional public schools.
2. The CREDO study did not collect data for a long enough period to provide useful conclusions about student performance in a charter school setting, as compared to the learning environment at a traditional public school.
3. Attendance records show charters do not “skim the best” students away from traditional public schools.
4. In states where charter schools are allowed, they are popular with parents; if the current ban is lifted, charter programs would likely be popular in Washington also.

Introduction

This year people across Washington are debating whether to lift our state’s ban on charter schools. Supporters and opponents of charter schools frequently cite academic studies to support their respective positions. Different studies appear to give contradictory conclusions about the performance of charter schools in other states, leading to confusion in the public discussion over Washington state’s charter school ban.

To help resolve this confusion, this paper provides an annotated bibliography of studies that employ rigorous statistical analyses to evaluate the performance of existing charter public schools. This paper presents the elements of a high-quality academic study, describes the two methods used to assess charter schools, and reviews the methodological flaws of one study, the CREDO study, that is most frequently cited by charter school critics.

As shown in the descriptions below, high-quality studies find that well-run charter schools consistently deliver a better public education to students than do typical traditional public schools. The studies show charter schools do not “cream off” the best students, as critics claim, nor do they negatively affect the achievement of students in traditional public schools.

Most studies find that in the states where charter schools exist, they bring six primary benefits to public education. They tend to:

• Improve student learning
• Often serve low-income communities
• Help close the achievement gap
• Be popular with parents
• Provide parents with education choice
• Encourage officials to raise the quality of traditional public schools

What is a Charter School?

A charter school is a public school that is tuition-free and open to all students. Charter school principals are allowed to control their budgets, teaching staff and educational programs with little or no central bureaucratic control. Charter schools hold administrators and teachers accountable for student performance.

Charter public schools that consistently fail to educate students can be closed or placed under new management. Managers of traditional public schools, by contrast, are not held accountable for raising student achievement. District officials continue to fund traditional public schools, year after year, even if they fail to teach their students adequately.

Today over two million students attend 5,637 charter public schools in 41 states and the District of Columbia, with 521 new charter schools opening in 2011–12. Charter schools are popular with parents, with an estimated 600,000 students on waiting lists, an increase of 30% in two years — enough demand to fill nearly two thousand additional charter public schools.

Washington is one of the few states that do not allow charter public schools. A bipartisan charter school bill (SB 6202) was introduced in the 2011 session of the state legislature. The bill had enough support to pass the Senate Early Learning & K–12 Education Committee, but the chair canceled all committee business for the year rather than bring the bill up for consideration.

After the legislative session ended, supporters launched a ballot measure, Initiative 1240, largely based on the reforms proposed in SB 6202. The initiative qualified and the question of whether to allow a limited number of charter schools, 40 over five years to open in Washington, will be decided by voters in the November 2012 general election.

Methodological Flaws in the CREDO Study

Opponents of charter schools often cite the findings of a study released in 2009 by the Center for Research on Education Outcomes (CREDO) at Stanford University. The CREDO study has also been widely reported in media. The study concluded that 17% of charter schools deliver a superior education for their students, about half provide educations not substantially different from traditional public schools, and 37% of charter public schools deliver results substantially worse than traditional public schools.

Research experts and statisticians have criticized the CREDO study for the weakness of its data and methodology. Though Stanford researchers made sweeping claims about charter school performance across the nation, they only looked at charter schools in 15 selected states and the District of Columbia, considerably less than half of the 41 states that have charter schools. Further, the laws authorizing charter schools and holding their leaders accountable for educational performance vary widely across these 15 states and the District of Columbia, yet the CREDO study made no effort to account for these important differences.

Experts in statistics observe that the CREDO study relies on the learning levels of large numbers of students who were only in their first year attending a charter school, too short a time to measure academic effects. In addition, a large number of the charter schools in the study sample had only been open for one year or less.

Thus, the CREDO study did not collect data for a long enough period to provide useful conclusions about student performance in a charter school setting, as compared to the learning environment at a traditional public school. Also, the CREDO study did not compare the academic scores of charter school students to actual public school students. The performance of charter school students was compared to the scores of hypothetical “virtual” traditional public school students that were invented by the study authors.

Rather than comparing the performance of charter school students to their real-world traditional school peers, CREDO researchers decided to create a “virtual twin” for each student. They then used computer modeling to extrapolate how that charter school student would supposedly have performed if he had attended a traditional public school. The “virtual twin” method has been criticized for being highly subjective and easily manipulated as a point of reference.

Despite its limited sample size and flawed methodology, however, the CREDO study still found that a large majority of charter schools, 67%, performed as well or better in educating their students than if these students’ imaginary counterparts had attended a traditional public school. The study found that only 37% of these select charter schools deliver learning results that are significantly worse than traditional public schools.

Yet this is not the view of charter school parents. Attendance at a charter school is voluntary — all students are there by choice. Parental satisfaction is likely much higher in charter schools than in other public schools. The same is not true of traditional public schools, where administrators often assign students to a particular school, with little or no input from their parents.

Traditional public schools operate as regional monopolies. The only education alternatives available to families are to pay private tuition, devote time and money to homeschooling, or buy a house in a different school district, options that are beyond the reach of many working families. The “opt-in” nature of charter schools indicates parents generally place a higher value on a charter school education than do the authors of the CREDO study.

Characteristics of High-quality Academic Studies of Charter School Performance

To achieve an accurate comparison of student learning at a charter school compared to a traditional public school, most researchers design studies that take account of differences in student motivation, family support, prior academic achievement, socio-economic status and other factors. These important variables are described by the Center on Reinventing Public Education at the University of Washington:

“Students may have weaker skills or more involved parents; they may come from poorer families or be recent immigrants just learning English. Any differences in outcomes could thus be due to differences in the children served, not due to the quality of instruction provided.”

Researchers use two statistical methods to account for student differences when comparing school performance. Neither method was used in the CREDO study. The two methods are:

(1) Randomized lottery studies. These studies compare the subsequent academic achievement of students who win and lose lotteries to attend a charter public school, and take into account differences in family background and student motivation. These studies also control for observable differences among students, including parent income, race and ethnicity, urban and rural settings, grade level, number of years a school has been open, previous test scores and students’ prior educational experiences.

(2) Observational studies. These studies follow individual student results over time to determine the increased learning “value” a charter school may or may not provide, as compared to the educational “value” provided by a traditional public school. Like randomized studies, these studies control for various observable differences among students, including parent income, race and ethnicity, urban and rural settings, grade level, number of years a school has been open, previous test scores and prior educational experiences.

These two types of studies give the most accurate information about charter schools’ real-world effects. They are designed to isolate the performance of the school from the characteristics of the students to create a true “apples-to-apples” comparison between charter schools and traditional public schools.

Following is a guide to significant academic studies based on these two methodologies.

Guide to Charter School Studies

“Informing the Debate, Comparing Boston’s Charter, Pilot and Traditional Schools,” by researchers at Duke University, the National Bureau of Economic Research, Harvard Graduate School of Education, University of Michigan School, the Ford School of Public Policy and The Boston Foundation, January 2009.

Summary: This study uses both randomized lottery and observational from all of Boston’s public schools to show that Commonwealth Charters do not “skim off” the best students. This study shows that half the achievement gap between minority and non-minority students was erased after students had attended a Commonwealth Charter for only a single academic year. Four years of attending a Commonwealth Charter erased the achievement gap between black and white students.

“Student Characteristics and Achievement in 22 KIPP Middle Schools,” by Christina Clark Tuttle, Bing-ru The, Ira Nichols-Barrer, Brian P. Gill and Philip Gleason; Mathematica Policy Research Inc., June 2010.

Summary: This study uses both observational and random lottery data comparing students in 22 Knowledge Is Power Program (KIPP) charter schools to similar students in traditional schools starting in 2005 and continuing through 2014. This series of studies shows there is no evidence that KIPP middle schools “skim off the best” students by systematically enrolling better-performing students than those attending other public schools in the same district.

To the contrary, these studies show KIPP enrolls students whose average fourth-grade achievement is lower than the district average; and that dropout rates are not significantly different than those experienced by other public schools. Yet KIPP public school students make significant advancements in reading and math scores in all four years of middle school, enough to reduce the achievement gap between black and white students by half in three years.

Over 90% of KIPP middle-school students go on to attend college-prep high schools, and over 85% of KIPP public school graduates, including students from low-income and minority families, attend college.

“Charter Schools in Eight States, Effects on Achievement, Attainment, Integration, and Competition,” by Ron Zimmer, Brian Gill, Kevin Booker, Stephanie Lavertu, Tim R. Sass and John Witte; RAND Corporation, Spring 2009.

Summary: This study uses observational student achievement data over six or more years from charter public schools and traditional public schools in eight states. The researchers found that charter schools do not selectively accept, or “cream,” the best students in a given area, leaving traditional public schools with lower-achieving students.

RAND researchers also found that opening a charter school does not drain resources away from other public schools in the area. The researchers found that students who attended a charter middle school and went on to a charter high school were 7 to 15 percentage points more likely to graduate than those who attended a traditional public high school. In addition, students who attended a charter high school were 8 to 10 percentage points more likely to go on to college than graduates of a traditional public high school.

“How New York City’s Charters Affect Achievement,” by Caroline M. Hoxby, Sonali Murarka and Jenny Kang, principal investigators; The New York City Charter Schools Evaluation Project, September 2009.

Summary: This is a study of students who participated in a lottery to attend a charter public school. Local officials randomly approved some of these students to attend a charter public school, while the remainder were assigned to a traditional public school. The researchers then compared the academic levels achieved by the two groups.

The study found that lottery winners scored considerably higher on standardized math and reading tests than charter school applicants who were required to attend a traditional public school. Students attending charter public schools on average scored 3.8 points higher in math and 1.6 points higher in English each year than similar students who attended traditional public schools.

“Public School Academies, 2006–07: Report to the Legislature,” Michigan State Board of Education, 2007.

Summary: Assessment data from the Michigan Department of Education show that learning by economically disadvantaged African-American and Hispanic/Latino charter school students exceeded that of students attending traditional public schools in the same district by four to five percentage points in both math and English language arts.

“Florida Charter Schools: Annual Accountability Report, 2002–2003,” School Choice, Florida Department of Education, 2004.

Summary: This report found that a greater proportion of charter schools made “Adequate Yearly Progress” than traditional public schools. The study compared the learning gains of students attending charter schools and traditional public schools both overall and within various student subgroups. Florida education officials found that on three of four academic tests analyzed, charter school students experienced greater learning gains in core subjects than students attending traditional public schools.

State officials concluded, “Charter schools provide parents with choices for selecting the most effective educational programs for their children.”

“Massachusetts Charter School Achievement Comparison Study: An Analysis of 2001–2005 MCAS Performance,” by Richard Hill and Bryan Gong; Department of Education, State of Massachusetts, 2006.

Summary: This four-year comparative analysis of the Massachusetts Comprehensive Assessment System (MCAS) performance of charter schools and their counterpart sending districts finds that the state’s charter school students are performing as well, and sometimes better than, their peers in traditional public schools.

“Exploring the Correlates of Academic Success in Pennsylvania Charter Schools,” by Christopher Nelson, RAND Corporation; Gary Miron, Western Michigan University; National Center for Study of Privatization in Education, Columbia University; 2005.

Summary: Researchers found that Pennsylvania charter schools that held teachers and administrators to a high degree of accountability produced stronger academic growth in student scores. Similarly, charter schools with a high degree of teacher mission and leadership stability produced strong student academic growth in reading and math.

“California’s Charter Schools: Measuring Their Performance,” by Eric Crane, Brian Edwards and Noli Brazil; EdSource, 2007.

Summary: Researchers found that charter schools managed by experienced private management organizations, such as KIPP, Green Dot, and Aspire, outperformed other charter schools. Classroom-based charters had higher math scores than non-classroom-based charters. Their strongest finding when comparing charter schools to traditional public schools, was that students at charter middle schools showed significantly higher academic performance than students attending traditional public middle schools.

“Findings from the City of Big Shoulders, Younger Students Learn More in Charter Schools,” by Caroline M. Hoxby, Harvard University; and Jonah E. Rockoff, Columbia Business School; Hoover Institution, 2005.

Summary: The authors investigate the effect of charter school attendance on student achievement, using data from Chicago’s admission lotteries to make apples-to-apples comparisons among like student groups. The study compares students who applied to charter schools and were accepted with students who also applied to charter schools but were not accepted. Students who attended charter schools at the elementary level scored higher in both math and reading than similar students attending traditional public schools in the same area.

“Key Issues in Studying Charter Schools and Achievement: A Review and Suggestions for National Guidelines,” National Charter School Research Project, University of Washington.

Summary: This independent study ranks more than 40 evaluations of charter school performance between 2000 and 2005. Evaluations that attempted to compare charter schools nationally or across state lines ranked “fair” or “poor” because of wide differences in state charter school laws and levels of oversight. The study found that in general more rigorous and accurate methods were used in evaluations that examined charter school performance within a particular state, rather than those, like the CREDO study, that attempt comparisons across groups of states.

“Louisiana Believes, 2012 Spring Testing Report, May 23, 2012,” State Superintendent of Instruction John White.

Summary: Test results for 2012 show dramatic improvement in New Orleans schools. After the devastation of Hurricane Katrina, 80% of New Orleans public schools were re-opened as independent charter schools. Since 2007, New Orleans and Recovery School District schools have more than doubled the percentage of students scoring “Basic and Above” on all tests.

Students at New Orleans charter schools represent typical public school students in the city. Attendance records show charters do not “skim the best” students away from traditional public schools. In New Orleans high school, students showed significant improvement in graduate exit exam scores, particularly in English and math.

Conclusion

The idea of allowing public education students to attend charter schools has sparked heated debate in Washington, but in most other states opening a charter school is noncontroversial and routine. The prevalence of charter schools in other states allows researchers to move from theory to practice. We can now measure not just how well charter schools are expected to perform, but how well they educate children in the real world.

Charter schools have been studied extensively for over 20 years by statistical experts, professors and researchers at the best universities and research institutions in the nation. Charter school studies have been produced by researchers at Harvard University, Duke University, Massachusetts Institute of Technology, Columbia University, University of Washington, University of California, National Bureau of Economic Research, RAND Corporation and the Hoover Institute.

In addition, dozens of charter school assessments have been issued by state departments of education. Others studies consist of reports on the performance of charter schools requested by state legislators following passage of a charter school law in their state.

On the whole, this extensive body of research shows charter schools have a strong record of success. Numerous studies cite instances in which charter schools have improved the education of children living in poverty, have narrowed the achievement gap, and have earned the approval of parents and the community. Of the 41 states, plus the District of Columbia, that have authorized charter schools, there is no instance in which a state has repealed its charter school law.

In addition to academic studies, however, the most definitive measure of educational success is that charter schools are popular with parents. Over 600,000 children are on waiting lists hoping to attend a charter school, indicating that families of all backgrounds find charter schools a desirable option for gaining a quality public education.

The experience of other states indicates that if the current ban is lifted, charter schools would likely become popular with Washington families as well. The data show that lifting Washington’s ban on charter schools would provide thousands of children the opportunity to escape the low academic performance and high dropout rates found in many of the state’s traditional public schools.

Liv Finne is director of the Center for Education at Washington Policy Center. She is the author of numerous studies on education reform, including “Washington Policy Center’s Education Reform Plan: Eight Practical Ways to Reverse the Decline of Public Schools,” “An Option for Learning: An Assessment of Student Achievement in Charter Public Schools,” “An Overview of Public School Funding in Washington,” “Early Learning Proposals in Washington State,” and “Reviewing the Research on Universal Preschool and All-Day Kindergarten.” Liv holds a law degree from Boston University School of Law and a Bachelor of Arts degree from Wellesley College. She retired from civil litigation practice to raise two children and work as the business partner for Finne Architects, a small business she owns with her husband.

Download a PDF of the Policy Brief here.