Reading the Research
Why research on charter schools matters
K–12 education can often be highly politicized and hotly contested.1 Charter schools are a particularly charged subject in K–12 education policy and reform. Research on charter schools has grown over the past decades as charter schools have expanded across the country. The scope and scale of charter school research continue to grow as more data become available.
Research on charter schools can help generate new information, uncover problems and point toward solutions. Findings from research can help foster a more civil dialogue and a more informed debate about how to improve education for all children, whether they attend a charter school or not.
This guide to research summarizes and explains research on charter schools across key topics, including Student Achievement, Diversity and Inclusion, Teachers and Teaching, Innovation, Finances, Governance and Regulation, Charter School Operators, Families and Public Opinion. The guide is designed to help research play a more meaningful role in how policymakers, journalists and communities think about charter schools and about children’s education generally.
However, research on charter schools can itself be the subject of vigorous debate. This guide includes research that has generated controversy. It attempts to explain some of those debates without taking sides. The section below outlines some research concepts and challenges to keep in mind as you read our guide and as you encounter new research on charter schools.
Terms and concepts to help understand the research
Some challenges researchers face studying charter schools
- Correlation and causation: Researchers often find that two things they are interested in occur close in time or space to each other or change together. That is called “correlation.”
For instance, a study might find that as schools introduce a new mathematics program, student achievement improves. It would be easy and perhaps intuitive to conclude that the new program causes better student outcomes. But unless the study is designed as an experiment or leverages a natural experiment—such as a lottery system that would randomly assign some students to the new program and other students to the existing mathematics program—researchers cannot conclude that the new program necessarily causes better outcomes. Other factors may be at play that are causing student achievement to improve. For example, principals may have sought to reward their best teachers by assigning them to the new program. Or the most affluent and engaged parents may have been the most successful in lobbying for admission into classes using the new program. Those other factors may be reasons for increased student achievement. That said, many studies find noteworthy correlations (or a lack thereof) that can contribute to a more informed and nuanced debate about schools and education and suggest hypotheses about causal relationships to be tested in further research.
It is also important to remember that even when an experimental study finds a causal relationship between two variables of interest, there are likely to be many additional factors that affect an outcome.2 For example, a study may find that being assigned a lottery-based seat in a charter school leads to higher student achievement on average. But there is still likely to be significant variation in the achievement of students within both charter and traditional public schools and hence many other factors that affect any individual student’s achievement.
- Significance and effect size: If a researcher finds that students who attend one type of school score higher on standardized tests than students who attend another type of school, how likely is it that those differences are just a fluke? Mathematical tests for statistical significance help researchers figure out how likely it is that the differences they observe between groups or the associations they observe between two variables are a result of chance.
However, even if a difference or association is statistically significant, that does not necessarily mean it is large enough to be of practical relevance. In general, studies with larger samples are more likely to identify statistically significant effects, even when those effects are substantively small. After having identified that a difference or an association is not a fluke (in other words, that it is statistically significant), researchers examine how large and how meaningful the difference or association (in other words, the effect) is. Effect sizes are calculated to help decide the size and practical relevance of an observed difference or association.
Effect sizes are calculated and reported in a variety of ways—some are more technical and others are more real-world. For example, differences in test scores in charter school research are typically measured in standardized forms rather than in actual test scores. Therefore, researchers sometimes report effect sizes using more technical terms, such as a “standard deviation.” When they do so, statistical differences may be reported as fractions of a standard deviation. But there are no set standards for interpreting standard deviation differences, and conventions vary across different analytic tests. Interested readers may want to look at Jacob Cohen’s writing on effect sizes, which has been particularly influential for researchers thinking about what constitutes small, medium or large standard deviation differences.3
Researchers may also translate statistical differences into more real-world effect sizes, such as “days of learning.” A statistically significant difference in test scores may turn out to be equivalent to just four or five extra days of learning. Or it might translate into several months of extra learning.
- Average and variation: Many research questions about charter schools revolve around comparisons. Studies may compare charter school students with traditional public school students, lower-income with higher-income students, different types of charter schools with traditional public schools or with one another, or charter school policies in one state with those in another.
These studies report on whether or not there are significant and relevant average differences among groups, schools or states. For example, a study may find no significant difference in charter school students’ average reading gains compared with the average reading gains of similar students in traditional public schools. Or a study may report that higher-performing charter schools on average have longer school years than lower-performing charter schools.
Highlighting average differences or lack thereof is an important research contribution. But part of understanding averages is also considering the degree of variation around that average. Even if a study finds no difference on average between charter school students’ and traditional public school students’ achievement scores, it may still report significant and relevant differences for certain types of students, for particular types of schools or in specific states or cities. Even within states or within a specific type of charter school, some charter schools may have greater impacts on student achievement than others, and impacts may be more notable for some types of students than others.
Paying attention to variation is crucial in charter school research, given that charter schools are a heterogeneous group of educational institutions, states vary in their regulations and oversight of charter schools, and charter schools—just like other types of schools—enroll many different kinds of students.
How to spot good research and how to interpret findings carefully
- How do we define and measure the phenomena we are studying? Turning an idea or issue into a researchable question requires clear definitions of terms and reliable measurements. Generating these definitions and measurements is rarely an easy process. For example, innovation is often cited as an important aspect of charter schools. But comparing innovation at charter schools with innovation at traditional public schools requires defining “innovation,” identifying specific practices as innovative and measuring them in precise and consistent ways. Standardized tests can provide useful metrics for some aspects of student achievement, but they do not measure every outcome that matters, such as future success in college and career, civic skills, communication or grit.
- Whom should we compare? When comparing charter schools with traditional public schools, or charter school students with traditional public school students, researchers need to think carefully about how to make fair comparisons. For example, charter schools are schools of choice, and there might be something inherently different about students and families that choose a charter school over a traditional public school. Even when studies carefully match charter school students with their peers in traditional public schools in the same district on demographic variables and test scores, it would be difficult to rule out the possibility that there are other important but nonmeasurable (or not measured) factors that distinguish charter students from those in traditional public schools.
Oversubscribed charter schools that admit students through a lottery system allow researchers to compare students who were randomly offered a seat in the charter with those who were not offered one. Such natural experiments provide opportunities for unbiased comparisons between a group of charter students and a group of students who also chose to apply to a charter school but did not get a seat.
- How much can we generalize? Students, schools and the laws governing them vary considerably across the country. For example, in the 2013–14 school year, 91 percent of public school students in New Orleans attended a charter school.4 But in Richmond, Virginia in 2012-13, only 0.8 percent of public school students attended a charter school, and eight states did not permit charter schools at all.5 Some charter schools are freestanding, while others are managed by larger organizations. States vary in their regulations, including whether or not they cap the number of charter schools that are allowed to open and the certifications they require for teachers. States, districts and schools differ in many ways, such as school financing and in demographics. And most charter school studies use samples of charter schools and students that are not representative of all charter schools and students. Researchers can therefore generalize their findings only to the specific student population, geographic location or type of charter school that they studied.
Nevertheless, any high-quality charter school study can help researchers, journalists, policymakers, practitioners and community members better reflect upon the dynamics and problems that may be occurring or about to occur with their students, in their schools or regions. Studies of other states, districts and schools can suggest questions that journalists, policymakers and practitioners may want to ask about their own communities. At the same time, it is important to consider individual studies as part of a larger body of research and to expand one’s scope beyond single findings to understanding the literature as a whole.
- Data can be difficult to find, inaccessible or nonexistent: Important questions about charter schools are often difficult to answer because data do not exist, are hard to access or are hard to compile. For example, the precise formulas for funding charter schools and funding traditional public schools can vary from state to state and district to district. Those formulas can change from year to year. Meanwhile, charter schools are often eligible for funding and finances through federal and state grant programs and from philanthropic foundations. This means that figuring out charter schools’ exact revenues, their spending and their contracting can be painstaking work—to say nothing of generalizing about those findings to the nation as a whole. About one-third of charter schools are operated by management organizations that run multiple schools.6 Some of these organizations are nonprofit and others are for-profit. Collecting and comparing data across operators can be difficult, in part because many operators manage schools in multiple states with different reporting requirements.
Research has sometimes been produced and interpreted to serve the political goals of advocates for and opponents of charter schools.7 Asking a few questions can help readers decide how seriously to take a study’s findings:
- How are the researchers defining and measuring the phenomena they are studying?
- What types of students, schools or locations are included in the study? To whom or where can the findings be generalized?
- How old are the data the researchers are using? How relevant are they to current policies and debates?
- If the study makes comparisons, are these fair and relevant comparisons? If it compares charter and traditional public school students, does it leverage a lottery admissions system or provide a detailed description of how it matches charter school students with traditional public school students in the same district?
- Do the researchers highlight variation in their findings? What kinds of variations are there?
- How relevant are the findings to the real world? How big are the differences the researchers find?
- What are the possible explanations for the correlations researchers find? Are there important variables the researchers overlooked or that were not available to them because of the limitations of their data? Are there alternative explanations to the ones suggested by the researchers?
- Does the study contain big news or surprises that seem too good or too bad to be true? Do the study’s findings complement or sharply diverge from those of other high-quality studies of the same issues? If they diverge, do the authors provide a compelling explanation for why this is the case and their study should be given greater credibility?
- Who wrote and funded the study? Were the research questions posed or were the findings presented in ways that suggest authors or funders have specific interests in promoting or opposing charter schools? Are the authors downplaying findings that do not advance their interests or overstating findings that do advance their interests?
- Was the study published in a peer-reviewed academic journal? Are the authors willing to share copies of their article and respond to additional questions about their work?
- Are media reports of research providing all the details you need? Are there important findings, methods, details or caveats discussed in the original publication that are missing from media reports?
For more detail on how to read, report on and use research with care, see the nonpartisan Council of State Governments’ "A State Official’s Guide to Science-Based Decision-Making”8 and the U.S. Department of Education’s What Works Clearinghouse.9 A recent article in The Atlantic describes some of the ways that journalists, policymakers and practitioners can avoid “oversimplifying or overstating” the results of education research studies.10 Harvard University’s Shorenstein Center has a useful primer on statistical terms used in research studies.