“No Excuses” charter schools for increasing math and literacy achievement in primary and secondary education: a systematic review

Abstract

This Campbell systematic review examines the effects of No Excuses charter schools on students? math and literacy achievement gains compared to similar students in public schools. The review summarizes evidence from 18 studies, including 5 randomized controlled trials and 13 quasi‐experimental studies. No Excuses charter schools, on average, produced larger math and literacy achievement gains for their students than their public school peers ‐ with higher gains for math. These benefits increase for three years, at which point the achievement gains stabilized or returned to lower gains. Plain language summary No Excuses charter schools associated with greater gains in math and literacy than traditional public schools The No Excuses charter school model focuses heavily on high academic expectations, rigid and consistent discipline, extended instructional time, intensive teacher training, and increased parental involvement. This review examines the effect of No Excuses charter schools on math and literacy achievement. On average, No Excuses charter schools are associated with greater student gains on standardized measures of math and literacy achievement when compared to traditional public schools – with higher gains for math. What did the review study? Students from low socio‐economic backgrounds or traditionally disadvantaged groups often underperform their peers on standardized tests of math and literacy achievement. This “achievement gap” is associated with negative education and career outcomes – both short‐term and long‐term. No Excuses charter schools are intended to reduce this gap and improve the achievement gains of traditionally disenfranchised groups. For the purposes of this review, No Excuses charter schools are those charter schools that highlight the importance of high academic expectations for all students, rigid discipline enforcement, extended time in school, intensive teacher training, and parental involvement. This review examines whether No Excuses charter schools are associated with greater achievement gains in math and literacy compared to the achievement gains of similar students enrolled in traditional public schools. What studies are included? This review includes studies that evaluate the effects of No Excuses charter schools on students’ literacy and math achievement gains. It includes 18 studies conducted in the United States spanning from 1990‐2015. What is the aim of this review? This Campbell systematic review examines the effects of No Excuses charter schools on students’ math and literacy achievement gains compared to similar students in public schools. The review summarizes evidence from 18 studies, including 5 randomized controlled trials and 13 quasi‐experimental studies. What are the main results of this review? No Excuses charter schools, on average, produced larger math and literacy achievement gains for their students than their public school peers – with higher gains for math. These benefits increase for three years, at which point the achievement gains stabilized or returned to lower gains. What do the findings of this review mean? No Excuses charter schools may help underperforming students make larger achievement gains in math and literacy, more so than at traditional public schools. Additional research is needed to examine the effects of this intervention while accounting for significant issues related to sample bias and to determine the scalability of the model. What are the limitations in interpretation of this review? Readers should take caution when interpreting this review due to several limitations. First, studies were inconsistent in reporting (or failed to report) important sample characteristics such as percentage of students receiving special education services, number of students exiting or expelled after enrollment etc., severely limiting interpretation of the equality of comparison groups and factors impacting differences in outcomes. Second, follow‐up years suggest a sustained significant effect on both math and literacy achievement. However, these results must be interpreted with caution as the number of samples decreased substantially at follow‐up years with just one study accounting for the majority of the samples. Third, studies included in this review did not report data indicating the degree to which each characteristic of No Excuses charter schools was implemented, prohibiting the interpretation of the presence, absence, or dosage of the individual characteristics of No Excuses charter schools. Fourth, only five of the 18 included studies employed random assignment, limiting interpretation of the equality of the comparison groups due to the small number in independent samples using random assignment. Fifth, a large number of the included studies examine the effects of the schools of a specific charter management organization: Knowledge is Power Program (KIPP). Though informative, the results of this review cannot be considered generalizable to all No Excuses charter schools. Finally, most studies measured effects with only one, annual standardized achievement measure. How up‐to‐date is this review? The review authors searched for studies through June 2016. Executive summary/Abstract BACKGROUND In response to the school accountability movement and the move toward market‐oriented education policies in recent decades, many schools and districts have felt increased pressure to find effective ways to boost student achievement. However, many districts—largely those serving low‐income black and hispanic students in urban settings—have still struggled to make sufficient academic gains (Barton & Coley, 2010; Jacob & Ludwig, 2008; Yeh, 2016). One model of urban school reform aimed at leveling achievement disparities in the public school systems that has garnered attention (both positive and negative) is the No Excuses model. The No Excuses model is driven by the philosophy that there should be no excuses (e.g., poverty) for low academic performance. This model has found its footing in charter schools—publicly funded but privately managed schools exempt from the restrictions of the laws and regulations affecting non‐charter public schools. Due to the autonomy afforded to the management of charter schools, educators have the freedom to implement the tenets of the No Excuses model without the restrictions placed on non‐charter public schools. Along with the expansion of charter schools, overall, No Excuses charter schools have been increasing in number since their recognition in the 1990s. No Excuses charter schools primarily draw from low‐income and minority communities with the aim of minimizing racial and income‐based achievement gaps by improving the academic performance of their students. Although a definitive definition of the No Excuses charter school model does not exist, most definitions share a number of key characteristics—most Not ably: (a) high academic expectations, (b) rigid and consistent discipline, (c) extended instructional time, (d) intensive teacher training, and (e) parental involvement (Dobbie & Fryer, 2013; Whitman, 2008; Carter, 2000; Golann, 2015). OBJECTIVES This systematic review and meta‐analysis aims to examine the available evidence on the impacts of No Excuses charter schools on students’ math and literacy achievement, relative to students enrolled in traditional public schools. The focal research questions are as follows: • Does the No Excuses charter school model enhance students’ math or literacy achievement more so than traditional public schools? • What is the magnitude and variability of the effects? • What are the limitations to the evidence? SEARCH METHODS We conducted a comprehensive systematic literature search to identify all eligible studies, regardless of publication status. For the purposes of this review, the term “studies” is used to refer to separate reports and “samples” to refer to independent samples of students within (and over multiple, in some cases) studies. The search was conducted using electronic databases, internet search engines, citations in previous meta‐analyses and literature reviews, citations in research reports screened for eligibility, conference listings, hand searches of relevant journals, and correspondence with experts in the field. We screened redactions and erratum for potentially eligible studies. We used date‐of‐publication restrictions limiting our search years to 1990 or later as No Excuses charter schools did not exist until the mid‐1990s. Selection criteria Published or unpublished studies using randomized controlled trials (RCTs) or quasi‐experimental design (QEDs) to assess the effects of No Excuses charter schools on math and literacy achievement of elementary and secondary students were included in this review. To be eligible for inclusion in this review, studies must have been conducted or reported between January, 1990 and June, 2016. The date range reflects the time period during which No Excuses charter schools have been present in the U.S. School system. In addition, studies must have reported or controlled for baseline and comparison data, and comparisons groups must have comprised students attending traditional public schools. Studies determined not to reflect charter schools using a No Excuses model, or comparing No Excuses charter schools to other charter schools were excluded from this review. DATA COLLECTION AND ANALYSIS A team of reviewers assessed each title and abstract for preliminary eligibility, which was confirmed during full‐text screening. All studies that met the eligibility criteria in the citation and abstract, and full‐text screening were coded using a detailed coding instrument developed by the authors. Data extracted from primary studies included information on methodology, design, intervention and comparison group characteristics, math or literacy outcome details and characteristics, etc. Coders met to review the coding agreement and any discrepancies were discussed and resolved by consensus. This review contains a description of the intervention, a summary of the authors’ findings and conclusions, and a narrative synthesis of the identified literature. We report standardized mean differences with Hedges’ g. Separate meta‐analyses were performed to synthesize the studies assessing the effects of No Excuses charter schools on math outcomes and to synthesize the studies assessing the effects of No Excuses charter schools on literacy outcomes. Separate meta‐analyses were used to estimate the cumulative effects after one, two, three, and four years of attendance at No Excuses charter schools. Meta‐analyses separately report the estimated effects from RCTs and QEDs. We calculated Q, τ2, and I2 to assess heterogeneity. Moderator analyses were planned to explain between‐study heterogeneity in effects for several school‐level moderator variables, including percent male, percent minority, percent free/reduced lunch, percent of students provided special education services, school size, average class size, and years that the charter school has been active. However, few studies reported these characteristics at the school level; further, several variables (such as percent male, percent minority, percent free/reduced lunch, and percent special education services) did not have sufficient variability between studies to serve as useful moderators for effect sizes. As such, no planned moderators were consistently significant predictors of math or literacy achievement gains. RESULTS The initial search of both academic and grey literature identified a total of 4,554 citations. After duplicates and removal of studies prior to 1990, 2,841 citations remained. Titles and abstracts were screened for initial relevance and 2,386 of the remaining reports were excluded for irrelevance or ineligibility. Full‐texts of the remaining 455 studies were retrieved (no unretrievable reports) and screened for eligibility—18 of which passed full‐text screening for inclusion in the primary review resulting in 73 independent samples over four years. Analysis of these samples examined the effects of No Excuses charter schools on the math and literacy achievement of 70,121 elementary and secondary students. Five of the included studies were RCTs (Abdulkadiroglu et al. 2011; Angrist et al. 2010; Curto & Fryer, 2014; Hastings, Nekilson, & Zimmerman, 2012; Tuttle et al., 2015) and the remaining 13 used quasi‐experimental designs. The weight given to RCTs in the overall estimates (which pool the QED and RCT effects) of math and literacy effects were approximately 8% for year 1, 7% for year 2, 6% for year 3, and 5% for year 5. Studies included nongovernmental reports (9), journal articles (5), dissertations (2), working papers (1), and papers presented at conferences (1). All studies evaluated results using comparison groups comprised of students attending traditional public schools. The effect of No Excuses charter schools on math and literacy outcomes was found to be significant in this meta‐analysis. Similar to previous findings, a substantially smaller but significant effect on literacy was found. Math results The year 1 math analyses included 73 separate samples. The overall mean effect size for math outcomes was positive and significant (g = 0.202, 95% CI [0.148; 0.257]). The mean effect sizes were significant for QEDs (g = 0.207, 95% CI [0.148; 0.266]) and RCTs (g = 0.149, 95% CI [0.015; 0.283]) and were not significantly different from each other. There was significant heterogeneity in the estimates of the effect size (τ2 = 0.0430). Given the significant heterogeneity and potential bias for non‐equivalent groups, the mean effect size should be interpreted with caution (I2 = 89.4, Q = 676.58). At follow‐up years, the mean effect on math outcomes remained significant: Year 2 (n = 48), g = 0.293, p < 0.001; Year 3 (n = 33), g = 0.434, p < .001; Year 4 (n = 22), g = 0.299, p < 0.001. The estimated mean effects after adjustment for potential publication bias using trim‐and‐fill for all four years were also significant: Year 1 (0 trimmed, g = 0.207, p < 0.001), Year 2 (4 trimmed, g = 0.250, p < 0.001), Year 3 (6 trimmed, g = 0.331, p < 0.001), Year 4 (0 trimmed; g = 0.299, p < 0.001). Literacy results The year 1 literacy analyses also included 73 separate samples. The overall mean effect size for literacy outcomes was positive and significant (g = 0.069, 95% CI [0.034; 0.104]). The mean effect sizes were significant for QEDs (g = 0.061, 95% CI [0.025; 0.097]) and RCTs (g = 0.164, 95% CI [0.019; 0.309]) and were not significantly different from each other. Heterogeneity was minimal (I2 = 70.6, τ2 = 0.0121, Q = 244.77). At follow‐up years, the effect on literacy outcomes remained significant and attenuated from the math results: Year 2 (n = 48), g = 0.120, p < 0.001; Year 3 (n = 33), g = 0.212, p < .001; Year 4 (n = 22), g = 0.178, p < 0.001). The Egger's test indicated potential publication bias for the Year 1 literacy results (p = .047). After adjusting the mean effect using trim‐and‐fill, the Year 1 mean effect was still significant (7 trimmed, g = 0.041, p = 0.028), as were all follow‐up years: Year 2 (6 trimmed, g = 0.091, p < 0.001), Year 3 (3 trimmed, g = 0.168, p < 0.001), Year 4 (1 trimmed, g = 0.178, p < 0.001). AUTHORS’ CONCLUSIONS The present review identified available evidence regarding the effectiveness of No Excuses charter schools on math and literacy achievement. This review found multiple studies providing tentative support for the No Excuses charter school model as an effective intervention for improving students’ math and literacy achievement, with limitations. Further, this intervention was found to improve students’ math achievement more significantly that literacy. There were several significant limitations to this review. First and of utmost importance, studies were inconsistent in reporting (or failed to report) important sample characteristics (e.g., percent of students receiving special education services, number of students exiting or expelled after enrollment, etc), severely limiting interpretation of the equality of comparison groups and factors impacting differences in outcomes. Second, follow‐up years suggest a sustained significant effect on both math and literacy achievement, however these results must be interpreted with caution as the number of samples decreased substantially at follow‐up years with one study (Tuttle, Teh, Nichols‐Barrer, Gill, & Gleason, 2010), accounting for the majority of the samples. Third, studies included in this review did not report data indicating the degree to which each characteristic of No Excuses charter schools was implemented. This is a significant limitation as it prohibits the interpretation of the presence, absence, or dosage of the individual characteristics of No Excuses charter schools. Fourth, only five of the 18 included studies (approximately 5.75% of the independent samples) employed random assignment, limiting interpretation of the equality of the comparison groups due to the small number in independent samples using random assignment. Fifth, a large number of the included studies examine the effects of the schools of a specific charter management organization: Knowledge is Power Program (KIPP). Specifically, approximately 57% of the weight in Year 1 effect size estimates came from KIPP samples. At year 2, 3, and 4 follow‐ups, more that 57% of the weight in the effect size estimates are from KIPP samples. Though informative, the results of this review cannot be considered generalizable to all No Excuses charter schools. Of note, the results of this review reflect one unique model of charter schools and must not be generalized to all charter schools. Finally, most studies measured effects with only one, annual standardized achievement measure.