Contributed by:
It has long been thought that the United States education system is the great equalizer, lifting less advantaged children out of poverty and improving their chances for success in adulthood. The opportunity for economic and social mobility depends heavily, however, on access to high-quality education. Recent research has raised concerns about degradation in the quality of schools serving higher-poverty neighborhoods: The achievement gap between low and high-poverty students appears to have widened over the last quarter-century. In response to these concerns, federal, state, and local officials have enacted countless
education reforms to improve the outcomes of low-income students.
1.
CHILDREN AND FAMILIES The RAND Corporation is a nonprofit institution that helps improve policy and
EDUCATION AND THE ARTS
decisionmaking through research and analysis.
ENERGY AND ENVIRONMENT
HEALTH AND HEALTH CARE
This electronic document was made available from www.rand.org as a public service
INFRASTRUCTURE AND
of the RAND Corporation.
TRANSPORTATION
INTERNATIONAL AFFAIRS
LAW AND BUSINESS Skip all front matter: Jump to Page 16
NATIONAL SECURITY
POPULATION AND AGING
PUBLIC SAFETY
Support RAND
SCIENCE AND TECHNOLOGY
Browse Reports & Bookstore
TERRORISM AND Make a charitable contribution
HOMELAND SECURITY
For More Information
Visit RAND at www.rand.org
Explore the Pardee RAND Graduate School
View document details
Limited Electronic Distribution Rights
This document and trademark(s) contained herein are protected by law as indicated in a notice appearing
later in this work. This electronic representation of RAND intellectual property is provided for non-
commercial use only. Unauthorized posting of RAND electronic documents to a non-RAND website is
prohibited. RAND electronic documents are protected under copyright law. Permission is required from
RAND to reproduce, or reuse in another form, any of our research documents for commercial use. For
information on reprint and linking permissions, please see RAND Permissions.
2.
This product is part of the Pardee RAND Graduate School (PRGS) dissertation series.
PRGS dissertations are produced by graduate fellows of the Pardee RAND Graduate
School, the world’s leading producer of Ph.D.’s in policy analysis. The dissertation has
been supervised, reviewed, and approved by the graduate fellow’s faculty committee.
3.
Dissertation
Three Essays on Education
Reform in the United States
Ethan Scherer
C O R P O R AT I O N
4.
Dissertation
Three Essays on Education
Reform in the United States
Ethan Scherer
This document was submitted as a dissertation in July 2014 in partial
fulfillment of the requirements of the doctoral degree in public policy
analysis at the Pardee RAND Graduate School. The faculty committee
that supervised and approved the dissertation consisted of Jim Hosek
(Chair), Paco Martorell, and Jennifer McCombs.
PA R D E E R A N D GRADUATE SCHOOL
5.
Published 2014 by the RAND Corporation
1776 Main Street, P.O. Box 2138, Santa Monica, CA 90407-2138
1200 South Hayes Street, Arlington, VA 22202-5050
4570 Fifth Avenue, Suite 600, Pittsburgh, PA 15213-2665
RAND URL: http://www.rand.org/
To order RAND documents or to obtain additional information, contact
Distribution Services: Telephone: (310) 451-7002;
Email: order@rand.org
6.
Table of Contents
Figures............................................................................................................................................. v
Tables ............................................................................................................................................ vii
Abstract .......................................................................................................................................... ix
Acknowledgements ........................................................................................................................ xi
1. Does Information Change Voter Behavior? An Analysis of California’s School Board
Elections .................................................................................................................................... 1
Abstract .................................................................................................................................................... 1
1.1 Background on California School Board Elections and Accountability ............................................ 6
1.1.1. California School Board Elections ............................................................................................. 6
1.1.2. Standardized Tests and Accountability in California ................................................................. 9
1.2. Information and Election Timing Framework ................................................................................. 10
1.2.1 Retrospective Voting and Accountability/Information ............................................................. 11
1.2.2. Election Timing ........................................................................................................................ 12
1.3. Data ................................................................................................................................................. 14
1.4. Basic Empirical Model .................................................................................................................... 17
1.5. Results ............................................................................................................................................. 18
1.5.1. The effect of PSAA on incumbent reelection rates in odd years.............................................. 18
1.5.2. The effect of PSAA on incumbent reelection rates in even years ............................................ 19
1.5.3. Robustness checks .................................................................................................................... 21
1.5.4. Decomposition of the treatment effects .................................................................................... 23
1.5.5. The effect of print media on the implementation of PSAA ...................................................... 24
1.5.6. The effect of newly elected school board members on API ..................................................... 26
1.6. Conclusions ..................................................................................................................................... 28
2. Can recruitment and retention bonuses attract teachers to low performing schools? Evidence
from a policy intervention in North Carolina ......................................................................... 46
Abstract .................................................................................................................................................. 46
2.1. The structure of the Mission Possible program ............................................................................... 50
2.1.1. Recruitment, Retention, and Performance Bonuses ................................................................. 50
2.1.2. Staff Development .................................................................................................................... 52
2.1.3. Performance Accountability ..................................................................................................... 52
2.1.4. Structural Support..................................................................................................................... 52
2.2. Data and methods ............................................................................................................................ 53
2.3. Results ............................................................................................................................................. 59
2.3.1. Description of observable high school teacher changes ........................................................... 59
2.3.2. Difference-in-difference model of Mission Possible ............................................................... 60
2.3.3. Difference-in-difference specification check ........................................................................... 62
iii
7.
2.3.4. The entry and exit of Algebra I teachers .................................................................................. 63
2.4. Conclusion and policy implications ................................................................................................ 64
References .............................................................................................................................................. 67
3. Can principal and teacher bonuses improve student achievement? Evidence from a policy
intervention in North Carolina ................................................................................................ 81
Abstract .................................................................................................................................................. 81
3.1 The structure of the Mission Possible program ................................................................................ 85
3.1.1 Recruitment, Retention, and Performance Bonuses .................................................................. 86
3.1.2 Staff Development ..................................................................................................................... 87
3.1.3 Performance Accountability ...................................................................................................... 88
3.1.4. Structural Support..................................................................................................................... 88
3.1.5 Comparison of Incentive Design with Prior Random Control Trial Research .......................... 88
3.2. Data and methods ............................................................................................................................ 90
3.3. Results ............................................................................................................................................. 94
3.3.1 Simple difference-in-difference model of Mission Possible on Algebra I test scores .............. 94
3.3.2. Year-by-year difference-in-difference specification ................................................................ 95
3.3.3. Variation in the program’s impacts .......................................................................................... 96
3.3.4 Evidence that the size of the recruitment/retention bonus matters ............................................ 97
3.4. Discussion and policy implications ................................................................................................. 98
References ............................................................................................................................................ 101
iv
8.
Figure 1.1: Turnout rate among registered voters from 1995-2006.............................................. 35
Figure 1.2. Incumbent reelection rates by odd election years since accountability reform .......... 36
Figure 1.3. Incumbent reelection rates by even election years since accountability reform ........ 38
Figure 1.4 – Articles on the “Academic Performance Index” in California ................................. 43
Figure 2.1: Average percent of teachers with three years or less experience over time by whether
the school is a Mission Possible school. ............................................................................... 73
Figure 2.2: Average percent of teachers fully licensed over time by whether the school is a
Mission Possible school. ....................................................................................................... 74
Figure 2.3: Average turnover rates over time by whether the school is a Mission Possible school.
............................................................................................................................................... 75
Figure 2.4: Distribution of teacher value-added for MP and non-MP schools, before and after the
implementation of MP. ......................................................................................................... 76
Figure 2.5 Difference-in-difference estimates of MP on standardized teacher effectiveness by
year with clustered standard errors and placebo confidence intervals. ................................. 78
Figure 3.1: The estimated impact of Mission Possible reform on Algebra I standardized test
scores................................................................................................................................... 109
Figure 3.2: The estimated impact of Mission Possible reform on black students’ Algebra I
standardized test scores. ...................................................................................................... 110
Figure 3.3: The estimated impact of Mission Possible reform on free/reduced price students’
Algebra I standardized test scores....................................................................................... 111
Figure 3.4: The estimated impact of Mission Possible reform on English I standardized test
scores................................................................................................................................... 112
v
10.
Table 1.2: Difference Estimates of the Effect of Public Schools Accountability Act on Incumbent
Reelection Rates During Odd Election Years ....................................................................... 37
Table 1.3: Difference Estimates of the Effect of Public Schools Accountability Act on Incumbent
Reelection Rates in Even Years ............................................................................................ 39
Table 1.4: Robustness Checks of Estimates of the Effect of Public School Accountability Act on
Incumbent Reelection Rates in the even years ..................................................................... 40
Table 1.5: Difference Estimates of the Effect of Public Schools Accountability Act (PSAA) on
Incumbent Reelection Rates During the First Two Years of PSAA and for Uncontested
Elections ................................................................................................................................ 41
Table 1.6: Difference Estimates of the Effect of Public Schools Accountability Act on Incumbent
Reelection Rates.................................................................................................................... 42
Table 1.7: Difference Estimates of the Effect of Public Schools Accountability Act (PSAA) on
Incumbent Reelection Rates Year-by-Year .......................................................................... 44
Table 1.8. Change in Academic Performance Index after the election of a challenger ................ 45
Table 2.1: Incentive Structure for the Original Mission Possible Schools ................................... 71
Table 2.2: Student characteristics in analytic sample by school type ........................................... 72
Table 2.3 Average and standard deviation improvements in teacher value-added at Mission
Possible schools .................................................................................................................... 77
Table 2.4 Difference-in-difference estimate at various teacher effectiveness percentiles............ 79
Table 2.5 The value-added of teachers entering and exiting MP schools, compared to non-MP
schools................................................................................................................................... 80
Table 3.1: Incentive Structure for the Original Mission Possible Schools ................................. 105
Table 3.2 Summary Statistics for Mission Possible (MP) and Comparison Schools in 2006 (the
year prior to implementation) ............................................................................................. 106
Table 3.3 Summary Statistics for Mission Possible (MP) and Comparison Schools in 2006 (the
year prior to implementation) using propensity score weights ........................................... 107
Table 3.4 Estimate of the effect on Algebra I standardized scores from Mission Possible students
............................................................................................................................................. 108
vii
12.
It has long been thought that the United States education system is the great equalizer,
lifting less advantaged children out of poverty and improving their chances for success in
adulthood. The opportunity for economic and social mobility depends heavily, however, on
access to high quality education. Recent research has raised concerns about degradation in the
quality of schools serving higher-poverty neighborhoods: The achievement gap between low-
and high-poverty students appears to have widened over the last quarter century (Reardon,
2011). In response to these concerns, federal, state, and local officials have enacted countless
education reforms to improve the outcomes of low-income students. This dissertation examines
two of those reforms to better understand how and if they are working.
The first paper focuses on California’s state education accountability reform, which
allowed the state to identify low-performing schools and target improvement efforts. The paper
concentrates on a previously unstudied potential consequence of the reform: Whether the
information on school academic performance, which had been previously unavailable, enabled
voters to hold local leadership accountable. The results indicate that voting behavior depends on
how well informed the voter is on school-related issues. Those who are less familiar with
current school operations are more likely to utilize summary performance information—such as
aggregate school test scores—to inform their choice of locally elected leader. Conversely,
constituents who are more familiar with current school policies often discard student
performance information. Taken together, these findings indicate that providing performance
information to the public does not guarantee greater accountability of local education leadership.
ix
13.
The second and third papers assess a comprehensive reform to improve teacher and
principal talent in high-poverty, low-performing schools. While the reform has various
components, its main features are recruitment, retention, and performance bonuses for teachers
and principals in schools with a greater concentration of high-poverty students. The second
paper explores the intermediate outcomes of the reform by describing how it affected the
movement of teachers within the district. The paper finds that the reform significantly improved
the effectiveness of the teachers in the treatment schools. However, the improvements did not
stem from attracting higher-performing teachers or removing low-performing teachers. Instead,
the improvements were likely associated with a combination of effective leadership, improved
working conditions, and performance bonuses. The third paper expands on these findings by
exploring whether improving the talent within a school has an effect on student outcomes.
Results suggest that while the program was not effective during the first three years of
implementation, in the fourth year it shifted student performance by 0.28 standard deviations.
These results provide evidence that targeted performance bonuses for teachers and principals
could be used to improve academic performance in schools with large proportions of low-income
x
14.
First and foremost, I would like to extend a huge debt of gratitude to my outstanding and
dedicated dissertation committee: Jim Hosek, Paco Martorell, and Jennifer McCombs. While at
the Pardee RAND Graduate School (PRGS), I experienced some truly challenging times, and
through it all, they stuck with me. More importantly, they provided invaluable insight,
comments, and edits to all three of these papers. The final product would not be what it is today
without their assistance. I would also like to thank my outside reader, Katherine Strunk from
USC, for her thoughtful and comprehensive comments on my papers.
On-the-job training was an important part of my PRGS learning experience; I took away
much from these projects that I applied to my dissertation and will undoubtedly be able to apply
to future work. Therefore, I would like to thank all those with whom I have worked closely over
the years including: Jennifer McCombs, who gave me my first job; Paco Martorell, who is my
mentor and whom I aspire to be like one day; Seth Seabury, whom I am glad to call a friend as
well as a mentor; and Louay Constant, John Engberg, Robert Bozick, Gaby Gonzalez, Lindsay
Daugherty, Gema Zamarro and J.R. Lockwood. I would also like to give a special thanks to Paul
Heaton who taught two of the best classes I took at PRGS.
In addition, I wanted to thank my classmates, who were always there to discuss work-
and non-work-related matters with me and who made my years at PRGS enjoyable. Adam
Gailey, Chris Sharon, Helen Wu, Jack Clift, Sarah Kups, Vicki Shier, Debbie Lai, Amber
Jaycocks, Aviva Litovitz, Susan Burkhauser, Matt Hoover, and Christina Huang were among
those who helped me. Special thanks go to Sarah Kups, Adam Gailey, and Chris Sharon for
reviewing and commenting on earlier versions of this dissertation.
xi
15.
Finally, I would like to thank my immediate and extended family for helping me through
the ups and downs of the dissertation process. Judy and Maureen Manning provided invaluable
child care so I could work on my dissertation while having an infant at home. Both my father
and mother, Paul and Sherry Scherer, provided unwavering support that continues to propel me
forward. Finally, and most importantly, is my loving wife, Jill Scherer. She is not only the
reason that this dissertation is readable, but more importantly, encouraged me when things got
tough and always celebrated my successes, no matter how small. Without her, I wouldn’t be
publishing this dissertation.
I gratefully acknowledge support from the Susan Way-Smith dissertation award. I would
also like to thank the North Carolina Education Research Data Center (NCERDC) at Duke
University for their assistance in providing the data on which I based two of my papers. I am
fully responsible for the opinions expressed herein as well as any errors or omissions.
xii
16.
1. Does Information Change Voter Behavior? An Analysis of
California’s School Board Elections
Government is increasingly allocating resources to measure the relative performance of
providers of public goods, but the effect of this information on voting behavior is relatively
unknown. In this paper, I study how voters react to performance information shocks in school
board elections. I employ a difference-in-difference approach that compares incumbent re-
election rates before and after the implementation of California’s Public Schools Accountability
Act (PSAA) in districts discovered ex post to be high and low performing, relative to their
apparent peers. Using detailed district election data between 1995 and 2006, I find support that
in low-turnout years, voters tend to disregard new academic performance indicators in favor of
other forms of information. However, in high turnout years, voters use the academic
performance metrics to hold incumbents accountable, though I find little evidence that school
board members can affect the performance metric. Finally, I show suggestive evidence that
despite relatively uniform election coverage in the media, as time elapses and more information
becomes available, the incumbent re-election effects strengthen. Taken together, my findings
contribute a more nuanced understanding to the expressive and retrospective voting literature.
Over the last decade, local, state, and federal governments have invested significant public
dollars to create standardized metrics of comparison to aid policy-makers in their resource
allocation and help constituents understand if providers of important public goods are working
effectively and efficiently. Particularly since the passage of the federal law, No Child Left
1
17.
Behind, the education sector has been particularly prolific in its use of standardized tests to
compare academic performance across geographies. The availability of information about
performance creates an opportunity for greater accountability of elected officials. For example,
if a school district performs poorly on standardized tests relative to other districts in the state,
presumably, voters can vote elected officials out of office. However, it remains relatively
unknown empirically if voters or candidates use this public information and how its uses vary.1
A growing domestic empirical literature analyzes voters’ responsiveness to information on
political candidates.2 In particular, several strong empirical studies examine how increased
information improves voters’ knowledge of a candidate and election-day turnout, and can have
consequences at the ballot (see Healy and Malhorta, 2013 for a recent analysis). A parallel
literature empirically investigates the importance of timing on electoral outcomes (Anzia, 2011;
Berry and Gersen, 2011; Anzia, 2012). The authors of these studies find that switching from an
off-cycle election3 year to an on-cycle election year decreases elected officials’ responses to
dominant interest groups. Despite these advancements, it remains an open question how a
“shock” in performance information affects voters with different electoral timing.
In this paper, I present new evidence on how a consistent signal of performance
information—test scores of a school district—affects school board electoral outcomes. In
An exception to this is Berry and Howell (2007) who examine the effect of performance accountability in South
Carolina in a cross-sectional setting.
2
There is a large international literature using information shocks to identify its effect on candidates and voters.
These studies show that when information on performance or service quality improves, it can enhance government
or program responsiveness to constituents, reduce corruption in the short run (Besley and Burgess, 2002; Reinikka
and Svensson, 2005; Olken, 2007; Bojrkman and Svensson, 2009), and promote electoral accountability (Ferraz and
Finn, 2008; Ferraz and Finn, 2009; Banerjee et al., 2010; Chong el al., 2010). Yet, other research has shown that
information can increase candidate corruption in the long run (Bobonis et al., 2010). Candidates selected for
random audits in previous periods, no longer fearing that they could be selected, actually increased their corruption
beyond the initial levels.
“Off-cycle election” refers to an election held on a different day than other major elections, such as a presidential
or gubernatorial election.
2
18.
particular, it takes advantage of panel data to examine how constituents and candidates respond
to district performance information and how their composite reactions vary if the election is off-
or on-cycle. The analysis utilizes the state’s requirement, established under California’s Public
Schools Accountability Act (PSAA), to post publicly test score information, to rank schools of
similar demographic make-up, and to impose sanctions on poor performing schools after 1999.
There were no uniform reporting requirements or sanctions across the state for the four years
prior to PSAA.
My research design exploits the fact that I can identify districts that are similar on variety of
observable characteristics and that are discovered ex post – after PSAA takes effect – to be high
and low performing, relative to their apparent peers. I then use a difference-in-difference model
to identify the causal effect of this school performance information on incumbent re-election
rates. I am particularly interested in whether these effects differ in on-cycle (even years) and off-
cycle (odd years) election years. Based upon my theoretical framework, I expect to find a
difference in the treatment of information between on- and off-cycle elections. This allows me
to employ a triple difference model for the on-cycle elections where the third difference is
between even- and odd-year elections. Finally, I perform supplemental analyses to better
understand how the short-term effects of the program, the rerunning behavior of candidates,
media coverage, and the ability of a school board member to influence test scores can inform my
I find that in off-cycle years, test score information has no effect on voters’ decisions. In
fact, while not statistically significant, the sign of the effect indicates that voters might be
deliberately placing little weight on information on test scores. One interpretation of this result
is that information is only useful if it improves or reduces the noisiness of the voters’ prior
3
19.
assessment, since it is likely that voters during these years are more directly involved with
schools (e.g., teachers, parents in the parent teacher association). An alternative view is that self-
interested groups represent a higher proportion of the votes in off-cycle elections and vote for
their narrow interests, disregarding the performance metric.
In years with higher voter turnout, low test scores have a negative effect on the reelection
rates in poorly performing districts. The results are robust to measuring test scores in levels as
well as gains (though the effects for gains are smaller in magnitude, providing some evidence
that voters care more about the absolute rather than improvements in performance). These
results have a corollary conclusion to the odd-year results: Either voters do not pay attention to
the day-to-day operations of schools and hence use summative metrics like the Academic
Performance Index, or that the electorate is collectively less self-interested in this measure of
school performance than those in off-cycle years.
Next, I try to further undercover the make-up of the effects I observe during the on-cycle
years. First, I try to separate whether the effect I observe arises from the accountability reforms
or mainly from the information requirement of the law by exploring the short-term effects of
PSAA. As the accountability reforms took several years to take effect, the early analysis should
highlight information’s effect on voters. I find more than 2/3 of the overall effect’s magnitude
stems for informing the public. Yet, it seems more likely that the difference I observe between
even- and odd-cycles is due to longer-term effects. Second, since the effect I observe is a
composite of candidates’ rerun and voters voting behavior, I examine the rerun behavior of
incumbents. I find surprisingly little evidence that candidates incorporate the increase in
information in their rerun decisions and the majority of the effect is driven by voters removing
4
20.
Finally, I explore two possible reasons for the effects that I observe. First, I perform a simple
analysis as to assess whether school board members have much control over academic outcomes
by examining the test score growth rates after the election of a new school board member. I find
that when new members are elected to the board, school districts do not experience changes in
performance. This result possibly indicates that a rational voter would throw out metrics like
performance due to beliefs that a single school board member has little influence on test scores –
such as I find during the off-cycle election years. Second, I present some correlational evidence
that the more accessible and longer information is available on school, district, and state
websites, the stronger the effects are between even and odd cycle years. This result holds despite
relatively consistent levels of media coverage over the eight years post-reform.
However, unfortunately, both of these investigations for the underlying reasons suffer from
data limitation. Ideally in additional to the detailed voting data, I would have reports of who
teachers unions endorsed. This would enable to understand whether union strength is greater
during the off-cycle elections, and corroborate what other authors have found and place my
finding about new school board members in a more complete context. Further, it would be ideal
to have information about website usage or even detailed local newspaper and non-print media
coverage during on- and off-cycle elections rather than the crude tool of newspaper mentions
from a large database. These limitations mean than future research must work to uncover the
“black box” of why I observe these effects.
Overall, my results reflect a nuanced perspective regarding the role of information on
political accountability. In particular, more information does not always mean that it will affect
voter’s choice. Yet, since much of the prior literature’s elections are coordinated with national
and state-wide elections, my results are consistent with prior research that information enhances
5
21.
political accountability during major election cycles. Voters tend to punish incumbents in school
districts that perform poorly on the state accountability scale. At the same time, I also raise some
doubts whether standardized test scores are the correct metric that should be used to evaluate the
performance of school board members. This question should make policy-makers cautious when
designing information accountability system. Finally, my paper lends some support to the value
of websites and graphics that allow districts, cities, and the federal government to illustrate their
performance for those who tend to incorporate the information. These simple visuals (assuming
they gain viewership) could be an important check, beyond the media, on electoral power.
This paper proceeds in six sections. The first section provides background on school board
elections in California and the state’s Public Schools Accountability Act. The second section
provides a framework to assess my results. The third section describes the data. The fourth
section lays out my empirical model. Fifth, I review the results, and finally, I conclude.
1.1 Background on California School Board Elections and Accountability
1.1.1. California School Board Elections
School board elections provide a unique environment to test theories of information and
accountability. While federal and state education departments mandate some policy decisions,
most of the power to determine the educational vision, goals, and policy for a district resides at
the local level. Much like corporate boards, the school board focuses on the ends, while their
elected official, the superintendent (the CEO of a school district) focuses on the means. The
board’s objective is to align the goals, standards, and strategy with the budget of the school
district. They are also responsible for using data and public testimony to assess their progress
6
22.
towards these goals and make corrections as are needed. One of their key metrics to evaluate
school performance in the modern day, are standardized test scores. Finally, unlike other locally
elected officials, like a mayor, school boards central focus is on a single public good: a student’s
education. This combination of a standardized metric of performance and the primary function
of providing each student with the highest quality education means school board elections are an
excellent stage to assess my research questions.
California is an interesting landscape to assess school board elections because of its diversity.
School districts in the state can contain only elementary schools (K-8), only high schools (9-12),
or they may be “unified” and include elementary and high schools. In the 2011-12 school year,
there were 540 elementary, 80 high school, and 338 unified school districts.4 Even among these
broad classifications, school districts can range significantly in population size and diversity.
About 10 percent of all districts are "very small" (serving fewer than 100 students) according to a
Legislative Analyst Office report in 2011.5 Moreover, 230 of the state's districts contain only a
single school. These “small districts” can be contrasted with Los Angeles Unified School
District (LAUSD) that manages 937 schools and serves over 650,000 students.6 Furthermore,
while LAUSD’s student population is 90 percent non-white, Laguna Beach Unified’s non-white
population is 18 percent.7 This heterogeneity enhances the generalizability of my findings to
other states.
The timing of California’s school board elections is fairly unique, and recent evidence has
shown that timing of elections is important. California is one of eight states with a mixture of
California Department of Education, http://www.ed-data.k12.ca.us/Pages/Home.aspx
California Legislative Analyst Office,
California Department of Education, http://www.ed-data.k12.ca.us/Pages/Home.aspx
Ibid.
7
23.
on-cycle and off-cycle elections (Anzia, 2011). While a considerable political science literature
examines the effect of non-presidential and off-cycle elections on voter turnout,8 a much smaller
and emerging literature has examined some of the policy and voting consequences of these off-
cycle elections (Dunne, Reed, and Wilbanks, 1997; Trounstine, 2010; Anzia, 2011; Berry and
Gersen, 2011; Anzia, 2012). Many of these studies focus on teachers unions, the largest and
most powerful interest group during these elections (Moe 2006; 2009). They find that changing
the timing of voting translates into much lower voter turnout, which favors organized interest
groups (Anzia, 2011; Berry and Gersen 2011; Anzia, 2012). The lower turnout allows teachers’
unions to wield greater power on the elected board, translating into higher teacher compensation.
For my analysis, I utilize the difference in voter turnout between elections during even and
odd years. In California, almost 40 percent of school board elections in my data occur in “off-
cycle” years (e.g., 1999). Figure 1 shows the turnout rate, defined by the total number of school
board votes in my sample divided by the number of registered voters reported on the California
Secretary of State website.9
Similar to the findings of the previously mentioned literature, I find that voter turnout is
lower during off-cycle elections. Turnout rates are about a third to a quarter lower in the off-
cycle years when compared with the even years. With one exception, California’s special
election for governor in 2005, voter turnout in odd years is consistently lower than 20 percent of
registered voters. As others have found, this pattern leads me to believe that the nature of these
elections are different. During the odd election years, it seems likely that voters are more likely
to be parents actively engaged in their children’s schools and/or teachers and administrators
Although not complete, see Wolfinger and Rosenstone, 1981; Patternson and Caldeira, 1983; Hess, 2002.
Note that these participation rates differ from Berry and Gersen (2011) because they used voting-age residents
rather than registered voters as reported by the state. In addition, they need to throw out several important large
districts from their data, such as Los Angeles Unified School District and San Francisco Unified School District.
8
24.
directly involved with schools. For these odd-year voters, a comprehensive score like the
Academic Performance Index may not be as informative since they may be closely involved in
the day-to-day activities of the school/district, or as some have argued, may care more about an
interest group’s endorsement than performance of the district (Moe 2006; 2009; Anzia, 2011;
Anzia, 2012).
1.1.2. Standardized Tests and Accountability in California
Although California has a history of measuring students’ academic performance, the state
lacked a common measurement system for several years leading up to accountability reform.
From 1990 to 1995, California used the California Learning Assessment System (CLAS) to
measure students’ academic outcomes. However, after poor performance on the National
Assessment of Educational Progress, sometimes referred to as the nation’s report card as it
provides rankings across states, CLAS was discontinued. School districts then selected their
own tests, making it impossible to compare student performance across the state. In an effort to
remedy this problem and enable reward of “good” schools and punishment of “bad” schools,
California enacted the Public Schools Accountability Act (PSAA) in April 1999. PSAA has
three main components: the Academic Performance Index (API), the Immediate
Intervention/Underperforming Schools Program (II/USP), and the Governor’s Performance
Award (GPA).10
By identifying how schools were performing, the state could reward good schools, and
provide additional monetary assistance to schools that consistently fail. Schools that continually
miss targets become labeled as II/USP, which requires them to offer additional tutoring for
2008-09 Academic Performance Index Report: Information Guide.
9
25.
students, provide options for sending students to an alternative school, or close programs down
entirely. The law also created a “similar school ranking” by identifying 100 school cohorts of
similar schools based upon observable characteristics and ranking them by deciles. Given the
circumstances in 1998, the introductory API scores represented a significant increase in the
information available to voters on the performance of school districts.
For this study, I focus on the information conveyed in the API score. The API is a numeric
index between 200 and 1,000 and reflects a school’s performance across all standardized tests
administered by the state. As a composite index, it combines into a single score the results of all
the tests from the Standardized Testing and Reporting Program (e.g., math, English and subject
tests) with the California High School Exit Exam. Depending on where a school falls on the
scale, a certain amount of growth is required every year to comply with PSAA. For example, if a
school is between 200 and 690, the required growth to pass is five percent of the difference
between the school’s score and 800. The first baseline API scores were collected in 1999, and
schools were subject to accountability in 2000.
1.2. Information and Election Timing Framework
The following framework is to clarify some of the underlying incentives and policy results
for an information shock when the timing of an election varies. In particular, I briefly discuss
the retrospective voting literature. Then I use a simple cost and benefit model of voting behavior
to show how the median voter changes based upon the specific election. The cost-benefit
framework is based upon the work of Dunne, Reed, and Wilbanks 1997 as well as the theoretical
model developed by Banerjee et al., 2010.
10
26.
1.2.1 Retrospective Voting and Accountability/Information
Retrospective voting, how citizens respond to their perception of government performance,
plays a key role in democratic accountability (Besley, 2006; Ashworth, 2012). Until recently,
much of the empirical work has focused on economic retrospective voting – the use of economic
performance indicators to hold candidates accountable (Anderson, 2007). However, there are
several reasons to be concerned about the validity of the observed effects of economic
retrospective voting including, but not limited to: 1) the extent to which a politician’s fiscal
policy can influence the larger economy, 2) what are the “right” economic indicators to utilize,
and 3) the right counterfactual – what the economy would have been like under a different
candidate (Healy and Malhotra, 2013). To address some of these concerns a recent portion of the
literature has begun to explore non-economic indicators, such as test scores, disaster response,
and war causalities (Berry and Howell, 2007; Healy and Malhotra, 2009; Bechtel and
Hainmueller, 2011; Karol and Miguel, 2007; Kriner and Shen, 2007; Grose and Oppenheimer,
2007). These studies generally find that voters respond to positive and adverse information
about candidates.
This paper builds upon this literature by using test score performance to understand
retrospective voting in school board elections. As noted above, a school board’s central goal is
to produce a high quality education for students and it can be evaluated using a standardized
metric: test scores. After the introduction of accountability in California, my data allows me to
identify districts that appear similar based upon observables characteristics, but differ based upon
their students’ performance. If voters use this information to hold candidates accountable, we
would expect to see differential incumbent reelection rates between “low” and “high”
performance districts.
11
27.
It’s important to note here that up until this point, that I have used information and
accountability interchangeably, since an important component of the accountability reforms was
providing information to the public. However, accountability, sanctions and rewards for
performance of a district/school, itself could be driving any effects that I observe. However,
since the reasons and actions items for low performing school improvement were not due to the
state until January 2002, it is possible to separate the effect of accountability and information
empirically during the early years of the program.
1.2.2. Election Timing
To understand how the median voter might change for different types of elections (e.g., on-
or off-cycle), first consider a single jurisdiction comprised of different voter types utilized in
Dunne, Reed, and Wilbanks (1997). To illustrate the different interest groups, the authors
consider a cash transfer allocated by school boards. The first group is teachers and
administrators who receive a direct benefit from the resource transfer, presumably in the form of
salary and benefits increases. The teachers and administrators are the smallest fraction of
registered voters. The second group is parents who experience an indirect benefit from the
resource transfer through enhanced quality of education for their children.11 Parents make-up a
larger portion of the electorate than teachers. The final group is the general electorate that
experiences positive externalities from high-quality education through reduced crime and other
outcomes. These voters are largest group among registered voters.
I now consider these types of voters within a cost-benefit framework. Consider a benefit
Bi(n, e), which is a function of n, the number of voters and e, the number of elections on the
ballot. The benefit is indexed by i, the type of voter. The benefit is decreasing at a decreasing
Note that empirically, the relationship between school expenditures and education quality has been shown to be
weak (Hanushek, 1997). However, for expositional purposes, it makes sense to highlight the connections.
12
28.
rate in n and increasing in e. Benefits can be ranked by the type of voter with teachers and
administrators receiving the largest benefit and the general electorate receiving the least based
upon the example in the prior paragraph. The changes in type can be thought of as an intercept
shift in the benefit. Based upon this ranking, different groups have different incentives to
turnout, holding the number of voters constant. There is a fixed cost, ̅ , of voting in an election.
Next I consider how these costs and benefits differ from year-to-year. The cost of attending
the polls is fixed because staying informed, etc., does not vary between an even and odd years.
In other types of elections it might be more costly to acquire information during the off-cycle
elections, however based upon a media scan in Lexis-Nexis of the term “school board,” and
“California” I find little difference in the print media coverage of school boards. Furthermore,
there is virtually no television and radio transcript coverage of school board elections in Lexis-
Nexis indicating that this cost assumption is reasonable. In contrast, the benefit of voting does
vary from year to year. As there are more elections on the ballot during even years (e.g.,
presidential as well as school board elections) the benefits are higher for everyone during these
years. Using the formulation above we can write a simple turnout equations where it must be
true that , ̅ . Considering this equation, those with the highest benefits (e.g., teachers
and administrators) will turnout in years when the number of elections on the ballot is small
(e.g., only the school board is on the ballot). Presumably this logic helps to create the pattern
noted in Figure 1. Based upon this model, the median voter will be closer to the administrator
and teacher group when the timing of the election includes fewer elections on the ballot.
Continuing with the example of the three voter types, we can also understand how useful
standardized test score would be. Teachers and administrators are likely the most informed
voters, while the 18 year old with no children in the school system is unlikely to follow the day-
13
29.
to-day operations of the school. Thus, a sudden increase in the amount information about a
school will be least useful to those with the most information, teachers and administrators. An
important parallel argument is special interest groups, teacher unions, might be the most likely to
vote their interest over objective information on schools (Moe 2006; 2009; Anzia, 2011; Anzia,
2012). Both of these will result in the same observed outcome in my data, that voters with a
median closer to teachers and administrators will be more likely to disregard objective
information shocks in elections. Conversely, those voters who are least informed are more likely
to utilize publically available information for their election decision.
Within the context of my data, I use this framework to make several predictions. Based upon
the framework developed above, when there is the smallest benefit (odd-year elections with only
the school board on the ballot) only those most motivated turn out. These voters will be more
likely to be part of interest groups (e.g. teachers, parents) who monitor decisions of the school
boards more closely. Thus, empirically, we would expect to see the odd-year voters disregard
the API. During even years, Figure 1 shows electoral participation is relatively high and likely
shifts the median voter towards the general electorate because the benefits of having a national-
or state-level office on the ballot increases the voting benefit. However, this means that the
median voter knows less about their local school board elections. If this is the case, empirically
we should observe that voters are more responsive to the API.
1.3. Data
The data used for this analysis is a combination of three large datasets. First and most
importantly, I use the California Election Data Archive (CEDA) compiled by California State
University, Sacramento to analyze voting behavior. CEDA has detailed local election results,
including election date and office, as well as the number of votes received by each candidate
14
30.
running for office for all state elections and ballot initiatives in California from 1995 through
2006.12 From this dataset, I kept all school board elections held during the period.
I merged these data with detailed information from the California Department of Education
(CDE) API data files. These data files not only include the API scores for 1999-2006, but other
school characteristics, including the California School Character Index (SCI) and its
components. The SCI calculates predicted API scores by taking into account important
characteristics of schools that might make them higher or lower performers.13 The SCI uses
characteristics, such as pupil mobility14, ethnicity, socioeconomic status, percent of teachers who
are emergency and fully credentialed15, percent of English language learners, average class size
per grade level, and type of school year (e.g., all year round or nine months), to estimate a single
index. All the school-level data, including the SCI and API are aggregated up to the district level
by weighting each school by its proportion of district enrollment.
Finally, in order to include district demographics prior to 1999, I use the National Center for
Education Statistics Common Core of Data (CCD). The CCD collects basic racial composition
There were some inconsistencies with the data. Sometimes the sum of the individual votes did not equal the total,
the percentage of the vote share did not match the percentage reported, or one of these three variables was missing.
In these cases, I imputed the variable based upon the other available variables. In all cases, at least two of the three
variables needed to calculate the third were available.
The SCI uses a regression-based method to construct the index, by regressing the API score on factors including:
pupil mobility, pupil ethnicity, the education level of parents, the percentage of free or reduced price lunch in the
school, percentage of students classified as English language learners, if the school is multi-track or a year-round
school, average class size, and percentage of credentialed teachers. After estimating the regression, the actual
school characteristics are multiplied by the regression coefficients to yield the index value. There are a handful of
other small augmentations. For further detail on the process, see the document available on the California
Department of Education website: http://www.cde.ca.gov/ta/ac/ap/documents/tdgreport0400.pdf. The SCI is then
used to create cohorts of similar schools and the state reports a similar school rank. The similar school rank is not
used for accountability purposes.
“Pupil mobility” is defined as the percent of students who first attended the school.
In the teaching profession, in order to teach, it is required that the teacher be certified by the state. However, in
cases where there are no certified teachers available, districts do allow “emergency credentialing” that allows
someone without a credential to teach in the classroom.
15
31.
and the percentage of free/reduced price lunch students (a proxy for poverty) for all schools in
the United States.
To construct treatment and comparison groups, I consider a “slightly sophisticated voter”
who examines the API scores of similar school districts in the state to compare against his
district’s API score. I elect this method because being able to rank similar schools is an
important part of PSAA.16 To accomplish this, I use the SCI index. I rank ordered the SCI and
matched each district to its closest in composition, as measured by SCI, without replacement.
Within each pair, I noted which school had the worse API score and designated them in the
treatment group. Thus, the post-period voting record in the revealed higher performing school
districts provide the estimate of the “counterfactual”—similar districts with better performing
school board members.
Because parents needed to make decisions about locating in school districts based solely on
observable factors, it seems reasonable that prior to the test score release in 1999/2000 they
would view two districts on the SCI index as similar in quality.17 Then the 1999/2000 API score
provides a strong signal of quality.
The top panel of Table 1 reports observable characteristics of high- and low-performing
districts for even and odd years using my preferred method to assign the groups: the SCI. Based
upon the p-values reported in columns (3) and (6), the observable characteristics of the two
samples, with the exception of the API score, are not statistically different from one another. Of
course, by design, the 1999 API scores differ significantly, where the high-performing group
consists of higher performing districts.
Note that I consider other methods in my specification checks in the results section.
The exact weighting by parent of certain district/school characteristics could be different than the weights in the
calculated SCI (e.g., only placing weight on school race or percent free/reduced lunch). However, absent knowing
these weights, the SCI is a good approximation.
16
32.
In the lower panel, we see that prior to the state accountability standards, there is no
difference in the incumbent rerun and re-election rates except in the odd years the high
performing group is a little more likely to rerun. This could potentially bias the outcome
downward, which I will address in my results section.
1.4. Basic Empirical Model
To analyze whether I observe retrospective voting for the California school board elections I
use a difference-in-difference estimation strategy. In particular, I am interested if after the
introduction of PSAA whether the incumbent reelection rates change differently for the relatively
lower performing district. To estimate this I use the following equation:
vit 1lowperformit * postt 2 lowperformit 3 postt 4 X it t t d i it (1)
Where vit is incumbent re-election rates for district i in period t, lowperformit is an indicator
for a lower performing district, postt is an indicator which is 1 for all years after 1998, Xit is a
vector of observed district covariates18, tt is a vector of year fixed effects, and di is a vector of
district fixed effects. It is important to note here than any unobserved factors not accounted for
by matching on SCI will be accounted for with the district fixed effects. The key coefficients of
interest will be β1, which will indicate the percentage point change in the incumbent re-election
variable due to the implementation of PSAA identify which districts with similar observable
characteristics are higher or lower performing. I calculate Bertrand et al., (2004) standard errors
by block bootstrapping (i.e., clustered at the district level) 400 samples. For further reference on
this technique, see Bertrand et al. (2004).
Note that this set of covariates is less complete than those in the API files since I use the Common Core data for
these covariates. It consists of measures of poverty and race within the district.
17
33.
1.5. Results
1.5.1. The effect of PSAA on incumbent reelection rates in odd years
Figure 2 plots reelection rates on the vertical axis and the horizontal axis displays the
calendar years before and after the implementation of PSAA (with the year of PSAA denoted
year 0) for the odd years (e.g., year 0 equals 1999). The vertical line in this graph separates the
pre- and post-implementation periods. Although these numbers are not regression adjusted, the
figure demonstrates a graphical form of the basic difference-in-difference model. This simple
graph does not allow for a thorough analysis of PSAA in the odd years, but it does show
preliminary evidence that the high- and low-performing groups tend to mimic each other fairly
closely indicating there is no effect of the revealed performance information for these voters.
Table 2 presents several specifications for the difference-in-difference models during the odd
election years. In column (1) I present the simple difference-in-difference model shown
graphically with year fixed effects in the model. We cannot distinguish the coefficient from
zero. While that is the case, the coefficient sign takes a positive value indicating that lower
performing incumbents are more likely to be reelected in periods after the release of additional
information. Column (2) adds key controls and column (3) adds district fixed effects to the
model, but this does not change the significance and only increases the magnitude of the
coefficient. Graphically we can see that this effect might be largely driven by the 2005 election
(three odd election cycles after accountability reform), which as noted above was unusual
because of the special election held that year. Column (4) reruns the specification in column (3)
excluding 2005 from the analysis. While this does reduce the magnitude of the coefficient, it
remains positive and not statistically different than zero. Thus, the fact that incumbents in worse
performing districts tend to have higher reelection rates after the implementation of PSAA seems
to be robust to various specifications.
18
34.
In line with the theoretical prediction, these results provide preliminary support that odd-year
voters are throwing out the information on API score although the effect is not statistically
different than zero. The fact that we can’t distinguish the effect from zero means the odd years
provide an important comparison group to the even years that I will discuss further in the next
1.5.2. The effect of PSAA on incumbent reelection rates in even years
The way to test if there is a statistical difference between even and odd years, would be to
compare (1) high- and low-performing districts, (2) before and after the implementation of
PSAA, and (3) between even years and odd years. The best model to test this would be a
difference-in-difference-in-difference strategy. Now our new empirical model equation is:
vite 1lpit * postt * evene 2 eveni * postt 3evene * lpit 4lpit * postt
(2)
5 eveni 6lpit 7 postt 8 X it tt d i ite
where e indexes even or odd year and evene is an indicator variable if it is an even or odd
year. I have abbreviated the lower performing variable to lpit. The other variables in equation
(2) are similar to those discussed in equation (1). In this case, the three-way interaction term
between lower performing, post and even represents the estimate of PSAA effect in even years
on the incumbent’s reelection rates.
Prior to assessing the third difference it is important to see graphically if there is a
relationship between the high- and low-performing group in the pre- and post-intervention time
period. Figure 3 shows the relationship between incumbent reelection rates and the number of
election years since PSAA for high and low performing districts. This figure is parallel to Figure
19
35.
We want to be caution once again, since these are unadjusted numbers, however, prior to
PSAA we see that the treatment and comparison group have relatively parallel trajectories, the
key identifying assumption for a difference-in-difference model. However, after the
implementation the treatment group experiences a rapid decline in reelection rates. The lower
rate persists throughout the three even post-intervention election cycles.
The first column of Table 3 presents the even year difference-in-difference model using
specification (3) in Table 2. As expected, based upon Figure 3, we see a significant negative
effect of PSAA on being reelected to office in the post-period. These results confirm that voters
during the even years are voting lower performing districts out of office at a higher rate than the
their higher performing counterparts in the post-PSAA period. Thus, voters are holding
incumbents accountable during the even years.
The second column shows the triple difference estimate with controls, years and district fixed
effects in the model. This increases the magnitude of the effect and is also statistically
significant at the 5 percent level. The increase in magnitude clearly shows that when compared
to the odd-year electorate, the even-year voters are much more likely to use API scores to hold
incumbents accountable. These results show that PSAA had an effect on incumbent reelection
rates compared to the off-cycle years. Since the mean reelection rate is 65 percent, a 13
percentage point reduction is almost a 20 percent decline in the reelection rates of incumbents.
Thus, the release of information had a practical effect on the incumbent election rates.
Up till this point of the paper, we have been examining the API levels as the key criterion
voter use to assess school board performance. However, as mentioned in the data section, for
most school districts with scores under 800, the PSAA law holds schools accountable based upon
the growth of the scores from the prior year. In addition, the growth score is a better metric for
20
36.
the efficacy of the board since it demonstrates how much things improve year-over-year. The
growth score is also reported by most districts along with the level score. The third column of
Table 3 replicates the difference-in-difference specification in column (1) for the gains in even
years. The difference-in-difference model provides evidence that while voters react to the
information in the direction we expect based upon the pervious results, because we can’t
distinguish this effect from zero, it appears voters do not focus on gains as much as the overall
API. However, when compared to the voters of the odd years, the magnitude of the estimate is
similar to the levels model. Once again we see that the DD estimates show that in the post-
PSAA period voters reelected incumbents at a differential rate in the lower performing districts.
But when compared with the off-cycle electorate, it is clear that the two groups use the API
differently from one another. The gains DD and DDD results provide some qualitative evidence
that while the gains a district makes in API are important; the focus of voters is on the absolute
1.5.3. Robustness checks
Table 4 presents several alternative specifications to assess the sensitivity of the estimates.
First I construct an alternative treatment groups. In specification 1, I assume a less nuanced
voter who does not care about the demographics and other factors of the district and only cares if
the district is high or low performing. To accomplish this specification, I assign the lowest
quartile of school districts as my treatment group and the remaining sample is the comparison
group.19 These results show weaker, but similar effects as Table 3 for the levels model. I repeat
this for the gains score; however, small gains could be because it is hard for a high performing
district to improve. This fact means that the lower quartile of gains lumps districts across the
One quarter is fairly arbitrary, thus I varied the break point by +/- 20 percentage points with qualitatively similar
results to those presented here.
21
37.
performance spectrum into the treatment group. With this understanding, it is reasonable that the
gains specification should be interpreted differently than they are in Table 3.
Second, rather than identify a treatment and comparison group based upon SCI or API, I
incorporate both metrics, API-SCI, into my model directly. The new variable is a continuous
treatment measure that identifies the performance of a district (API) above or below its expected
performance based upon the characteristics of the district (SCI). I then plug my new measure in
place of β1 in equation (1) and (2).20 In this case rather than a negative effect, we hope to observe
a positive relationship between the continuous measure and incumbent election rates. This effect
would mean that as a district improves its performance over what is expected, incumbent
reelection rates should increase. In fact, in specification 2 in Table 4, this is what we observe for
the even-year difference-in-difference. The point estimate indicates that for a one point
improvement in API above the SCI increases incumbent reelection rates by 0.4 percentage points
though it is only marginally statistically significant. A one standard deviation in API-SCI is
about 50 point which would increase reelection rates by 20 percentage point. For the triple
difference, the observed effect is not statistically different than zero, though like the estimates in
Table 3, it is double the size of the difference-in-difference specification.
A third concern to the observed effect is that many of the school board elections are
uncontested (i.e., close to 40% of my sample). Uncontested elections build in a mechanical
relationship in my data such that regardless of the API score, the incumbent will be elected. To
address this potential concern, I include a dummy variable in the equation indicating whether or
not the election was contested. The results are in the third row of Table 4 show that while
For equation (2), I plug in the continuous measure and multiple it by even and include all the underlying
22
38.
uncontested election slightly reduce the magnitude of my estimates, compared to Table 3, they
have a minimal effect overall.
Finally, as noted in Figure 1 and 2, 2005 was an unusual election year and there is some
indication that it skews the results. As such, I run a specification without 2005 or 2006 in the
model to see if it changes the result. This specification does not substantively change the results
in Table 3. Taken together, my results are robust to alternative specifications of the estimates.
1.5.4. Decomposition of the treatment effects
Until this point in the analysis, I have combined the effect associated with the additional
information required by PSAA and the accountability mechanisms that it put in place. To
disentangle these effects I employ an additional strategy. While the information on test scores
were available immediately, the sanctions associated with PSAA were not fully put in place until
several years after the implementation of PSAA. Thus, if I observe an effect, especially in the
difference-in-difference model for the early years of PSAA, it will provide evidence that the
effect is due to information rather than accountability. The first column of Table 5 shows the
even-year difference-in-difference estimator in Table 3 column (1) where I exclude 2001-2006
from the model (i.e., the only post-PSAA year is 2000). While not statistically significant, the
magnitude of the effect is more than 2/3 of the size of that observed in Table 3. As such, there is
evidence that potentially a large portion of the observe effect in Table 3 is related to simply the
provision of information. However, the second column of Table 5 shows the triple difference
effect for the same time period. Here we observe the magnitude makes up a much smaller
portion of the total effect in Table 3. Yet, the triple difference specification measures something
slightly different than the difference-in-difference model. It measures how different types of
voters use the information/accountability. It is less likely that I would observe an effect of
23
39.
accountability using the triple difference between even and odd-year because accountability
should not affect schools differentially based upon the timing of their election. Thus, even
though the difference in the earlier years of PSAA is relatively small compared to the overall
effect that I observe in Table 3, I still believe the effect is mainly due to information rather than
Second, an important piece of the analysis is not only whether voters react, but do candidates
react strategically to the information. The effects that I observe in Table 3 are a composite of
candidates deciding not to rerun because of PSAA and the incumbents being defeated. To better
understand where the effect stems from, I perform a separate analysis of the candidate rerun
behavior using similar models. Table 7 replicates the even- and odd-year analysis for the
difference-in-difference model as well as the triple difference. While the signs of these effects
go in the same direction as what we observed in the Tables 2 and 3, the magnitudes are too small
to differentiate them from zero. However, the magnitudes of the coefficients tend to be about 1/3
to 1/2 of the size of the total effect. Therefore, some of the effect can be explained based upon
incumbents declining to rerun for office.
1.5.5. The effect of print media on the implementation of PSAA
Berry and Howell (2007) observe an initial effect of information in South Carolina’s 2000
school board elections that fades away in the 2002 and 2004. They explore this pattern and
conclude that a significant change in the type of news media coverage over time combined with
less precise measure of school performance lead to this result. While the next results are based
on a correlation, I observe a different pattern in California.
24
40.
First I scan large and local newspapers in California for the two months prior to school board
elections using the key search terms “school board” within the Lexis-Nexis database.21 In the
two months prior to the election this amounted to approximately 1100-2500 articles which given
the size of the state and the number newspapers included in the search, is fairly minimal
coverage. Within each of these searches I added the key term “Academic Performance Index.”
For the years of interest between 200022 and 2006, I observe that the API is mentioned in about 1
to 2 percent of the articles consistently across the years, with little difference between even- and
odd-cycle elections--a uniform distribution rather than the declining one found by Berry and
Howell. These results would imply that we would see consistent effect of the information across
the years after the intervention. Figure 4 shows a second test where I examine more broadly in
California newspapers the number of times the term “Academic Performance Index” appears.
Here we see again, that while there is a large jump in 1999 and 2000, the coverage is relatively
consistent in the subsequent years. A secondary point to the graph is that despite there being
over 700 districts in my sample and over 1,000 in California; we see in Figure 4 that the API is
mentioned less than 300 times in the two month prior to the election. Thus, among school
coverage, mentions of API are relatively infrequent.
To correlate the newspaper coverage findings with my data, I employ a third empirical
strategy. I continue to use a triple difference to measure the difference between the on- and off-
cycle election, but instead of using the pre- and post-intervention indicator (e.g., posti), I multiply
the even*treatment interaction by individual year dummies, where the omitted category is one
year prior to the intervention. Table 5 shows the three-way interaction by year from
LexisNexis does not compile all local newspapers in California, but tracks many of the major newspapers in
various regions of the state. This poses a limitation on the analysis.
There was little coverage of the Academic Performance Index in 1999, as can be seen in Figure 4, thus I exclude
it from the analysis
25
41.
intervention. As this requires considerably more covariates in the model, I only find results
statistically different than zero in the final year of my data. However, looking at the point
estimates we see that in the post-intervention period, the magnitudes of the coefficients are
increasing in size. These results conflict with the simple newspaper analysis in Figure 4. The
juxtaposition of the raw media attention on the API and the steadily increasing difference
between API usages between on- and off-cycle voters indicates that there is something else
happening at the same time. It could be that information presentation improved during this time
period and while the usage of API increased for on-cycle users, off-cycle users continued to use
other sources of information. While I don’t have direct evidence from saved web sites, the data
websites for API certainly improved in their data availability as well as their sophistication over
time. It seems reasonable that as districts and schools became better at communicating
performance to constituents, the effect of the information strengthened over time. Alternatively,
it could be the case that as teachers’ unions became more opposed to standardized test scores,
they increasingly rejected API as the performance metric over time. However, to investigate this
research question, a more thorough analysis with superior data to mine must be performed.
1.5.6. The effect of newly elected school board members on API
One possible reason that the low turnout years uses information about performance
differently than the high turnout years is that individuals could be skeptical of school board
members’ ability to change API. An easy test for this possibility is to examine how much a
school improves after the election of a new school board member. To accomplish this, I limit
my sample to 2000-2006, years where gains in the API could be measured. I created an indicator
whether any new school board members were added during a particular cycle. Then controlling
for the demographics in the SCI index and the lag of student test scores, I examine if there were
26
42.
any gains either two years or four years after the new school board members were elected. I
chose two and four years after the election cycle, since some members of the board (not the
individual) would be up for re-election two years afterward. After four years, the newly elected
school board member would be up for re-election again. 23
The results of the analysis are shown in Table 6. The first column examines if there is any
effect on all elections held 2-years after the initial one. While the magnitude indicates there is
actually, on average, a single point loss in the scaled score when a new school board member is
elected, this can’t be differentiated from zero. To understand what is driving this relationship
better, in columns 2 and 3 of Table 6, I break apart the effect into even and odd election cycles.
Based upon these results we actually see a small, but statistically significant negative effect of
adding a new school board member during the even years. This presents some evidence, that
often new school board members actually damage the progress of a school district. However,
given that the average gain in test scores over 2 years is 24 points, this “damaging,” effect is
practically small. One possible reason that we observe a small decline in district performance
after 2-years is that it takes time to adapt to the school system, and thus, two years is too small of
a period to try and measure performance.
To assess this theory, I also examined the performance of school board members four years
after they were elected. A cost to this analysis is that I lose further observations in my sample.
Columns four through six show the results of the four year gain. None of these results can be
distinguished from zero, although the magnitudes are similar to the 2-year gains. Furthermore,
with an average gain of 46 points in my sample, these results are practically equal to zero.
To be complete, I also examined results one and three years after the election with similar results.
27
43.
A final reason why we might not be observing an effect is that my indicator of a newly
elected school board is weak. The indicator is a one if any new members are elected. However,
if a single new member is added to a board of five or seven people, they have little power to
make changes where a majority is needed to pass any policy. As such, in results not presented
here, I also examined cases where all the incumbents running for office were defeated. The
results did not differ from what is presented in Table 6.
Certainly when talking about large districts, one could imagine that a two- or four-year time
period would be too short to see an effect, but there appears to be little evidence that school
board members have much influence on API. In fact, four years after an election, new school
board members have a small negative effect. As such, it would make sense if, as we observe in
the odd-years, that voters disregard the API scores as a good metric to evaluate school board
1.6. Conclusions
Prior research has shown that the publication of government performance through
newspapers, literature, or websites enhances the accountability of elected officials. These
findings imply that if the media or government improved reporting on politicians, then
government might be more congruent with what voters want. At a time when the national
congressional approval rate hovers around 15 to 20 percent, increasing transparency by
improving objective information on the effectiveness of elected officials appears to be good
public policy to better align congress and their constituents. Similar to other studies on
information, I find evidence that, during high turnout years, voters increasingly hold elected
officials accountable for their performance. On average, poorer performing incumbents’
28
44.
reelection rates decline by 7 percentage points in the eight years after the implementation of
PSAA. We can conclude that, during these high turnout years, information matters.
However, importantly, and central to this study, I find that the effect of information is not
uniform across the timing of elections. Based upon my conceptual framework, I predict that
those who benefit the most from district policy decisions, teacher and administrators, will turnout
in off-cycle elections, while the electorate without children in schools will be less likely. The
model then predicts that the off-cycle years will have a different median voter closer to the
interests of teachers and administrators. I then find evidence empirically that the interaction
between information and voters is different during off-cycle than on-cycle elections. First, I find
no statistically significant effect of PSAA on voters during the off-cycle elections. Then I show
that there is a 13 percentage point difference, in my preferred specification, between how voters
in the off-cycle and on-cycle elections treat PSAA information.
The paper then attempts to uncover one possibility for this effect: school board members
have a minimal effect on school performance. I find little evidence that new school board
members have a statistically discernible effect on students’ test score improvement. There are
many reasons why we would observe this effect. First, it takes more than a single term for a new
school board member to understand the workings of a district and have a lasting impact. I am
unable to test this hypothesis due to the limited size of my panel, but presumably, after a single
term, voters would expect to see some results. Second, it could be the case that getting a single
new member on a five person board has little effect on the policy decisions made by the school
board since it takes a majority of three to make any policy changes. I attempt to test for this by
examining the case where two or three new members were added to the school board in an
election. While I find no effect for this case either, it is likely there are other unobserved
29
45.
characteristics of an election which I don’t account for in my model when none of the
incumbents is reelected. Third, as reported in Table 1, incumbents have approximately a 65
percent reelection rate, thus I might not have enough instances in my sample of challengers
getting elected to find a statistically significant result. However, the point estimates on my
results indicate that during the even-years, where incumbents are held most accountable, there is
a practically small negative effect of electing a new school board member. Yet, the most
plausible explanation is that test scores might not be the best metric to evaluate school board
performance, despite the fact the even-year electorate consistently uses it to hold incumbents
How then do I interpret the difference between how off- and on-cycle constituents utilize
standardized test score information? There are two potential answers. First, prior research
(Anzia, 2011; Anzia, 2012; Berry and Gersen, 2011) has shown that teachers’ unions wield
significant power in off-cycle elections. Teachers’ unions have often adopted an adversarial
relationship with standardized test scores (Hernandez, 2013). Thus, the disregard of test score
information could be because teachers tend to use other metrics, like their union’s position, to
determine who they vote for during an election, and these effects are stronger on the overall
outcomes during off-cycle years. The second explanation is that because mainly teachers,
administrators, and active parents turnout for these off-cycle elections, they should have many
metrics of performance available to them (e.g., voting records on specific policy issues). If the
API is a poor or noisy predictor of school board performance, then it would make sense to
disregard this information, especially when compared to voters who know less about school
board politics. Unfortunately, my data provides limited ability to test which theory is more
applicable in the case of California, and future work must investigate which is more plausible.
30
46.
References
Anderson, CJ. 2007. “The end of economic voting? Contingency dilemmas and the limits of
democratic accountability. Annual Review of Political Science, 10: 271‐96.
Ashworth, S. 2012. “Electoral Accountability: recent theoretical and empirical work. Annual
Review of Political Science, 15:183-201.
Anzia, S F. 2011. "Election Timing and the Electoral Influence of Interest Groups." Journal
of Politics 73 (2): 412‐427.
Anzia, S. 2012. "The Election Timing Effect: Evidence from a Policy Intervention in
Texas." Quarterly Journal of Political Science. 7 (3): 209-248.
Bechtel, MM. and J. Hainmueller. 2011. “How lasting is voter gratitude? An analysis of short-
and long-term electoral returns to beneficial policy.” American Journal of Political
Science, 55(4): 851-67.
Banerjee, A, S Kumar, R Pande, and F Su (2010). “Voter Information and Electoral
Accountability: Experimental Evidence from Urban India.” unpublished manuscript,
Harvard University.
Berry, C. and W. Howell. (2007) “Accountability and Local Elections: Rethinking Retrospective
Voting.” Journal of American Politics 69(3): 844-858
Berry, C. and J. Gersen. (2011) “Election Timing and Public Policy.” Quarterly Journal of
Political Science 6(2): 103-135
Bertrand, M., E Duflo, and S. Mullainathan. (2004). “How much should we trust differences-in-
differences estimates?” The Quarterly Journal of Economics 119(1): 249-275.
Besley, T. 2006. Principled Agents? The Political Economy of Good Government. New York:
Oxford University Press.
31
47.
Besley, T. and R. Burgress (2002). The Political Economy of Government Responsiveness:
Theory and Evidence from India.” Quarterly Journal of Economics, 177, 1415-51.
Bjorkman , M. and J. Svensson (2009). “Power to the people: evidence from a randomized field
experiment on community-based monitoring in Uganda.” The Quarterly Journal of
Economics 124(2): 735-769.
Bobins, G., L. Camara Fuertes, R. Schwabe (2010). “Does Exposing Corruption Politicians
Reduce Corruption?” unpublished manuscript, Massachusetts Institute of Technology.
Chong, A., A. O, D. Karlan, and L. Wantchekon. “Looking Beyond the Incumbent: Effects of
Exposing Corruption on Electoral Outcomes.” Cambridge, MA: National Bureau of
Economic Research, NBER Working Paper No. 17679, December 2011.
Dunne, S., W. Reed, and J. Wilbanks. 1997. “Endogenizing the Median Voter: Public Choice
Goes to School.” Public Choice 93: 99-118
Ferraz, C. and F. Finn (2008). “Exposing Corrupt Politicians: The Effects of Brazil’s Publicly
Released Audits on Electoral Outcomes.” Quarterly Journal of Economics, 123, 703-45.
Ferraz, C. and F. Finn (2009). “Electoral Accountability and Corruption: Evidence from the
Audits of Local Governments.” Cambridge, MA: National Bureau of Economic
Research, NBER Working Paper No. 14937, April 2009.
Grose A., and B. Oppenheimer. 2007. “The Iraq War, partisanship, and candidate attributes:
variation in partisan swing in the 2006 U.S. House elections.” Legislative Studies
Quarterly 32(4): 531-57.
Healy, A. and N. Malhotra. 2009. “Myopic Voters and Natural Disaster Policy.” American
Political Science Review, 103(3): 387-406.
Healy, A. and N. Malhotra. 2013. “Retrospective Voting Reconsidered.” Annual Review of
32
48.
Political Science, 16: 285-306.
Hanushek, E. A. (1997). Assessing the effects of school resources on student performance: An
update. Educational evaluation and policy analysis, 19(2), 141-164.
Hernandez, J. “Union Chief Recommends Delay in Use of Test Scores” New York Times.
April 30, 2013.
Hess, F. M. 2002. “School Boards at the Dawn of the 21st Century.” National School Boards
Association
Karol, D. and E. Miguel. 2007. “Iraq war causalities and the 2004 U.S. presidential elections.
Journal of Politics 69(3): 633-48.
Kriner, D. and F. Shen. 2007. “Iraq war causalities and the 2006 Senate elections.” Legislative
Studies Quarterly 32: 507-30..
Lim, C., JM Snyder, and D. Stromberg (2010).“Measuring media influence on US state courts.”
Unpublished manuscript
Moe, T. 2006. “Political Control and the Power of the Agent.” Journal of Law, Economics, and
Organization. 22(1): 1-29
Moe, T. 2009. “Collective Bargaining and the Performance of the Public Schools.” American
Journal of Political Science. 53 (1):156-174
Olken, B. (2007). “Monitoring Corruption: Evidence from a Field Experiment in Indonesia.”
Journal of Political Economy, 115(2), 200-49.
Patterson, S. C., & Caldeira, G. A. (1983). Getting out the vote: Participation in gubernatorial
elections. The American Political Science Review, 675-689.
Reinikka, R. and J. Svensson (2005). “Fighting corruption to improve schooling: Evidence from
33
49.
a newspaper campaign in Uganda.” Journal of the European Economic Association, 3(2-3),
259-67.
Snyder JM and D. Stromberg (2010). ”Press coverage and political accountability.” Journal of
Political Economy 118(2):355–408
Wolfinger, R. E., Rosenstone, S. J., & McIntosh, R. A. (1981). Presidential and congressional
voters compared. American Politics Research, 9(2), 245-256.
34
50.
Figure 1.1: Turnout rate among registered voters from 1995-2006
35
51.
Figure 1.2. Incumbent reelection rates by odd election years since accountability reform
36
52.
Table 1.2: Difference Estimates of the Effect of Public Schools Accountability Act on Incumbent
Reelection Rates During Odd Election Years
DD
(1) (2) (3) (4)
Reelection Rates 0.042 0.050 0.054 0.036
(0.040) (0.034) (0.035) (0.043)
First Difference pre‐post pre‐post pre‐post pre‐post
Second Difference low‐high API low‐high API low‐high API low‐high API
excludes 2005
Year Fixed Effects Y Y Y Y
Controls N Y Y Y
District Fixed Effects N N Y Y
Number of districts 297 297 297 297
N 1,782 1,782 1,782 1,782
Note: The dependent variable is the district's incumbent reelection rates for a particular district. The years
included are odd years 1995‐2005. The affected years are 1999‐2005. The demographic controls for students
receiving free/reduced price lunch for school, a proxy for poverty levels, and percent Hispanic and black. The N
represents district‐year observations. Bertrand et al. (2004) serially correlated corrected standard‐errors are
reported. The standard errors are clustered at the district‐level and bootstrapped 400 times. DD = difference in
difference; pre‐post =post‐accountability reform ‐ pre accountability reform. API, the Academic Performance
Index, is a composite measure of test scores * indicates significance at the 10% level, ** the 5% level, and *** the
1% level.
37
53.
Figure 1.3. Incumbent reelection rates by even election years since accountability reform
38
54.
Table 1.3: Difference Estimates of the Effect of Public Schools Accountability Act on Incumbent
Reelection Rates in Even Years
Levels Gains
DD DDD DD DDD
(1) (3) (3) (4)
Reelection Rates ‐0.073** ‐0.134** ‐0.051 ‐0.123**
(0.034) (0.058) (0.035) (0.054)
First Difference pre‐post pre‐post pre‐post pre‐post
Second Difference low‐high API low‐high API low‐high API low‐high API
even‐odd year even‐odd year
Third Difference
election election
Year Fixed Effects Y Y Y Y
Demographic Controls Y Y Y Y
District Fixed Effects Y Y Y Y
Number of districts 452 749 452 749
N 2,712 4,494 2,712 4,494
Note: The dependent variable is the district's incumbent reelection rates for a particular district. The years
included are 1995‐2006. The affected years are 1999‐2006. The demographic controls for students receiving
free/reduced price lunch for school, a proxy for poverty levels and percent Hispanic and black. The N represents
district‐year observations. Bertrand et al. (2004) serially correlated corrected standard‐errors are reported. The
standard errors are clustered at the district‐level and bootstrapped 400 times. DD = difference in difference; DDD
= triple difference; pre‐post =post‐accountability reform ‐ pre accountability reform; API, the Academic
Performance Index, is a composite measure test scores. * indicates significance at the 10% level, ** the 5% level,
and *** the 1% level.
39
+1-315-636-4485
+91 98151-74050 
