Originally published on Brooking Institution's Brown Center Chalkboard, February 13, 2013
The question in the title should be easy to answer. Everyone knows that policies are debated and adopted by a legislative body (Congress, state legislature, or city council), signed by an executive leader (president, governor, or mayor), and then implemented by an administrative unit (federal, state, or city department). Once implemented, a policy starts.
The purpose of this Chalkboard post is to convince you that the easy answer can be misleading. A second purpose is to persuade you that the establishment of a starting point can affect the judgment of whether a policy has been good or bad–or has had no effect at all.
Let’s begin by considering a famous example, No Child Left Behind (NCLB). The basic timeline is this: NCLB was proposed by President Bush in January 2001, debated in Congress that year, passed in December 2001, and signed into law in January 2002. Most state accountability plans were not approved by the U.S. Department of Education until 2003.
Take a look at NAEP (National Assessment of Educational Progress) data for fourth grade reading. (Please play along by setting aside the daunting problems with drawing causal inferences from NAEP).
| Year | Score |
|---|---|
| 2011 | 221 |
| 2003 | 218 |
| 2002 | 219 |
| 2000 | 213 |
| 2000^ | 217 |
| 1992^ | 217 |
^ No accommodations permitted.
Notice that scores jumped from 2000 to 2002, an increase of 6 points in only 2 years. Even larger gains were registered from 2000 to 2003 on NAEP’s 4th grade math test (9 points) and similar gains on the 8th grade math test (5 points).
Not surprisingly, critics and supporters of NCLB have wrestled over the 2000-2003 time interval like troops battling over a strategically important hilltop. Critics start the clock on NCLB in 2003, a year after the law was signed. With 2003 established as a post-intervention baseline, one can see that the gains immediately before NCLB were greater than the gains after adoption. Progress slowed after NCLB started.
Not so fast, say supporters of NCLB. It’s also clear from the NAEP scores that schools made little headway in the 1990s. How about using an earlier baseline—before 2003? Holding schools accountable for student performance, the key component of NCLB, started in the states before 2003. In 1999, Education Week’s annual publication, Quality Counts, was on the theme, “Rewarding Results, Punishing Failure.” It reported that 48 states were testing students, 36 issued annual report cards on schools, and 19 rated school performance and identified poorly performing schools. In 2000, both George W. Bush and Al Gore campaigned on promises of tough school accountability. Bush’s plan was to elevate the Texas school accountability program to all states. That became NCLB. People forget that Gore’s plan gave poorly performing schools only two years to improve; if they didn’t, they would be shut down, converted to charter schools, or reconstituted with a new principal and teaching staff.
Accountability was a hot topic in 2000 and figured prominently in state and local reforms. In 2001, as Congress debated NCLB, more states passed test-based accountability programs and by the time NCLB was signed into law in 2002, at least half of the states had some kind of test-based accountability in place. In the summer of 2002, states identified 8,652 schools in need of improvement under NCLB’s provisions. None of this was secret.
A standard rule of interrupted time series analysis (the formal name for before-and-after comparisons) is that the time of the policy intervention should be precise. To use the 2003 NAEP as a baseline for post-intervention assumes that schools were unaffected by the 1999-2002 activities described above when the NAEP was administered in early 2003. That’s unlikely.
Serious policy analysts are aware of the problem and in their evaluations of NCLB have addressed it in a variety of ways. Thomas Dee and Brian Jacob compare states that already had NCLB-style accountability systems in 2002 with states that were adopting accountability for the first time under NCLB. They define the treatment as the number of years states did not have accountability systems between 1992 and 2002, effectively putting each state on its own clock (see here for the study and also here for an alternative approach less favorable to NCLB). Dee and Jacob find a significantly positive effect for 4th grade math (effect size of 0.23), a small positive effect for 8th grade math, and no effect for either 4th or 8th grade reading. The math finding holds up using several different NCLB starting points.
Let’s not linger over statistical techniques or get side-tracked by NCLB. The point is that a counter-intuitive idea—that policy effects can occur before a policy is adopted—must be taken seriously.
Consider anticipatory effects, changes in behavior in anticipation of a policy. If you’ve been following the national news lately, two vivid examples should come to mind. Sales of firearms have surged in the wake of several horrifying mass shootings. Gun owners fear that stricter gun control laws are on the horizon so they are buying more firearms. In December, 2012, corporations paid out a whopping $30.9 billion in special dividends, shifting payouts into 2012 to avoid the scheduled expiration of the Bush tax cuts and the potential for sharply higher rates in 2013. Corporations would rather reward shareholders earlier than risk more highly taxed dividends later.
Note that these are observable changes in behavior before policies have been passed, let alone implemented. Indeed, the policies inducing these behaviors may never be passed. This makes careful estimates of policy effects difficult. If later this year Congress passes stricter gun control measures and gun sales drop, will it be because of the passage of the law? Or because gun buying had been shifted ahead in time to avoid new restrictions?
Federalism is another factor to take into account. In education, we have a multi-leveled system of federal, state, district, and school policies. When the federal government adopts an education policy you can almost be sure that it is already being done somewhere in some fashion. As noted above, the core of NCLB—accountability—was already in place in many states when NCLB was signed into law.
The impact of federalism is well known in other policy domains, and policy analysts in those fields incorporate state-level sources of variation into their work. For example, the National Minimum Drinking Age Act of 1984 prohibited the purchase or public possession of alcoholic beverages by anyone under the age of 21. Federal highway funds were used to persuade states to go along with the wishes of Congress on what historically had been strictly a state concern. Many states already had a 21-year age limit (affecting the pre-policy period). For those that allowed purchases at an earlier age, the change was more difficult, and they dragged their feet implementing the law (affecting the post-policy period).
A similar phenomenon occurred with 1974 federal legislation establishing a national speed limit of 55 mph. Some states already had a 55 mph limit–a few even had 50 mph—but most had to lower their speed limits to gain compliance. Western rural states reluctantly adopted the limit, and in an obvious ploy to subvert the law’s intent, also adopted $5 fines for speeding. When did the national speed limit law truly start? It depends on the state. We are more certain of when it ended—in 1995 when Congress repealed the law and gave authority over speed limits back to the states.
So now consider what all of this means for the Common Core. Currently, forty-five states and the District of Columbia have adopted the Common Core and school districts have begun implementing them. Kentucky recently gave a test which state officials claim is faithful to the Common Core standards. Textbook publishers declare that their latest series are Common Core compatible. Curriculum guides have been written and teachers all over the country are attending professional development sessions. This is all in the name of the Common Core.
It sure seems like the Common Core has started. But important pieces are missing. The federally funded state consortia who are writing Common Core assessments have not completed their task. Cut points for student proficiency haven’t been set. No tests have been administered (other than Kentucky’s). No consequences based on performance will be implemented until 2014-2015 at the earliest.
Sometime down the road, someone will declare the Common Core a success or failure, and that judgment will probably be based, as NCLB evaluations have been based, on a before and after analysis using national data. Some will say the Common Core started in 2010, when the states adopted the standards. Others will say 2011 or 2012, when state and local curriculum changes began. And still others will declare 2015 as the Common Core’s start date, after the first assessments have been conducted.
What is the best answer? Not only is the best answer unclear—despite the fact that we are nearer to the starting point today than we will be in another decade—but it’s also highly probable that future advocates, both for and against the Common Core, will exploit that lack of clarity to focus on the time period most pleasing to them. Analysts should be thinking about Common Core’s possible starting points right now.