Appendix
Additional State Examples
The appendix lists state examples pertaining to each of the above principles.
1. Agree Upon and Codify Standards and Terms Used to Describe Evidence
Minnesota Management and Budget defines process, outcome and impact evaluations and terms used to describe a program or service’s impact on outcomes, such as “proven effective” or “promising.”
In 2019, Mississippi lawmakers passed legislation that updated statutory definitions of terms, including “evidence-based,” “research-based,” “promising” and other types of programs. The law defines an evidence-based program as an intervention program with multiple-site randomized controlled trials across diverse populations and promising programs as those that have had at least one rigorous controlled evaluation. The law requires the state’s corrections, health, education and transportation agencies, and others determined by the Legislative Budget Office, to catalogue and categorize all funded programs based on these standards.
North Carolina’s Appropriations Act of 2018 instructed the Office of State Budget and Management to define tiered levels of evidence—ranging from services or practices that are “proven effective” or “promising” down to those that are “proven harmful.” The law also calls for an evidence hierarchy to describe the relative strength of different types of evidence.
Florida, Oregon and Utah have established basic definitions of evidence, spelling out what constitutes evidence.
Nebraska, Minnesota, North Carolina and Texas have established tiered definitions of evidence (i.e., an advanced definition that distinguishes multiple levels of rigor, such as “proven effective” and “promising”).
2. Build Consensus Across Branches of Government
Alabama lawmakers passed legislation in 2019 to create a Commission on the Evaluation of Services. Co-chaired by executive and legislative leaders, the commission evaluates how effective state services are and advises the legislature and the governor on program evaluation and how to allocate resources.
In lieu of legislation, Colorado’s Joint Budget Committee and the Governor’s Office of State Planning and Budgeting produced a joint memo outlining the definitions and process for evidence-based policymaking, allowing for greater flexibility to quickly enhance what works and change what does not.
In 2019, New Mexico lawmakers passed SB 58, which requires the state budget division and the legislative finance committee to jointly develop an annual inventory of programs and services.
3. Commit Resources and Staff Capacity to Generating and Using Quality Data and Research
As a nonpartisan fiscal and policy advisor to the legislature, California’s Legislative Analyst’s Office analyzes the annual governor’s budget, reviews program and departmental proposals, and prepares reports on topics of interest to the legislature. One example is “Improving In-Prison Rehabilitation Programs,” a 2017 report that recommended evidence-based rehabilitation programs and regular evaluations to ensure that the programs are implemented with fidelity.
Colorado lawmakers established and funded the Evidence-Based Practices Implementation for Capacity Resource Center in 2013 to support the use of evidence-based practices among agencies that serve juveniles and adults involved in the justice system. In addition, the legislature appropriates $500,000 annually to the Office of State Planning and Budgeting to fund multi-year evaluations of any program.
The District of Columbia funded The Lab @ DC, a dedicated scientific team inside the Office of the City Administrator to design and evaluate evidence-based policy and program interventions. The team recently partnered with the Metropolitan Police District to study the effects of police officer body-worn cameras on police and community interactions.
Minnesota and New Mexico have dedicated evaluation teams linked to their budget offices.
Mississippi lawmakers in 1973 established the Joint Legislative Committee on Performance Evaluation and Expenditure Review, or PEER, to conduct performance evaluation and expenditure reviews. Guided by a mission of improving “the economy, efficiency, effectiveness, and accountability of state government through its reviews of state agency programs and issues,” the nonpartisan standing committee analyzes state agency programs and provides timely and accurate information to enable legislative oversight. In 2015, PEER added a Performance Accountability Office to support the legislature’s performance budgeting revitalization initiative, including implementation of the Pew-MacArthur Results First Initiative.
Pursuant to New Hampshire law that requires certain state-funded services to be delivered as intended, or with fidelity to their original models, the Department of Health and Human Services conducts fidelity reviews to make sure programs are delivered as intended.
4. Direct Resources to Programs, Policies and Practices That Are Backed By Research—and Encourage Promising Ones to Build a Research Base
As part of its partnership with the Pew Results First Initiative, the Iowa Department of Corrections (DOC) inventoried current programs, collected data on the evidence of effectiveness, and used research studies to categorize programs based on their likelihood of reducing recidivism. As a result, the DOC shifted resources to proven programs that support the department’s goals.
Colorado’s evidence continuum (see figure on previous page) provides a framework for describing the evidence supporting a program currently, and how a program can move along the continuum with evaluation and implementation support.
Mississippi’s Joint Budget Committee requires state agencies to summarize research and specify whether the new program or activity is evidence-based, research-based, a promising program or none of the above. If a proposal lacks research, the agency describes its plan for evaluating a pilot program.
New Mexico’s SB 58 requires agencies to specify how much of the funds they request in their budgets will be spent on evidence-based programs. The law also requires state agencies to prioritize evidence-based programs and helps lawmakers decide whether to shift dollars to programs and services that are more effective than others not validated by research.
In 2003, Oregon lawmakers passed legislation requiring five state agencies to gradually increase funding for evidence-based programs from 25% in 2007 to 75% in 2011 and beyond.
5. Embed Evidence Into State Budgeting Processes and Decisions
Beginning in 2016, Colorado included evidence requirements in budget instructions for state agencies. When applying for new or expanded programs or services, agencies must summarize the available research and expected effects on outcomes, the expected return on investment for the program or service, and the agency’s evaluation plan.
In 2018, Minnesota Management and Budget introduced evidence-based budget proposal forms. Agencies use the forms to highlight the evidence base for proposals that they want categorized as “evidence-based” during the budget review process.
State legislators, in consultation with staff of Mississippi’s Joint Legislative Committee on Performance Evaluation and Expenditure Review, developed “Seven Elements of Quality Program Design.” The process ensures that new funding requests are supported by existing research demonstrating a program’s effectiveness or a plan for rigorous research of a pilot program. The framework offers questions policymakers can ask regarding the program’s research base and implementation plan. For the fiscal year ending in June 2016, the Legislative Budget Office began requiring agencies seeking new program funding to complete the seven elements as part of their annual budget request. A copy of the seven elements is included in the Legislative Budget Office’s annual budget instructions.
In 2003, New Mexico’s Legislative Finance Committee (LFC) developed a Legislating for Results framework to use research and performance data to inform budget decisions. Analysts review appropriation requests and use the framework to prioritize programs with evidence of effectiveness. Noting that committee hearings offer legislators a “key opportunity to use performance data and evidence to build a budget and inform policy development,” LFC lists questions that lawmakers may ask agency staff. They can ask, for example, how much of their budget request will be used to implement or expand evidence-based interventions.
Tennessee’s Department of Finance and Administration requires agencies to tie new budget requests to policy objectives, such as investing in programs and initiatives that are supported by evidence and research.
Utah’s Office of Management and Budget requires state agencies to submit a business case form, which describes what problem the new funding will solve, how the new funding will be used, the expected results or outcomes, and which measures will be used to track the changes over time.
6. Foster a Culture of Continued Learning
In California, the Legislative Analyst’s Office advises the legislature on evidence-informed policymaking and is developing a training curriculum for legislators and legislative staff.
The Result First Initiative collaborated with its in-state partners in Illinois, Minnesota and North Carolina to deliver evidence trainings to agency staff for both educational purposes and to leverage the content and engagement to further the states’ goals and build on evidence-based policymaking progress to date.
To increase the use of science to inform policy decisions, California and New Jersey have established legislative science fellow programs. The New Jersey Legislature in 2019 funded the Eagleton Science and Politics Fellowship program. Run by the Eagleton Institute of Politics at Rutgers University, New Brunswick, the program appointed four Ph.D. scientists and engineers to work full time for one year in the executive or legislative branch, where they serve as science advisors to agency leaders and legislators.
7. Garner Support Through Clear Communication and Messaging
The District of Columbia registered a pre-analysis plan for evaluating its police body-worn camera program and presented the plan for feedback across a diverse set of public meetings and with an interactive website designed for lay audiences.
Minnesota Management and Budget’s inventory of programs assigns color-coded ratings of effectiveness for over 400 publicly funded programs and services. Users can filter data by area of interest (e.g., criminal justice or early childhood), outcome measured, population and rating type (e.g., proven effective, promising and theory-based).
New Mexico’s Legislative Finance Committee created a dashboard report and agency report cards to synthesize performance data and facilitate focused discussions on evidence-based initiatives.