The ABCs of Evidence-Informed Policymaking

Developing common terminology promotes transparency about how government leaders make funding decisions and clarifies what constitutes evidence. Definitions can provide a framework for making budget decisions based on program effectiveness.

States have taken different steps to define terms that reflect strength of evidence—some through legislation and others through less formal approaches. Work group members agreed on the importance of engaging executive, legislative and judicial branch stakeholders to define terms and determine how to use them to inform policy decisions.

Options and Strategies

• Engage cross-branch stakeholders to determine how to develop or refine definitions to inform policy decisions. Assess if, where and how evidence standards are currently described (e.g., in statute, budget instructions or federal grant agreements), whether there is a common terminology, and whether existing definitions provide meaningful information that help policymakers make evidence-informed decisions.
• Consider adopting or modifying definitions of evidence from national clearinghouse resources or from other states’ definitions.
• Determine how to use evidence standards to inform policy and budget decisions and whether they will be used broadly across policy areas, or specifically for certain policies and programs.

Prioritize high-quality and objective research resources to analyze program effectiveness, such as the Results First Clearinghouse Database, or others specific to policy areas, such as the Title IV-E Prevention Services Clearinghouse for child welfare.

Effectively implementing and sustaining evidence-informed practices and policies over time requires collaboration among legislators and legislative, judicial and executive branch staff. Cross-branch partnerships can facilitate a shared understanding of what evidence-informed policymaking is and how public sector stakeholders can use evidence to inform policy decisions.

Partnerships can also help stakeholders set priorities and identify the results they want to achieve, such as lower recidivism or increased employment. Once priorities have been established, stakeholders can identify programs and policies with evidence of positive impact.

Options and Strategies

• To facilitate buy-in among legislative and executive branch stakeholders, engage partners early and often in decision-making processes. Consider training government staff on evidence-informed policymaking, including new processes for preparing budgets or reporting on program outcomes.
• Engage with stakeholders to identify priority areas for evidence-based reforms. Some states focus on one or more key policy areas with a strong research base, such as criminal justice reforms to reduce crime at lower costs—and gradually expand their approach to other policy areas.

Policymakers need access to reliable and meaningful data and program evaluations to make the best use of public resources. States have taken steps to build the research base for existing programs and develop staff and policymaker capacity to use the research to make decisions. Collecting and using data and evidence to inform policy decisions is an iterative process that benefits from leadership support and dedicated staff resources.

States have improved access to reliable data, shared data across state agencies, and increased staff capacity to analyze and distill meaningful data for policymakers. Monitoring and reporting on program outcomes helps policymakers determine whether programs are achieving their expected results (discussed in more detail on page 7).

Options and Strategies

• Invest in and support access to high-quality research and data.
• Build state government’s capacity to use research evidence in policy and budget decisions. A centralized and nonpartisan staff or government unit with knowledge about research methods and evidence-based budgeting can help support evidence-based reforms.
• Partner with research universities or research-based organizations to analyze and leverage existing state data.
• Review available evidence on the effectiveness of public programs and share that information with decision-makers and the public.
• Encourage or require state agencies to monitor programs to ensure they are being implemented as intended—or with fidelity—and therefore more likely to achieve their intended outcomes.
• Offer incentives or require agencies to set aside funding for program evaluation.

States are increasingly investing in programs and services that have been proven effective by rigorous research. Such investments should produce better outcomes and are a more efficient use of public resources. States also recognize the importance of promoting innovative or new approaches that—while untested—could address specific needs. These approaches may be necessary when research is incomplete or when existing programs do not address the contexts of all populations.

Many states have taken steps to fund promising programs that address state needs but lack rigorous research, as well as to provide support for evaluating such programs. For example, they have developed tiered grant programs, an evidence continuum or other frameworks that give preference to programs with strong evidence while also providing an opportunity for new or untested programs to develop research that demonstrates their results.

Options and Strategies

• Require or encourage state agencies to develop an inventory of funded programs, categorize them based on their evidence of effectiveness, and use this information when preparing budgets or making funding decisions.
• Policymakers and staff across government can use evidence-based clearinghouses, such as the Results First Clearinghouse Database, to determine a program’s evidence of effectiveness. If programs don’t yet have an evidence base, agencies could explain why a program is promising and expected to bring about change.
• Collaborate with university and other researchers when developing and evaluating new programs, and when evaluating more mature programs that lack evidence.
• Use a tiered funding approach to provide incentives to promote the use of evidence-based programs by state agencies and local grantees. Tiered funding means providing the greatest level of funding to programs that meet the highest tier of evidence and a smaller portion of funds for untested programs.
• Prioritize funding for cost-effective programs and services. Consider dedicating a portion of an agency’s programming dollars to evidence-based or evidence-informed programs, gradually increasing funding levels over time.
• Create innovation funds or provisions to fund promising programs and pilot projects and support research and evaluation to demonstrate their results.

Several states have integrated performance information and evidence of program effectiveness into the budget process. This provides an opportunity for agencies to share critical information with policymakers about a program’s effectiveness and to improve procedures for prioritizing funds.

States incorporate evidence into the budget development process broadly across state government or for specific policy areas with strong evidence of effectiveness, including criminal justice, child welfare and mental health. By embedding practices into state budgeting processes, states are working to sustain evidence-informed reforms and change the culture of how decisions are made.

Options and Strategies

• Determine how broadly to apply evidence-informed budget procedures and which programs or services to include.
• Estimate programs’ anticipated return on investment. By comparing the costs of public programs to their benefits, state policymakers can allocate resources more effectively.
• Develop evidence guidelines or budget directives that embed research and data in funding decisions. Require an executive or legislative budget office or other government unit to ensure that agency appropriation requests meet evidence standards. Require or encourage agencies to submit evidence-based budget forms that show how a proposed program can deliver proven outcomes.
• Ask key questions during committee hearings to understand the research base supporting a funding request, a program’s expected outcomes and the plan for evaluating its effectiveness.

Evidence training for legislative and executive branch stakeholders will help buffer against staff changes or legislator term limits while simultaneously building a deeper bench of support for an evidence-informed approach. Such trainings could discuss what evidence is, where to find it, what questions to ask to assess it, and how to use it in the budgeting and policymaking process. Legislative and executive branch staff in some states have developed and delivered such training modules or consultation services for specific audiences—including agency leaders, legislators, legislative staff and external stakeholders, such as community service providers.

Additionally, policymakers rely on performance tracking and reporting to assess whether funded programs have achieved their desired results and to identify opportunities for improvement. A 2018 report published by The Pew Charitable Trusts recommends several strategies for strengthening performance management systems, such as analyzing and reporting targeted performance information and creating opportunities to make better use of performance data (e.g., through regular meetings between leadership and agencies).

Options and Strategies

• Determine where research expertise resides in legislative, executive and state agencies—and where training or capacity-building is needed.
• Building on staff research expertise, provide ongoing training and technical assistance to other staff and prepare agency leaders and budget analysts to implement budget practices that consider evidence.
• Consider opportunities to partner with external stakeholders—such as research universities, nonpartisan and philanthropic organizations, and subject matter experts—to convene and train legislative and executive staff on the state of the evidence in areas where leaders will be making decisions.
• Consider opportunities to deliver cross-branch training and technical assistance to promote collaboration and facilitate implementation.
• Examine opportunities to improve collection, reporting and policymakers’ use of performance data. For a more complete picture of how public programs are performing and where resources are needed most, the 2018 report by The Pew Charitable Trusts recommends aligning state performance improvement and reporting efforts from state evaluation and audit offices, state performance reporting systems and other sources.
• Learn from other states’ experiences with implementing evidence-informed practices by participating in convenings, joining peer learning networks or reading case studies on evidence-informed policymaking from NCSL, Results First and other nonpartisan entities.

Using evidence to inform policy decisions can help policymakers invest wisely and achieve meaningful results. To reach its full potential, the benefits—and limitations—of an evidence-informed approach must be communicated clearly to policymakers, the public and other key constituencies.

It is also important that research findings are accessible and meaningful to policymakers and address key policy objectives. Recognizing that a variety of factors influence policy decisions, work group members support steps that ensure that evidence has a “seat at the table” in those decisions.

Options and Strategies

• Consult other states’ approaches for conveying research findings. Identify data visualization and other communication strategies that capture research findings clearly and accurately.
• Assess whether research findings are presented and translated in a concise and actionable way. Convene stakeholders so policymakers can convey what questions they want answered and what information they need to make decisions.
• To guard against skepticism about the research process or claims that researchers may alter their methods to achieve publishable findings, or what’s known as publication bias, policymakers can engage with researchers to understand whether they have a predetermined research plan. Policymakers can also ask researchers if they employ “open science” best practices. That is, have they publicly committed, in advance, to the methods of measurement to be used and which results will be released?

Legislators and executive branch policymakers from leading states refer to the reform process as a marathon, not a sprint, pointing out that changing the way states fund and evaluate programs takes time. Moreover, the state experiences profiled above reinforce that the pursuit of evidence-informed policymaking varies considerably across states and is not an all-or-nothing prospect. Evidence-informed policymaking can help stakeholders across government allocate resources to programs that are effective, promote innovation, and build and sustain a culture of continuous learning and improvement. The seven principles outlined in this report offer options for states that are striving to advance the use of evidence in state policymaking and improve the culture of how policy decisions are made.

Additional State Examples

The appendix lists state examples pertaining to each of the above principles.

1. Agree Upon and Codify Standards and Terms Used to Describe Evidence

Minnesota Management and Budget defines process, outcome and impact evaluations and terms used to describe a program or service’s impact on outcomes, such as “proven effective” or “promising.”

In 2019, Mississippi lawmakers passed legislation that updated statutory definitions of terms, including “evidence-based,” “research-based,” “promising” and other types of programs. The law defines an evidence-based program as an intervention program with multiple-site randomized controlled trials across diverse populations and promising programs as those that have had at least one rigorous controlled evaluation. The law requires the state’s corrections, health, education and transportation agencies, and others determined by the Legislative Budget Office, to catalogue and categorize all funded programs based on these standards.

North Carolina’s Appropriations Act of 2018 instructed the Office of State Budget and Management to define tiered levels of evidence—ranging from services or practices that are “proven effective” or “promising” down to those that are “proven harmful.” The law also calls for an evidence hierarchy to describe the relative strength of different types of evidence.

Florida, Oregon and Utah have established basic definitions of evidence, spelling out what constitutes evidence.

Nebraska, Minnesota, North Carolina and Texas have established tiered definitions of evidence (i.e., an advanced definition that distinguishes multiple levels of rigor, such as “proven effective” and “promising”).

2. Build Consensus Across Branches of Government

Alabama lawmakers passed legislation in 2019 to create a Commission on the Evaluation of Services. Co-chaired by executive and legislative leaders, the commission evaluates how effective state services are and advises the legislature and the governor on program evaluation and how to allocate resources.

In lieu of legislation, Colorado’s Joint Budget Committee and the Governor’s Office of State Planning and Budgeting produced a joint memo outlining the definitions and process for evidence-based policymaking, allowing for greater flexibility to quickly enhance what works and change what does not.

In 2019, New Mexico lawmakers passed SB 58, which requires the state budget division and the legislative finance committee to jointly develop an annual inventory of programs and services.

3. Commit Resources and Staff Capacity to Generating and Using Quality Data and Research

As a nonpartisan fiscal and policy advisor to the legislature, California’s Legislative Analyst’s Office analyzes the annual governor’s budget, reviews program and departmental proposals, and prepares reports on topics of interest to the legislature. One example is “Improving In-Prison Rehabilitation Programs,” a 2017 report that recommended evidence-based rehabilitation programs and regular evaluations to ensure that the programs are implemented with fidelity.

Colorado lawmakers established and funded the Evidence-Based Practices Implementation for Capacity Resource Center in 2013 to support the use of evidence-based practices among agencies that serve juveniles and adults involved in the justice system. In addition, the legislature appropriates $500,000 annually to the Office of State Planning and Budgeting to fund multi-year evaluations of any program.

The District of Columbia funded The Lab @ DC, a dedicated scientific team inside the Office of the City Administrator to design and evaluate evidence-based policy and program interventions. The team recently partnered with the Metropolitan Police District to study the effects of police officer body-worn cameras on police and community interactions. 

Minnesota and New Mexico have dedicated evaluation teams linked to their budget offices.

Mississippi lawmakers in 1973 established the Joint Legislative Committee on Performance Evaluation and Expenditure Review, or PEER, to conduct performance evaluation and expenditure reviews. Guided by a mission of improving “the economy, efficiency, effectiveness, and accountability of state government through its reviews of state agency programs and issues,” the nonpartisan standing committee analyzes state agency programs and provides timely and accurate information to enable legislative oversight. In 2015, PEER added a Performance Accountability Office to support the legislature’s performance budgeting revitalization initiative, including implementation of the Pew-MacArthur Results First Initiative.

Pursuant to New Hampshire law that requires certain state-funded services to be delivered as intended, or with fidelity to their original models, the Department of Health and Human Services conducts fidelity reviews to make sure programs are delivered as intended.

4. Direct Resources to Programs, Policies and Practices That Are Backed By Research—and Encourage Promising Ones to Build a Research Base

As part of its partnership with the Pew Results First Initiative, the Iowa Department of Corrections (DOC) inventoried current programs, collected data on the evidence of effectiveness, and used research studies to categorize programs based on their likelihood of reducing recidivism. As a result, the DOC shifted resources to proven programs that support the department’s goals.

Colorado’s evidence continuum (see figure on previous page) provides a framework for describing the evidence supporting a program currently, and how a program can move along the continuum with evaluation and implementation support.

Mississippi’s Joint Budget Committee requires state agencies to summarize research and specify whether the new program or activity is evidence-based, research-based, a promising program or none of the above. If a proposal lacks research, the agency describes its plan for evaluating a pilot program.

New Mexico’s SB 58 requires agencies to specify how much of the funds they request in their budgets will be spent on evidence-based programs. The law also requires state agencies to prioritize evidence-based programs and helps lawmakers decide whether to shift dollars to programs and services that are more effective than others not validated by research.

In 2003, Oregon lawmakers passed legislation requiring five state agencies to gradually increase funding for evidence-based programs from 25% in 2007 to 75% in 2011 and beyond.

5. Embed Evidence Into State Budgeting Processes and Decisions

Beginning in 2016, Colorado included evidence requirements in budget instructions for state agencies. When applying for new or expanded programs or services, agencies must summarize the available research and expected effects on outcomes, the expected return on investment for the program or service, and the agency’s evaluation plan.

In 2018, Minnesota Management and Budget introduced evidence-based budget proposal forms. Agencies use the forms to highlight the evidence base for proposals that they want categorized as “evidence-based” during the budget review process.

State legislators, in consultation with staff of Mississippi’s Joint Legislative Committee on Performance Evaluation and Expenditure Review, developed “Seven Elements of Quality Program Design.” The process ensures that new funding requests are supported by existing research demonstrating a program’s effectiveness or a plan for rigorous research of a pilot program. The framework offers questions policymakers can ask regarding the program’s research base and implementation plan. For the fiscal year ending in June 2016, the Legislative Budget Office began requiring agencies seeking new program funding to complete the seven elements as part of their annual budget request. A copy of the seven elements is included in the Legislative Budget Office’s annual budget instructions.

In 2003, New Mexico’s Legislative Finance Committee (LFC) developed a Legislating for Results framework to use research and performance data to inform budget decisions. Analysts review appropriation requests and use the framework to prioritize programs with evidence of effectiveness. Noting that committee hearings offer legislators a “key opportunity to use performance data and evidence to build a budget and inform policy development,” LFC lists questions that lawmakers may ask agency staff. They can ask, for example, how much of their budget request will be used to implement or expand evidence-based interventions.

Tennessee’s Department of Finance and Administration requires agencies to tie new budget requests to policy objectives, such as investing in programs and initiatives that are supported by evidence and research.

Utah’s Office of Management and Budget requires state agencies to submit a business case form, which describes what problem the new funding will solve, how the new funding will be used, the expected results or outcomes, and which measures will be used to track the changes over time.

6. Foster a Culture of Continued Learning

In California, the Legislative Analyst’s Office advises the legislature on evidence-informed policymaking and is developing a training curriculum for legislators and legislative staff.

The Result First Initiative collaborated with its in-state partners in Illinois, Minnesota and North Carolina to deliver evidence trainings to agency staff for both educational purposes and to leverage the content and engagement to further the states’ goals and build on evidence-based policymaking progress to date. 

To increase the use of science to inform policy decisions, California and New Jersey have established legislative science fellow programs. The New Jersey Legislature in 2019 funded the Eagleton Science and Politics Fellowship program. Run by the Eagleton Institute of Politics at Rutgers University, New Brunswick, the program appointed four Ph.D. scientists and engineers to work full time for one year in the executive or legislative branch, where they serve as science advisors to agency leaders and legislators. 

7. Garner Support Through Clear Communication and Messaging

The District of Columbia registered a pre-analysis plan for evaluating its police body-worn camera program and presented the plan for feedback across a diverse set of public meetings and with an interactive website designed for lay audiences. 

Minnesota Management and Budget’s inventory of programs assigns color-coded ratings of effectiveness for over 400 publicly funded programs and services. Users can filter data by area of interest (e.g., criminal justice or early childhood), outcome measured, population and rating type (e.g., proven effective, promising and theory-based).

New Mexico’s Legislative Finance Committee created a dashboard report and agency report cards to synthesize performance data and facilitate focused discussions on evidence-based initiatives.

The NCSL Evidence-Informed Policymaking Work Group was staffed and this report was prepared by Kristine Goodwin, program director, and Luke Martel, group director, for NCSL’s Employment, Labor and Retirement Program. Other NCSL staff contributors included Kate Blackman and Iris Hentze, who reviewed and offered valuable insights on various drafts of this report, Linnette Vasquez, who provided administrative and meeting support, and Jane Andrade and Steve Miller, who edited and designed the report.

This report was prepared under the direction of the NCSL Evidence-Informed Policymaking Work Group, which was made possible through generous support from The Pew Charitable Trusts. Their continued support and assistance to NCSL and state legislatures are gratefully acknowledged.

NCSL first convened the work group in late 2019 when the group began a months-long process of identifying and refining key principles that help to support evidence-informed policymaking. The bipartisan, crossbranch member group includes state legislators, senior legislative and executive branch staff, researchers and NCSL leaders who are dedicated to advancing the use of evidence in their states. With support from The Pew Charitable Trusts, the group has met virtually in 2020 to refine these principles and advise NCSL on its new Center for Results-Based Governing.

NCSL extends deep appreciation to each work group member for reviewing drafts and lending their time and insights as this report took shape—from the first brainstorming discussions in late 2019 to the final review of this publication. We are grateful to them for dedicating their energy and time, especially during the difficult early months of 2020, when the onset and fallout from COVID-19 demanded so much of both.

Through their engagement and dedication, we learned what has worked in multiple states to successfully implement an evidence-informed approach, and which factors have paved the way for states to use quality data and research to inform their decisions. It is the intent of NCSL and this work group that the principles and examples presented here will help guide and inform results-based governing now and well into the future.

Evidence-Informed Policymaking Work Group Members