The role of data- and research-driven decision-making is particularly crucial in times of lower revenues and budget deficits, such as those states are facing as a result of the global coronavirus pandemic. These circumstances make it even more critical for policymakers to use reliable information and target limited resources to the most effective programs and policies. As policymakers and their staff prepare for upcoming budget cycles, data and evidence will be vital for setting priorities. Evidence and data can “help states get back on their feet faster,” Oregon Senator Elizabeth Steiner Hayward told NCSL’s evidence-informed policymaking work group last summer. By understanding what works and how to use evidence to inform state policy and budget decisions, Steiner Hayward said, “we can figure out which are the best policies to spend money on to get our economies going.”
Rather than making indiscriminate, across-the-board cuts to government programs, legislators in many states have used evidence-based policymaking to make strategic and results-driven budget decisions that are tailored to their states’ needs. To support data-driven decision-making at the state and local levels, organizations such as the Pew Results First initiative have created tools for jurisdictions to incorporate data into the budgeting process. NCSL recently highlighted state strategies in its 2020 “ABCs of Evidence-Informed Policymaking” report. Over the past decade, governments have seen a rise in the use of such tools, and many best practices for evidence-based policymaking have emerged.
This brief outlines six key questions legislators and legislative staff can ask to assess how data is being collected and used to inform budget decisions. By asking these questions, and others proposed in the table below, legislators and legislative staff can use data and evidence to make strategic decisions and investments. The strategies outlined below can help states navigate through challenging economic times and will serve them as they rebuild and set priorities in the years to come.
1. Have agencies been instructed to develop budgets that incorporate evidence of effectiveness?
Several states have integrated performance information and evidence of program effectiveness into the budget process. This provides an opportunity for agencies to share critical information with policymakers about a program’s effectiveness and improve procedures for prioritizing funds. To ensure they are investing dollars in programs that are proven to work, many state executive budget offices have modified budget guidelines, instructing agencies to describe evidence of effectiveness as part of their budget requests. State legislators and legislative staff can begin by asking how evidence is integrated into budget requests and how their state’s budget office uses budget guidelines to encourage or require the use of evidence from agencies.
For example, Colorado and Minnesota’s executive budget offices have included evidence requirements in budget instructions for state agencies. In 2020, North Carolina’s Office of State Budget and Management (OSBM) modified budget guidelines to encourage agencies to submit more information about existing data and evidence supporting their requests. The guidelines advise that “requests for new or expanded programs or services should include evidence and research supporting the program’s effect on desired outcomes.” OSBM will evaluate requests based on the strength of this evidence.
2. Does this budget address the most critical problems and unmet needs—and how do we know?
Policymakers operating under budget constraints are especially concerned they are allocating funds to programs and services, such as reemployment or housing assistance, that are most needed in the current economic environment. Legislators and legislative staff can ask agencies to gather information on population needs, report information from evaluations, or determine who is signing up for current programs and why. Policymakers can also rely on academic researchers and non-governmental partners to gather and translate data for policy decisions. In many instances, this data is already being collected.
For example, Utah recently demonstrated how research partnerships can support data-driven policy decisions. With support from The Pew Charitable Trusts, the Kem C. Gardner Policy Institute at the University of Utah analyzed the availability of evidence-based medication treatment programs for opioid use disorder. Its report identified gaps in services and access barriers and offered several steps policymakers could take to help ensure sufficient treatment access for communities statewide.
3. Which programs and services are currently funded?
Several states have required or encouraged agencies to compile comprehensive lists—or inventories—of the programs a state funds in a particular policy area. Inventories typically include information such as the duration or frequency of programs, populations served, and the program cost. Once completed, legislators can use inventories to inform their budget decisions or ask agency officials questions at oversight hearings to better understand what works, for whom, and under what conditions. Inventories are also a valuable tool to help understand gaps in service delivery and can even raise questions about shifting population needs.
For example, in 2015, Minnesota enacted bipartisan legislation requiring the state’s corrections and human services agencies to assess the effectiveness and cost of their programs. Minnesota’s Department of Management and Budget worked with agencies to create program inventories that would help to gather this information.
Through the inventory process, they found that several counties lacked evidence-based programs for reducing recidivism among the adult criminal justice population. Several localities could not meet the demand for these programs. In response, the legislature invested in two state programs to help fill these gaps: Community Offender Supervision and Alternatives to Incarceration. In the current fiscal environment, legislators can prioritize sustained funding for evidence-based programs and focus spending cuts elsewhere in the budget.
4. Are we funding programs that are proven to work?
With limited funds and a pressing need to do more with less, state legislators and legislative staff want to ensure they are investing in programs that have been proven effective in achieving the outcomes that matter most. After identifying unmet needs, state legislators and legislative staff can use research clearinghouses such as the Results First or Social Programs That Work, to identify evidence-based programs that meet these needs. Typically managed by federal agencies or non-governmental organizations, clearinghouses are databases that provide information about a program’s effectiveness and impact on achieving outcomes of interest.
Legislators can also work with executive budget offices to encourage or require agencies to justify requests for funding by citing evidence from clearinghouses. State agencies can use the contracting process in grant programs to incentivize or require local grantees to use evidence-based programs in order to receive funding. Clearinghouses can help these states craft clear contract guidelines and evaluate subsequent applications. Finally, clearinghouses can be used to guide decisions about how to use limited evaluation resources to study programs that may lack a clear evidence base.
For example, in Mississippi, state legislators in both chambers partnered with the state’s Joint Legislative Committee on Performance Evaluation and Expenditure Review to create a screening tool called the Seven Elements of Quality Program Design. This tool helps legislators ensure that any new spending requests are backed by evidence of effectiveness.
5. Are our programs cost-beneficial?
In addition to understanding evidence of program effectiveness, many states also collect information that can help legislators understand a program’s return on investment. An inventory might find that two child welfare programs are effective based on two separate evaluations. While this information is useful, it would be even more helpful to understand if one of these programs was shown to be more cost-effective than the other (i.e., achieving similar benefits for less funding per participant). This information, often referred to as benefit-cost analysis, can help state legislators and their staff understand the long-term benefits of investing in one approach over another.
New Mexico’s Legislative Finance Committee uses benefit-cost information through the Results First Cost-Benefit Model to inform its budget decisions across several policy areas. This model is an online tool that allows state policymakers to estimate the return on investment for various programs within a policy area. The model uses research that shows whether a program achieves its intended outcomes and estimates the monetary benefits of these outcomes. New Mexico’s work with this model has allowed the state to target more than $130 million toward evidence-based programs that are shown to be more cost-effective than their predecessors.
Washington state lawmakers created the Washington State Institute for Public Policy (WSIPP) in 1983 to identify evidence-based policies. The goal for the nonpartisan research agency is to provide policymakers and budget writers with “public policies that can, with a high degree of certainty, lead to better statewide outcomes coupled with a more efficient use of taxpayer dollars.” WSIPP compares the benefits and costs of policy options across a range of research areas, such as adult criminal justice, child welfare, and public health and prevention. Case in point: A 2020 report found that the state’s Extended Foster Care program produced $3.95 of lifetime benefits for every $1 invested. Youth participants were more likely to be employed and have higher earnings than non-participants.
6. How can the legislative and executive branches partner to strengthen evidence-based budgeting?
State legislators hoping to increase their use of evidence-based budgeting practices can work with their executive branch partners to build evidence into the policymaking process over time. These conversations often begin by reflecting on what information would be most valuable for making spending decisions (whether these are budget increases, level funding or informed budget decreases).
By approving budget requests each cycle and providing oversight of executive branch agencies, legislators are in a unique position to identify data gaps and incorporate evidence into future decision-making processes. These conversations can lead to a clear set of priorities for government partners across the state.
Legislatures can also establish clear thresholds for funding decisions that are based on evidence, signaling their desire for the executive branch to invest in evidence-based programs. Funding thresholds can encourage or require the use of evidence-based programs and limit funding for programs that have not been evaluated or shown to be ineffective. These laws can apply to both entire agencies or specific sets of programs within agencies, such as substance use disorder programs. These thresholds then create further incentives for legislators to routinely assess the effectiveness of state programs. For example:
- Alabama lawmakers passed legislation in 2019 to create a Commission on the Evaluation of Services. Co-chaired by executive and legislative leaders, the commission evaluates how effective state services are and advises the legislature and the governor on program evaluation and how to allocate resources.
- In 2019, New Mexico lawmakers passed SB 58, which requires the state budget division and the legislative finance committee to jointly develop an annual inventory of programs and services.
- In 2003, Oregon lawmakers passed legislation requiring five state agencies to gradually increase funding for evidence-based programs from 25% in 2007 to 75% in 2011 and beyond.
Data and evidence in decision-making do not have to be a panacea reserved only for states with a sophisticated infrastructures of practices and supports already in place. Instead, any state can begin to use data and evidence regardless of where it is starting. This applies to the current fiscal environment as well. State legislators and their staff can use evidence in times of rising revenue when new funding is available, as well as in times of concern when budgets may be facing deficits or constituents have increased needs.
Appendix A: Other Key Questions
Other Key Questions And Considerations
1. Budget Guidelines
- Has the budget office issued budget guidelines that require or encourage agencies to submit evidence of effectiveness in funding requests?
- What types of evidence meet the budget office guidelines?
- Do these guidelines apply only to spending increase requests or are they used more broadly?
- Does the state set certain funding thresholds that require agencies to use a portion of available funds on evidence-based programs?
- How much of program spending funds evidence-based interventions?
2. Needs Assessments
- What data is currently being collected to demonstrate unmet needs for services?
- Who collected the data and how did they measure need?
- Who can we partner with to learn more?
3. Program Inventories
- Do we have a list of currently funded programs in this area?
- What do we know about program effectiveness in this policy area?
- Are there programs that need further evaluation?
- Do we know if this program is effective for specific populations or in specific settings?
4. Strategic Analysis
(Are resources addressing most critical needs?)
- Have agencies changed funding requests based on inventory or needs assessment findings?
- How does an agency’s performance or strategic plan inform its budget request?
- Are we funding programs that meet the greatest need?
5. Analyzing Benefits
- Are we comparing benefits and costs for this policy or program? If so, how do they compare with other interventions?
- How do we measure benefits and costs? Where does the data come from?
- Is this program achieving its intended outcomes? What are the monetary benefits of these outcomes?
6. Cross-Branch Engagement
- How is the legislature engaging with the executive branch to set priorities and agendas? Is there an opportunity to engage with stakeholders across government to set priorities or thresholds for evidence-based budgeting?
- Have we established thresholds for funding decisions based on evidence?
Kristine Goodwin, Program Director
Center for Results-Driven Governing