Toolbox | 5 Questions to Help You Test Evidence-Based Policy

11/20/2019

By Iris Hentze

If you haven’t yet heard the term “evidence-based policy,” you haven’t been listening. It was first used in 1992 in medicine and has since spread to other fields, including education, law and public policy. It’s now used so often, it’s easy to gloss over. But what does it really mean? When your colleagues in the legislature take policy positions, claiming the evidence supports their opinion, here are five questions to ask them.

1 | How Is ‘Evidence’ Being Defined?

Often, lawmakers will claim they have evidence to support a policy, but rarely do we stop and ask them to define how exactly they are using the term. In some states, evidence is statutorily defined. Knowing the definition in your state, and clarifying the definition when needed, is important. This will help ensure that everyone is speaking the same language and adhering to the same standards.

2 | Is It Causal or Correlational?

Causal evidence tells us whether a policy produced its intended outcomes. It’s the strongest type of evidence you can use to predict a policy’s impact on constituents. When used as part of a well-done study, it can even measure the extent to which a change can be attributed to a given policy. Correlational evidence can tell us that there is a relationship between a policy and its outcomes, but not that the policy caused the outcomes. The results may come from other factors or policies. Be sure to look for these differences.

3 | How Strong Is the Evidence?

If your colleague’s evidence is in fact causal, rather than anecdotal, it’s time to dig deeper. The scientific method teaches us that a study’s results aren’t completely reliable until the experiment has been replicated, producing the same or similar results. Policymaking is no different. The more a policy has been replicated and studied, the stronger the evidence and the more confident you can be that it will have the same effect in your state. Ask whether there is evidence from other states that shows similar, as well as different, results. Better yet, ask if there is a systematic review, which summarizes the best available research on the policy.

4 | What’s the Goal of the Policy?

At this point you may know the evidence is scientific, and that studies have been replicated. But does the evidence show that the policy achieves the specific outcomes that you want? It’s important to understand what outcomes the evaluations studied since a policy can be effective for many reasons. For example, a policy addressing substance use disorders among offenders may have been evaluated only for its impact on drug use and not on recidivism or employment. When assessing a policy’s evidence of effectiveness, it is crucial to ensure the evidence matches your state’s objectives.

5 | How Will We Know the Policy Works?

The policy may be evidence-based, with a clearly described goal, but does it include specific measures or benchmarks to track its progress and success? If such measures are left out, the agencies tasked with implementation will develop their own. Lawmakers can help ensure success by writing benchmarks into the policy and by requiring regular evaluations.

Iris Hentze is a policy associate in NCSL’s Employment, Labor and Retirement Program.

Additional Resources

NCSL Resources