• Mission & Governance
• Services Overview
• How to Get Involved
• Standing Committees
• NCSL Foundation

• Issue Areas: A-Z
• State-Federal Relations
• NCSL Standing Committees
• News from D.C. and the States

• About State Legislatures
• Elections, Campaigns & Redistricting
• NCSL's StateConnect Directory
• Web Sites

• Online Guide to NCSL Services
• Staff Sections and Networks
• Meetings & Professional Development
• Staff Directories

• Meetings Calendar
• Online Registration
• Meetings Home Page
• Legislative Summit

• Bookstore
• State Legislatures Magazine
• NCSL's StateConnect Directory

2002 NLPES Fall Training Conference Notes
AzENET Workshop - Part 1
"Methods for Evaluating Behavioral Health Programs"

This two-part session was jointly sponsored by NLPES and the Arizona Evaluation Network (AzENET), and its primary objective was to learn about the methods the Arizona Department of Health Services (Department), Division of Behavioral Health Services (Division) has used to evaluate the "House Bill 2003" programs.

The Arizona Legislature passed HB2003 in the Fifth Special Session 2000 to provide the Department one-time only funding to enhance behavioral health programs and services for both adults and children. Funding came from the state's tobacco litigation settlement account. Laws 2000, Fifth Special Session, Chapter 2, §5 appropriated an additional $50 million for adults with serious mental illness, and $20 million for children's services.

The HB2003 Adult Services Program provides services such as:

• House and apartment model purchases and support services
• Expanded vocational rehabilitation and recovery support services
• Enhanced clinical team services - Assertive Community Treatment (ACT) teams
• Extended Evaluation Services

The HB2003 Children's Services Program provides services such as:

• Services targeting non-Medicaid eligible children and their families identified through collaboration with the Department of Economic Security (who serve children in Child Protective Services and the Division of Development Disabilities).
• Enhanced benefits package for children involved in the juvenile justice system (i.e., juvenile probation or parole), including intensive outpatient therapy, substance abuse treatment, pharmacy, and other outpatient services.

The Division of Behavioral Health Services is responsible for overseeing the State's public behavioral health care system for the state's Medicaid-eligible population. However, it receives limited state funding to provide services to non-Medicaid-eligible persons. The Division does not provide direct services, but instead contracts with five Regional Behavioral Health Authorities (RBHAs), which function like health maintenance organizations. The RBHAS, in turn, contract with provider networks and individual providers who provide direct services. The Division allocated the special HB2003 funding to the RBHAs according to state population distribution. The five RBHAs serve six specific geographic service areas (GSA) as identified below:

• Community Partnership of Southern Arizona (CPSA)- Serves two southern Arizona geographic service areas. One GSA includes Cochise, Graham, Greenlee, and Santa Cruz counties. The second GSA covers Pima County.
• Value Options - Maricopa County in central Arizona
• Excel Group - southwestern Arizona covering La Paz and Yuma counties
• Northern Arizona Behavioral Health Authority (NARBHA) - northern Arizona covering Coconino, Hopi, Mohave, Navajo, and Yavapai counties.
• Pinal/Gila Behavioral Health Authority (PGBHA) - east central Arizona covering Pinal and Gila counties.

Lois Sayrs, Ph.D., Senior Methodologist, Office of the Auditor General (Arizona)-Lois explained the purpose of the session. The topic was the evaluation of the HB2003 adults and children's programs. The first part of the session would address the Division's goals for the evaluation projects. The second part of the session would address the issues and challenges that the RBHA evaluation staff have faced in conducting program evaluations for both programs. She introduced the Division panelists:

• Dr. Michael Franczak: Has a Ph.D. in psychology and has been with the Division since 1995, and was recently appointed Chief of Clinical Services where he oversees the clinical bureaus for children, adults, and substance abuse/prevention. He has worked in Pennsylvania, North Carolina, and Arizona. He is interested in housing for adults with serious mental illness (supportive housing), and integrated substance abuse/mental health treatment models.
• Dr. Bernadette Phelen: Has degrees in economics. She has been with the Division since 1997, and manager of research since 1999. She provides technical assistance to the Division and to other state agencies. She is also involved in the national effort to develop mental health indicators and statistics, i.e., the Mental Health Statistics Improvement Project (MHSIP).

The children's and adults' programs have two very different objectives. For adults, the program was targeted to develop recovery supports. For children, the program was focused on improving collaboration between multiple child-serving agencies. Both program evaluation projects examine process and outcome measures. They are asking questions such as: 1) Was the program implemented as planned? and 2) Was the program implemented with fidelity to the proposed evidence-based practice model? Dr. Franczak explained that "service planning" and "best-practice" guidelines are also called "evidence-based" guidelines in health care. When a specific clinical practice or treatment is considered "evidence-based," it means research supports its effectiveness in treating specific disorders. However, not everyone in health care uses evidence-based practices. The Division tries to educate providers to use "evidence-based" practices. Doing so could also save money. If a clinical practice is evidence-based, then why do something else that will not work?

Dr. Franczak explained what was meant by "fidelity." Fidelity is a process to measure the match between the structure and the process of a specific program compared to the evidence-based model. "High fidelity" means that the program was a close match to an evidence-based practice. "Low fidelity" means a low match. Most adult programs stick to evidence-based models. Each RBHA had to submit their spending and program plans. The Division asked each agency to match each proposed adult program with evidence-based models. Each program has its own specific prediction. Sometimes they can experience "program drift" which means stray from the evidence-based model. Results from programs that have drifted are not comparable to predicted results from evidence-based programs.

Behavioral health programs are an example of a "dynamic system of care." Don Abedian (sp.?), an engineering professor at Michigan and the father of dynamic system research, stated that dynamic systems have three components: The structure represents the tangible aspects of the program, i.e., people and infrastructure; the process is how the structural elements interact; the outcome is the result of the process. Any dynamic system produces an outcome or output. When the structure and process are not aligned to produce a specific outcome, you will still get a result, however, it may not be the outcome or output you desire. The Division has been examining both process and outcome measures in its HB2003 children and adult program evaluations. They are asking questions such as:

• How many people were served? - process
• Did the program improve consumer functioning? - outcome
• Did the program reduce psychiatric symptoms? - outcome
• Did the program reduce adverse outcomes? - outcome
• Did the program improve interagency collaboration? -outcome

Dr. Franzcak applied this to a couple HB2003 programs. Assertive Community Treatment (ACT) is an established program that has known fidelity features. They are not evaluating in a black box. Each program has been tested using specific kinds of patients. Measures include: staff/client ratio, qualifications and types of staff, frequency and location of appointments, availability of crisis services, characteristics of patients. He also discussed their housing programs, both supportive and independent-housing with supports.

Due to time limitations, Dr. Franzcak could not complete his entire presentation. Dr. Franczak concluded by saying there is no evidence-based program that has 100% success with every consumer. Improvement potential is different for each person, and also depends on the person's motivation. You have to deal with the whole human being.

Dr. Phelen's presentation covered a couple major areas: review of program design including evaluation design and measurement tools, and evaluation challenges, including challenges in data collection and data analysis.

Basic Programmatic Features: In doing program design, there are some basic programmatic features. You have to understand what the program is trying to do. In the adult program, the legislation asks the program to provide employment, housing, extended assessment, and consumer-operated programs. In the children's program, there is the potential to assist the child and family and improve interagency collaboration. The basic programmatic features in each program design are:

Evaluation approaches: The Division is using four evaluation approaches: 1) logic model and theory-based evaluation; 2) process study, which captures program implementation structure, including changes in program design and cost allocation; 3) impact/outcome study, which captures the impact of the program in the service system and client outcomes; 4) cost study, which captures cost allocation and expenditures across service categories. The methods they are using to address process and cost outcomes include: 1) program document review, such as proposals and progress reports; 2) program monitoring, such as client rosters and site visits; 3) financial report review, including review of financial statements and evaluation of service encounters (claims). They are using 7 different methods to address outcome questions including:

• Pretest-Posttest
• Focus Groups
• Repeated measures (survey data from 3 surveys, an adult survey, a family survey, and recovery instrument scale) issued at baseline, 6-month follow-up, and program discharge.
• Process indicators (Children)
• Interagency collaboration - through a collaboration survey instrument (Children)
• Case file review (Children)
• Program fidelity (Adults)

Dr. Phelen said they could be "wasting their time" with the pretest-posttest method due to "confounding factors." Experimental design can fail, which is why they are using multiple methods to evaluate the programs.

Dr. Phelen identified challenges in the evaluation design phase:

• Use of comparison groups: It is difficult to use and study comparison groups, but they still might try to do it. If they can't, then the second best alternative is "benchmarking."
• Capturing unquantifiable benefits/externalities: They have been working with the Department of Housing and the DES Vocational Rehabilitation. They are not sure how to capture these benefits.
• Controlling confounding factors: How do they isolate the outcomes of the HB2003 programs? Most people receive a bundle of services.
• Isolation of programmatic/service effects: How do you isolate getting housing, and being in a program such as Assertive Community Treatment?
• Limited literature in the study of system support in client recovery: Dr. Phelen said there was limited literature on the study of system support in client recovery. She said things were happening at the national level through MHSIP.
• Attrition: Some people are leaving the program because they have become eligible for Medicaid. The funding is for non-Medicaid eligible persons; however, the state expanded Medicaid coverage after the passage of the HB2003 program.

The last part of Dr. Phelen's presentation discussed measurement tools and data analysis. They are using a variety of survey instruments, administrative data on client assessments and service data (encounters), a data validation tool, and possible benchmarking data from MHSIP and other sources. They experience several challenges in data analysis. For example, there is a lag time in encounter reporting because providers have up to six-months to turn in their claims. Some people are also feeling "over-surveyed." They are using a variety of research methods, and they are triangulating data sources. They will need to validate their findings. The challenge they face will be if the different methods result in different results.

This ended part 1 of the AzENet workshop.

Representatives from three Regional Behavioral Health Authorities presented in part-2 of the AzENet session. See separate write-up for their comments.

• Karen Tiggs, Outcome Evaluation Quality Management Department, Value Options.
• Dr. Michael Funk, Research and Evaluation Manager, Northern Arizona Regional Behavioral Health Authority
• Jim Gross, Data Evaluator, Community Partnership of Southern Arizona.

2002 Fall Training Conference Notes