Performance Auditing and Program Evaluation: Where Do We go From Here?

NLPES Conference Notes

Performance Auditing and Program Evaluation: Where Do We go From Here?
Guest Speaker: Harry Hatry, Director, Urban Institute’s Public Management Program
Thursday, September 6, 2001

The focus of performance auditing, Hatry stated, is the evaluation of the outcomes, rather than the process. There are 2 principal forms of information on performance:

1) ad hoc studies (evaluations)
2) regular performance monitoring

Performance audits attempt to get at the reasons performance is good or bad, while performance measurement only tells the score, not how it came to be.

There are 4 types of performance information that should be provided:

1) Inputs: costs, staff time, etc.
2) Outputs (amount of work completed)
3) Outcomes
a. Intermediate
b. End

Hatry explained the need to distinguish between intermediate and end outcomes. He used the analogy that an intermediate outcome is improving people’s diets, while an end outcome would be improving someone’s health.

4) Efficiency
a. Amount of input per unit of cost
b. Amount of output

An outcome is a characteristic of a service important to customers/the Public.

Outcome indicators are measures of the number and type of actions taken by customers/target audience (not actions taken by the Program).

Hatry explained the use of Logic Models to show what a program is supposed to be doing. He used the example of a stop-smoking program, showing flow chart of what a successful stop-smoking program would look like.

Information Collection Procedures:

1) obtain data from agency records
2) trained observer ratings: i.e. photos or written criteria by which to categorize cases/information
3) client survey
4) household surveys
5) role-playing/testing
6) physical measurement equipment
7) systematic use of “expert” judgements.

Hatry noted that surveys don’t usually reveal causality. He further noted 3 numbers affecting survey cost and accuracy: 1) acceptable response rates, 2) desired confidence interval, 3) acceptable confidence level.

He further cautioned against using overly aggregated data. Also, consider the clients when looking at agency’s outcomes-was there something unique about the client base at that time that made outcomes worse?

Hatry pointed out 5 recommendations for legislative-oriented performance auditing:

1) Legislature should require outcome measures.
2) Legislature should require agencies to provide explanations for deviations from target.
3) Legislature should require regular quality control measures for agencies.
4) New programs should have built-in evaluation components.
5) There should be an agenda setting process for what is to be audited during the year.