NLPES 2001 Conference Notes: Evaluating and Using Agency Performance Measures

NLPES Conference Notes

Evaluating and Using Agency Performance Measures
Friday, September 7, 2001

Moderator: Gary Brown, Audit Division Administrator, Michigan Office of the Auditor General

Presenter: Verma Elliott, Project Manager, Texas State Auditor’s Office

Texas jumped into requiring agencies to use performance measures with a budget reform proposal in 1991, and those measures had to be in place by 1993. Those performance measures are used

by state leaders to make funding decisions and to evaluate state programs
by state employees to assess an agency’s performance and make budget recommendations
by the State Auditor’s Office (SAO) to determine audit priorities and trends within a function of government

An effective measurement system should be results-oriented (measuring both outputs and outcomes), selective, useful to decision-makers, accessible, and reliable. The measures themselves should be responsive, valid, cost-effective, and relevant (i.e., they should line up with the performance goals the Legislature set), and they should provide comprehensive coverage of the agency’s activities. To help ensure Texas agencies had a common language and understanding for their performance measures, the SAO developed the Guide to Performance Measure Management (available at http://www.sao.state.tx.us under Resources, Performance Measures).

The SAO each year does certification audits for a portion of the state’s 7,000 key performance measures at the 250+ agencies to verify that systems include adequate internal controls to increase the probability that reported measure data will continue to be accurate. (SAO staff have just completed reviewing each agency at least once.) There audits have found a gradual increase in the reliability of agency data, but still only about 62% have been found to be reliable. If the audit can re-create the numbers reported within ± 5%, the SAO “certifies” the agency’s reporting. However, the most prevalent rating is “certified with qualification,” meaning that the number reported probably is right but that it probably couldn’t be replicated.

Lessons learned so far:

Measuring results is a difficult long-term process.
Any measurement system requires refinement and adjustment.
Some agencies must change their information and other accountability systems to ensure performance measures are reported accurately.
There is significant risk to the state when agencies fail to achieve their performance targets or take corrective action on uncertified measures.

Ms. Elliott can be reached at measures@sao.state.tx.us.

Presenter: Gary VanLandingham, Deputy Director, Florida Office of Program Policy Analysis and Government Accountability (OPPAGA)

In 1994, the Florida Legislature mandated that performance-based program budgeting was to be phased in over a seven-year period. Agencies must establish comprehensive performance measurement systems for their programs, measuring inputs, outputs, unit costs, and outcomes. The Legislative Budget Commission uses the information collected to conduct “0-based” budget reviews. The Legislature approves the measures and incorporates them in the budget. OPPAGA uses the measures in its studies, and it also provides technical assistance to the agencies that are developing their measures.

Performance measures are useful in addressing bottom-line effectiveness questions. For example, the state now has data that wasn’t available before on the types of services being provided to mental health clients, the cost of those services, and client outcomes. With that data, programs in various parts of the state can be compared and, it is hoped, effectiveness can be improved by replicating successful programs.

Often, for an evaluation, the data available isn’t what you want, what you think you want doesn’t turn out to be what you need, and what you want and need doesn’t exist. Good performance measures should minimize the “doesn’t exist” category and provide better accountability information for managers, evaluators, and policy makers. Too many measures can overwhelm an agency; too few mean performance can’t properly be assessed. Ideally, measures can show trends in how funding affects performance.

Performance measures can be for tasks, programs, and policies.

The task level measures agency activity.
Program measures gauge the extent to which key strategies are carried out successfully.
Policy measures gauge the extent to which the program meets the purpose for which the legislature created it.

The latter two are, more often than the first, reported to a legislature.

Lessons learned, so far:

While performance measures improve accountability, they are more of an early warning monitoring system than an in-depth research system. They won’t replace performance audits.
Audit agencies can provide technical assistance, but make agencies be responsible for verifying their own performance measures.
Providing technical assistance will increase your workload.
Performance measurement requires a long-term commitment. It takes time to develop and improve them and for policy makers to learn to use them. The system must have high-level support over this long haul.

Status reports on performance budgeting in Florida, OPPAGA assessments of agency performance measures, and performance evaluations of agencies operating under performance budgets are available at www.oppaga.state.fl.us.