Criteria/Checklist

Harry P. Hatry
The Urban Institute
August 28, 2001

Criteria/Checklist for Assessing

1. An Agency Performance Monitoring Process
and
2. The Quality of a Legislative Performance Audit

The following sections present sets of criteria that legislative auditors and evaluators can use to:

1. Assess an on-going performance monitoring process of an operating entity.
2. Assess the quality of their own performance audits.

Note that these sets of criteria have many overlapping criteria.

Criteria for Assessing an On-going Agency Performance Monitoring Process

1. The examination covers in a substantive way one or both of: (a) service outcomes/results/quality; and (b) service efficiency.
2. The process contains an explicit identification of each program's mission and objectives. All, or at least most, service programs are included in the performance measurement process.
3. The performance indicators are each clearly defined and labeled accurately as to into what category each indicator fits, distinguishing among: (1) inputs; (2) level of workload coming into the program; (3) amount of activity actually completed; (4) intermediate outcomes; (5) end outcomes; and (6) efficiency (productivity). In the latter case, preferably, the process should also distinguish whether the indicator of efficiency measures "technical" efficiency (the amount of physical output divided by the amount of input) or "outcome" efficiency (the amount of outcome related to the amount of input).
4. Each indicator has a reasonable causal link to program activities, the program's mission and its goals and objectives. That is, there is empirical evidence or reasonable logic to support the assumption that program activities impact the program's mission, goals, and objectives and that changes in performance indicators can be affected at least partially by program activities. Does each indicator measure what it purports to measure?
5. The set of performance indicators for each program that the process is using to assess program performance is comprehensive. That is, it covers most, if not all, of the measurable program goals and objectives. (This is primarily of importance for audits in which service outcomes, results, and quality are being assessed.)
6. The performance indicators cover the perspectives of the various interested parties, such as program clients, organizations affected (e.g., businesses), elected officials, the general citizenry, etc.
7. The performance indicators have been reasonably stable over time. That is, the indicators used and reported should not have been significantly changed or altered each year. Indicators which cannot be compared over time may suggest that program staff are engaged in selective reporting, which may misrepresent program effectiveness or efficiency. When necessary modifications are made, significant changes and their impact on reporting performance are explained.
8. Information collection meets reasonable tests of validity and reliability. That is, the procedures used to obtain the data for each performance indicator are sound so that the information obtained is reasonably valid and reliable. For example (1) sampling procedures are as representative of the surveyed population as feasible; and (2) efficiency indicators include some form of quality control over the output count to ensure that increased output has not been achieved at the expense of service quality and, thus, is truly an efficiency improvement.
9. Performance reports describe (possibly in footnotes) the information collection procedures and their limitations to the extent necessary to users to understand clearly what the measurements mean and put them into perspective. For example, if survey sampling has been used, the report should identify the sampling frame and procedures for drawing the sample, the mode of survey (e.g., whether the survey used mail, telephone, in-person surveys or combinations of these), the response rates, the time period of the survey, and likely precision of the estimates.
10. Performance reports do not only show aggregate data but also show relevant disaggregated data (i.e., breakouts), such as performance indicator values for various geographical regions, for each different facility, for groups of clients based on key customer demographic characteristics, and for each major category of incoming workload difficulty.
11. Performance reports are timely. (a) They are provided to users within a reasonable amount of time after the close of the reporting period; and (b) the reports are produced frequently enough to permit program personnel and elected officials to take timely action based on the performance reports' information.
12. Performance reports presents relevant comparisons in order to provide a perspective on the level of performance achieved in the reporting period. Major types of comparisons include: (a) prior periods' achievement levels (preferably both year-to-date as well as the most recent period); (b) comparisons to targets set for the period and year-to-date; (c) performance of selected subgroups to other subgroups or the overall average; and (d) available norms or standards.

If the reports include targets, these should have been set in some reasonable way so as to identify targets that are neither too easy to reach nor unreasonably difficult. The targets should have been established after considering program resources, i.e., staffing and dollars. And are the targets reasonably current or have circumstances altered so as to obsolete the meaningfulness of the targets?

13. Performance reports clearly identify the limitations and uncertainties in the reported data.
14. The performance reports are clearly labeled and presented so that expected users are able to understand the findings.
15. Data sources are fully disclosed.
16. There is evidence that the data are being used by program personnel.
17. The performance measurement process encourages programs to provide explanatory information (whether qualitative or quantitative) for unusual/unexpected findings. Evidence exists that explanations for unusual/unexpected performance findings are being systematically provided.
18. The agency/program has a reasonable quality control process that periodically reviews the data collection procedures and, at least a sample of, data.

Criteria for Assessing the Quality of a Performance Audit

1. The examination covers in a substantive way one or both of: (a) service outcomes/results/quality; (b) service efficiency.
2. The report identifies a mission statement for the program, including program objectives, and the set of evaluation criteria the audit has used to assess program performance. (This is primarily of importance for audits in which service outcomes/results/quality are being assessed.)
3. Each indicator should have a reasonable causal link to program activities, the program's mission and its goals and objectives. That is, if the value of the performance indicator changes, this should indicate the possibility that the program itself could have helped cause the change. (It is not necessary that there be a certainty that the program has itself caused the observed the change. This would delete most indicators of outcomes.)
4. Information collection stands reasonable tests of validity and reliability; that is, the data collection procedures, and procedures used to draw conclusions as to the program' own impacts on the evaluation criteria, are sound. The evaluation criteria used should be indicators of outcomes/results/quality and/or efficiency. If efficiency indicators are used, the measurements should include some form of quality control over the output count to ascertain that increased output has not been achieved at the expense of service quality. Any sampling procedures used by the auditors should be as representative of the surveyed population as feasible.
Special attention in these validity and reliability assessments should be on measurements for which the program itself "controls" the measurements (and which, therefore, are particularly susceptible to manipulation and credibility questions).
5. The report fully describes the report preparer's information collection procedures and their limitations. For example, if sampling has been used, the report should identify the sampling frame and procedures for drawing the sample, the mode of survey (e.g., if people are being surveyed, whether the auditors used mail, telephone, in-person surveys or combinations of these), and the response rates. The report should also, to the extent practical, provide evidence about the representativeness of the responses, such as how the demographic characteristics of the surveyed persons compared to known demographic characteristics of the total population that the sample is supposed to represent.
6. The report does not only show aggregate data but also shows relevant, important breakout data for key performance indicators, such as by incoming workload difficulty, region, and type of client.
7. The audit makes it clear to what the audit is comparing actual performance in order to allow judgments to be made as to the level of performance that the audit found to be achieved. Major types of comparisons include:

(a) Prior years' achievement levels;
(b) Targets set (and by whom and how the targets were established);
(c) Private sector experience;
(d) Performance of other agencies;
(e) Performance of selected subgroups to other subgroups or the overall average; and/or
(f) Available norms or standards.

The report should provide a justification for each comparison used and discuss the strength of the comparisons. Almost all, if not all, comparisons will inevitably have significant limitations. Thus, the key here is not full comparability but rather the reasonableness of the choices and the completeness of the report's discussion of the limitations of whatever comparisons have been made. Preferably, the assessment will have used more than one type of comparison so that performance can be viewed from a variety of perspectives.

8. The methods and findings used to establish the extent to which the program being audited had itself caused the outcomes (whether the outcomes are good or poor) should be fully documented -- and their limitations identified.

If the audit used comparison groups, the comparisons should have been fully described and limitations clearly identified. To the extent feasible, the more powerful designs should have been used, such as carefully matched comparison groups. However, it will usually not be possible for auditors to use program evaluation designs such as randomized control groups. Therefore, auditors, working after-the-fact will inevitably need to use less powerful assessment/evaluation designs. This will have to suffice, but the procedures and their limitations need to be fully and clearly laid out in the audit report.

9. The audit has made a real attempt to consider the perspective of the various interested parties, such as program clients, organizations affected (e.g. businesses), elected officials, the general citizenry, etc.
10. The report clearly identifies the limitations and uncertainties in its findings.
11. The report is written clearly and presents the substantive findings so that the expected users will be able to understand the findings.
12. Full citations as to data sources are provided.
13. The program being reviewed has been given an opportunity to review the report and its findings and recommendations prior to its being disseminated or shown to others. (Note there may be exceptions, but this probably should be the rule.)
14. Conclusions drawn from the findings follow from the evidence presented in the report.
15. Recommendations are presented. They, in general, indicate the information from the report findings used as their primary basis. If the auditors make judgments in the report's conclusions or recommendations, it should be clear that these are judgments. The cost and staffing implications of the recommendations should be estimated.