Evaluation Planning - Staying on Target

Evaluation Planning - Staying on Target

Friday, September 7, 2001
John Sylvia

Evaluation Planning -- Staying on Target
According to Generally Accepted Government Auditing Standards (GAGAS), the first field work standard is planning (GAS 6.2). Under GAGAS (GAS 6.3), planning involves determining the audit's:

Objective;
Scope; and
Methodology.

Objective: What the audit is to accomplish. Audit objectives can be thought of as questions about the program that auditors seek to answer.

Scope: The boundaries of the audit. It addresses such things as the period of time and number of locations to be covered.

Methodology: The data that will be needed and analytical methods used to achieve the objective. Methodology includes not only the nature of the auditors' procedures, but also their extent (for example, sample size, etc.).

Planning should continue throughout the audit. Audit objectives, scope, and methodologies are not determined in isolation. Auditors determine these three elements of the audit plan together, as the considerations in determining each often overlap. (GAS 6.6)

Developing an Understanding of the Agency

To start the planning process, the auditor must gain an understanding of the agency. The following items can facilitate this part of planning:

Survey Summary Checklist (See Attachment A)

Survey Summary Report

Once the Survey Phase has been completed, the auditor should write a Survey Summary Report describing what is understood about the program. The report should contain at least the following basic information about the agency:

1. Indicate the agency's mandate and overall outcomes.

What is the agency's budget, source of funding, constituents served, and the number of staff.
A list of criteria such as statutory requirements (Federal and/or State), regulatory (or rules) requirements, self-imposed (administrative) goals, etc. Distinguish between "may" provisions and "shall" provisions in statute or rules.
Significant processes, transactions, controls, and outcome statistics.
A discussion of any problems that have been mentioned by the agency, board members, newspaper articles, etc.
Potential areas of evaluation focus.

II. Identifying Problem Areas or Areas to Focus On

If you are not told specifically what to examine either by law or by legislative leadership, then auditors must determine the areas to examine. While collecting information in the Survey Phase and completing a Survey Summary Report, various weaknesses and potential problems may be discovered. However, sometimes the survey phase may not identify any significant problems.

What do you do when no problems surface during the survey phase?

Answer: Avoid the approach of "Searching for Problems", which tends to be a "fishing" expedition.

Reason #1: Searching for problems can be an inefficient use of time because there may not be any problems in the program.
Reason #2: A loss of objectivity occurs when planning is motivated by a search for problems.

A better approach is: Examine areas that are critical to the success of the program, and simply report what is found, whether it is good, bad, or mixed.

Reason #1: It reduces the likelihood of missing something important if you are examining critical elements of an agency's operations.
Reason #2: It is a more efficient use of time.
Reason #3: It is more objective. It is difficult to question the motives of an audit when important areas of an agency are examined.

In order to identify critical areas to examine, dissect the agency by identifying its Outcomes and the Processes that are critical to achieving the outcomes.

Example 1:

Child Protective Services (CPS):