NLPES 2001 Conference Notes: Evaluation Planning: Staying on Target

NLPES Conference Notes

Evaluation Planning: Staying on Target
Friday, September 7, 2001

Moderator: Wade Melton, Staff Director, OPPAGA, Florida

Speaker: Cindy Lash, Audit Manager, Legislative Post Audit, Kansas

Ms. Lash said that her office uses a series of 7 meetings to keep individual audits on track. She referred to a handout which showed the points at which these meetings are held. She proceeded to describe each meeting and its purpose.

The FOCUSS meeting is generally held within 1 - 2 days of the start of the audit. In this meeting the audit team identifies the concerns underlying the audit, and the types of information and analyses needed during the audit. The second meeting is the Audit Guide meeting, and during it the team agrees on the specific analytical tasks to be performed. The third meeting is the 2/3 meeting. At this meeting the audit team brings an informal written document that identifies the findings developed up to that point in the audit and also discusses the work that remains.

The fourth meeting is the Message meeting. It takes place the day after workpapers are completed. It purpose is to agree on a structure for telling the audit story. The fifth meeting is the Detailed Outline meeting. At this meeting, the audit team discusses whether the proposed outline makes sense, and identifies areas where the story isn’t as clear. The sixth meeting is the Draft Report meeting. At this meeting the audit team goes over the draft report that was submitted to upper management and identifies areas needing revision. The seventh and final meeting is the Lessons Learned meeting. At this meeting, the audit team discusses activities and processes that worked well and those that didn’t.

Ms. Lash said that the overall purpose of these meeting was to keep all members of the staff “in the loop” throughout the course of the audit.

Speaker: John Sylvia, Director, Performance Evaluation and Research Division, West Virginia

Mr. Sylvia said his office uses 2 primary tools to define an audit’s objectives, scope, and methodology.

The first is a Survey Summary Report. In the early stages of the audit, the team uses a checklist to gather and review certain information, such as the agency’s mandate, outcomes, and statutory requirements. Through this work the audit team gains an understanding of the agency and may identify potential audit topics.

The second tool requires the use of logic models. The purpose of this work is to dissect the agency by identifying its outcomes and the processes that are critical to achieving those outcomes. In some instances, his staff only evaluates how well the agency carries out those critical processes. Mr. Sylvia pointed out that it is much wiser to use limited audit resources to evaluate only the processes that are critical to citizens being served well.

Lastly, Mr. Sylvia stated that, in his opinion, a well written audit plan with a clear and complete methodology should keep an audit on target.

IU: Kevin Dervin, Managing Consultant, Baird, Kurtz & Dobson

Mr. Dervin spoke about the controls his firm uses to keep their consulting projects on target.

He stated that during the initiation phase of each project it is extremely important to know exactly what the client needs and to establish the limitations regarding what the project will and won’t accomplish. He emphasized that what’s included is just as important as what’s not included. After the scope of the project is set, his firm “decomposes” the project to assign staff, sequence the work, and set time schedules. They then monitor the project against those parameters.

Mr. Dervin noted that another way to stay on target during the execution of the work is to maintain consistent contact with the customer. This communication is essential because often times the customer can recognize potential problems–such as his firm may not have been all the correct information, or may have misinterpreted certain data.

Mr. Dervin noted that good project management involves many skills and computer software can’t do it all. Interaction with the customer is essential to staying on target.