• Mission & Governance
• Services Overview
• How to Get Involved
• Standing Committees
• NCSL Foundation

• Issue Areas: A-Z
• State-Federal Relations
• NCSL Standing Committees
• News from D.C. and the States

• About State Legislatures
• Elections, Campaigns & Redistricting
• NCSL's StateConnect Directory
• Web Sites

• Online Guide to NCSL Services
• Staff Sections and Networks
• Meetings & Professional Development
• Staff Directories

• Meetings Calendar
• Online Registration
• Meetings Home Page
• Legislative Summit

• Bookstore
• State Legislatures Magazine
• NCSL's StateConnect Directory

NLPES Site Map

NLPES Surveys

Previous Years:

• Ensuring the Public Trust: How Program Policy Evaluation Is Serving State Legislatures (2000) --This report summarizes information about these legislative program evaluation offices, jointly conducted by NLPES and the Fiscal Affairs Committee.
• Follow-Up Procedures (1999)
• Salary Survey (1998)
• Legislative Evaluators:  A Diverse Group of Professionals (1997)

---

Topic Selection Practices:  What useful practices have your staff or oversight committee adopted to help identify the projects your office should undertake?  (Source:  June-July 2001 NLPES survey)

(Editor’s note:  Some of the more intriguing suggestions below include:  the Texas State Auditor’s “risk-based” topic identification model; Arizona’s “multi-attribute utility model” (hmmm… tell us more!); Wyoming’s method of categorizing potential studies (Types 1 to 5); formal surveys of legislator preferences (Minnesota and Tennessee); creating an expectation for evaluation staff to develop good topic ideas (Colorado); and soliciting ideas for studies from financial auditors (Michigan and Montana).  Also, some states have developed formal criteria to help their oversight committees select topics—see the examples provided by Washington.)
 

Michael J. Mayhew, Michigan Office of the Auditor General

1.  We solicit ideas for performance audit topics from our auditors and supervisors familiar with various agencies.  2.  We monitor legislative activity to identify emerging areas of interest.  3.  We make it clear to legislators that they may request audits and/or information relative to particular programs.  4.  We prioritize potential audits based upon the above factors and such other risk/opportunity indicators as:  (a) time since most recent audit, if any,  (b) potential weaknesses based upon past audit findings and/or audit work in related programs,  (c) reports from legislators or other reliable sources of potential problems which an audit might help resolve, and  (d) etc.

Most of the audits we do are required by the Sunset Law and are set years in advance.  However, we also have "Special Reports" which are in response to particular concerns from legislators about an agency or program that is not on the Sunset schedule or will not be evaluated for several years.  Generally, the Legislative Auditor receives requests for a special audit from the Senate President, House Speaker or the Chair of a legislative committee.  We determine if we have the resources to do some of these audits, but we do between 1 to 5 special reports each year.

The Legislature defines our schedule in statute.  However, we have considerable latitude in the program and policy areas we concentrate on in a Sunset review of an agency.  Our focus is primarily determined through review of legislation introduced over the previous two legislative sessions, interviews with interest groups, such as those representing regulatees or public consumer groups and other interested parties, and through our initial overview of the agency and its operations.

We solicit requests for evaluation from all Legislators at least once each year, and focus efforts even more intently on germane committee chairmen.  We develop 1-2 page "scope papers" for the Oversight Committee's consideration when selecting from among several requested evaluations.  These papers provide a brief explanation of the concerns voiced, general background, and proposed research questions.

All audits originate from legislative requests.  The office will, as needed, work with individual legislators and subcommittees in their development of requests.  Requests are reviewed and prioritized by a legislative management oversight subcommittee.  The office is a participant in this process with the subcommittee frequently requesting information before priorities are finalized.   The subcommittee’s prioritization is based on legislative informational needs and their perception of the office’s ability to address the request’s issues.

About half our projects are the result of House or Senate resolutions.

All projects selected internally are based on the results of a preliminary survey which is used to determine if there are issues which could result in significant recommendations dealing with efficiency and effectiveness.

Our projects are chosen by our 5-member citizen Council from written requests submitted by members of the General Assembly.  Based on information from staff about the scope of the project, they prioritize the requests based on materiality and choose the projects they perceive as being most useful to the General Assembly.  Legislative ex officio members participate in this process as well. Also, staff are frequently consulted about requests before they are submitted, and we try to work with requesters so that their requests will be "doable" and relevant.

We identified five types of studies that the Committee could direct, ranging from Types 1 and 2 (efficiency and effectiveness studies) to Type 5, which would be studies of large and complex policy issues on which there may not be fundamental agreement within the Legislature.  When orienting our newly-appointed oversight committee every two years, we give some examples of, and make a definite pitch for avoiding, Type 5 studies.  That doesn't always work because sometimes those are the topics that have the attention of legislators, and they think we can at least contribute to the understanding of whatever the problem is.  But we do work with members to get them to understand the difference between a compliance audit and a "Type 5 study."

We use a risk based approach to select our discretionary projects.  Using a self-developed risk model that evaluates each agency's business, service, and financial risk, our risk assessment team evaluates and prioritizes risk in the state and develops both our work plan and a statewide risk map.

Developed a multi-attribute utility model to guide overall audit selection, meet with legislative leadership and oversight committee members to get their input, meet with large agencies in advance of audits to develop understanding of agency and their input into areas they could benefit from us looking at.

We held a series of three topic selection meetings with the elected members of JLARC.  These three meetings brought member interests to the attention of staff and familiarized the Commission members with existing research, staff workload, and certain research constraints.

1.  Review the Analysis of the Budget Bill prepared by the Office of the Legislative Analyst.  2.  Track auditees' progress in implementing our past recommendations.  3.  The staff of our oversight committee maintains a continuous dialogue as to the current interests of the legislative committees and of the legislative leadership, which can lead to members requesting that we do audits.  4.  Each year we prepare reports for the various legislative subcommittees.  In these reports, we summarize the major findings and recommendations from audit and investigative reports we issued during the previous 2 years related to entities under their purview.  The reports also identify the corrective action, if any, these entities have taken in response to our recommendations.  Not only have these reports proven useful to the subcommittees, we have found them to be an effective way to educate new legislators on the types of audits we do.  This is particularly important in light of term limits.

1.  Solicit ideas for projects directly from legislators via a letter from the chair of our Legislative Audit Commission.  2.  Commission subcommittee selects a "short list" of 12-15 topics for in-depth research, using a set of topic selection criteria for guidance.  3.  We survey all legislators to find out the level of interest in the "short listed" ideas.  (Typically, we have received responses from about half of the legislators.)  4.  Subcommittee chooses finalists based on background research and survey results.

We recently formed an ad hoc committee of all staff levels to improve our process for selecting audit topics.  Several ideas developed, but the most significant was to build an expectation into the office culture (and performance evaluation system) that all staff are responsible for identifying audit topics. In the long run, the office sinks or swims based on the quality of topics selected for audit.

Most project topics are selected by our oversight committee, the Joint Legislative Audit Committee.  The topics are based on requests submitted by legislators.  The Committee assesses the level of broader legislative interest in the requests and Legislative Audit Bureau staff provide advice and counsel on the type of research that could be conducted on the topics under consideration.

Most PEER projects are mandated by state law or requested by legislators or legislative committees.  Fortunately, the release of some of these reports seems to coincide with 'hot topics' being considered by the Legislature.

We have no oversight committee, but staff is attentive to issues discussed in all committees.  We also monitor many national news sources that sometimes indicate how Tennessee ranks compared to other states.  (We recently undertook a study on air quality as a result of a national study citing Tennessee's four largest cities as ranking in the 25 worst in the country.)

We annually survey all members of the legislature for topics they would like to have evaluated.  We then meet individually with members responding to the survey and prepare scope statements on each topic.  Our committee, the Joint Committee on Legislative Research, then prioritizes the projects and assigns them to our office to complete.

Since we have an office with performance, financial/compliance, and information technology auditors, projects can be identified by financial/compliance audit during their biennial audits of each agency.  Projects are also identified during performance and IT audits.  This information is compiled along with areas requested by legislative committees; individual legislators; house, senate, or joint resolutions; and requests from the agencies.  A brief explanation of the questions to be addressed and a priority checklist are provided to the Audit Committee.  Each member rates every potential project.  The scores are summarized and list of priority projects is developed.  The projects are presented again to the Committee members to identify any "major" issues that should be addressed.

Most audits are requested by legislators and are authorized by our legislative committee.  When a legislator contacts us with an audit idea or request, we work with him/her to develop a scope statement that focuses the concerns as tightly as possible.  A scope statement summarizes the audit issues and legislative concerns, then lists one or more questions we would answer and the general methods we would use to answer them.  This allows us to answer the legislator's concerns efficiently, without delving into areas nobody is particularly concerned about.

Virtually all of our projects are initiated at legislative direction.  This occurs in several ways.  We are assigned topics during session as part of substantive bills and/or provisions in the appropriations act; this accounts for about 35% of our workload.  Statutes require us to undertake other projects on a rotating basis.  For example, the Legislative Budget Commission conducts Zero-Based Budget Reviews of each agency on a 5-year schedule, and we conduct a Program Evaluation and Justification Review of each agency prior to this date to assist the Budget Commission.  We are also required by law to conduct Best Financial Management Practices reviews of each school district on a 5-year schedule.  OPPAGA is also required by law to conduct a follow-up study beginning 18 months after we issue a report.  Together, these scheduled reviews are about 50% of our workload.  Finally, the Presiding Officers (Speaker of the House and Senate President) and/or the Joint Legislative Auditing Committee direct OPPAGA to conduct other studies.  These special studies constitute the remaining 15% of our workload.  OPPAGA's Director is authorized by law to initiate projects on his own discretion, but we rarely due this as our capacity is generally consumed through projects that are specifically directed.

We recently formalized our internal process for identifying audit topics.  It involves continually soliciting input regarding potential audit topics from all staff, compiling the suggestions in a centralized database, and then periodically reviewing the data to choose topics for the audit schedule (i.e., management staff meet several times a year to review the suggestions and then make scheduling and staffing decisions accordingly).

State law (Act 1465) requires a performance audit to be done in each state department at least once every 7 years.  Meet with legislative staff to identify "hot" issues.  We also look at what's "hot" in the news and get suggestions from our financial and compliance audit division.

The Committee has five criteria it uses to assess the priority of potential JLARC studies—whether mandated or selected by the Committee.  These are:

• Is the study consistent with the JLARC mission or is it mandated?
• Is the subject of significant fiscal or program impact, a major policy issue facing the state, or otherwise of compelling public interest?
• Will there likely be substantive findings and recommendations?
• Is this the best use of JLARC resources (e.g. is JLARC the most appropriate agency, is the study non-duplicating, and would it be cost-effective?)?
• Is funding available (within JLARC’s budget or through additional funding) to carry out the project?

---

Project Scoping:  What steps have you found to be effective in helping to ensure that your studies address the issues of greatest concern to legislators and the public? (Source:  June-July 2001 NLPES survey)

Michael J. Mayhew, Michigan Office of the Auditor General

1.  We meet with legislators who have an apparent interest in a program to be audited to identify their concerns and ideas concerning the audit.  2.  We perform stakeholder surveys during the preliminary review phase of each audit to identify issues of concern.  3.  We provide our preliminary audit objectives to interested legislators and meet to discuss them.

We sometimes receive feedback from the co-chairs of our committee on what they would like us to look at in an agency.  On occasion, we discuss initial concerns with various committee staff who often know particulars of the concerns legislators have about an agency that arise especially during the budget process in the finance committees.

We meet with staff of legislative leadership and members of the Sunset Commission.  We will also contact members that have an established interest in the subject area and committee chairs.  In addition, our focus is determined through review of legislation introduced over the previous two legislative sessions, interviews with interest groups (such as those representing regulatees or public consumer groups and other interested parties), and our initial overview of the agency and its operations.

We work with the project requestor to ensure the potential research questions included in the "scope paper" accurately reflect his or her interest.  Once a project is selected, and we have done initial scoping work, we compile a refined scope summary, which the Oversight Committee reviews and signs off on.  Beyond this point, all research is confidential until report release.

Primary audit scope is directly related to the original request and the Audit Subcommittee’s directions.  A great deal of control is retained by the office with the ability to add or subtract from the original request.  As needed, both the original requestor and subcommittee members are consulted with audit scope changes.  These consultations, in combination with the assignment process, give us a balanced approach that provides information desired by the Legislature while allowing us to maintain our independence.

1.  We give high priority to projects we receive as the result of a House or Senate resolution.  2.  Meet with legislators/staff if they express an interest in a study.

All members of the cognizant legislative committees are notified when we initiate a project and are asked if they have any areas of special interest or concern that relate to the topic.

We always consult the audit requesters to get their view of what issues are of most interest or concern to them.  We also conduct survey fieldwork to gain an understanding of the topic and issues involved.  Once we determine our audit objectives, we send them to the requesters again and ask for further input if they have questions.

Listening very carefully when legislators discuss their concerns, and taking copious notes.  Sitting down with the requester to review core issues. Occasionally, calling a member of the minority party to get a feel for what might not have been said in an open meeting.  Checking with legal and fiscal staff in our office on the history and context for the request. Sending the committee a list of proposed objectives, and hearing no objection, proceeding with them.

We use a risk assessment approach to help ensure that we focus our resources on the most important areas to our legislators and the public.  Our indicators in the three categories of business, service, and financial risk help to assess whether the public is getting what the legislature intended and whether those services are provided in an efficient and effective manner.

Ensuring that the audit addresses the determinations in the legislative resolution (since these are the questions the General Assembly wants answered)

Frequent team meetings and meetings with agency to discuss and assess the areas; contact with chairs of legislative committees that oversee agencies to solicit their input; contact with legislative staff members; legislative interest letter to solicit their input.

The topic selection subcommittee helped ensure that JLARC staff work was of interest to members.  At this time, virtually all staff work results from a legislative mandate (e.g. a study resolution or similar requirement) or from the direction of the Commission.  During the meetings of the topic selection subcommittee, the Commission members prioritized areas of interest.

All of the audits we perform are requested by the Legislature or mandated by law.  Our office prepares an analysis of the audit request that outlines the audit scope and objectives.  Our executive management and staff attend the hearings where the audit request and our analysis of the request are discussed.  In some instances, our executive staff will assist the legislator in drafting the audit request.  At times it is necessary to ask the requester for clarification.  As a rule, contacts with the Legislature are limited to the executive office until the report is issued.  To ensure the audit team understands the issues of concern, the executive management and staff meet with the audit team before the audit begins.  During scoping and fieldwork, executive management meets with the audit team periodically to ensure the audit work is focused, to approve changes in the audit approach when warranted, and to assess progress.  Finally, executive management reviews the report in various stages of development to ensure the report addresses the requesters’ concerns.

Roger Brooks, Minnesota Office of the Legislative Auditor

Hold a "roundtable" discussion, open to all legislators and staff, to talk about project scope.  This helps to determine what issues are of most legislative interest and build a "constituency" for the completed study.

As part of developing a study plan, we identify the targeted "study questions" within the larger topic we have been assigned.  Then we ask key legislators to review these  study questions to be sure we are addressing their concerns.  The study questions may be modified before we start implementing our study plan.

1.  Meetings with legislators to help shape audit requests.  2.  Preparation of scope memoranda for discussion at meetings of the Joint Audit Committee.  3.  Conducting interviews with interested legislators and interest groups as the study      begins.  4.  Keeping an "open door" throughout the audit process, including an open door in the state auditor's office, to allow modifications to the workplan based on legislative interests, as the project budget permits.  5.  Periodic meetings between the state auditor and the audit team to discuss audit planning, audit findings, and early conclusion development.

Our staff has tended to be more reactive rather than proactive regarding the scope of projects.  However, over the recent past, we have worked with legislators who request projects to encourage them to allow us to design projects that address the "bigger picture" rather than a specific program or process.

As part of our planning process we always interview key legislators such as bill sponsors, committee chairs, etc.  We also surveyed legislators this year for the first time, although their response rate wasn't very good.

We meet individually with key members of the legislature (member requesting project be done, chairmen of committees overseeing the state department or program, members of appropriations committees).  We also survey customers of the program regarding their satisfaction or areas of concern.

During scoping we involve all staff who have knowledge in the general area.  The scoping meeting is free form—brain storming.  Once in-house knowledge is identified contacts are made with other legislative divisions that may have knowledge of the area.  These staff would be fiscal and research staffers.  Their knowledge of committee actions and discussions during the legislative session is communicated.  We then send a letter to upper management of the agency informing them of the upcoming work (still general).  We do not formally entrance until we have talked to operations management and get their views of the potential areas for review.

When a legislator contacts us with an audit idea or request, we work with him/her to develop a scope statement that focuses the concerns as tightly as possible.  A scope statement summarizes the audit issues and legislative concerns, then lists one or more questions we would answer and the general methods we would use to answer them.  This allows us to answer the legislator's concerns efficiently, without delving into areas nobody is particularly concerned about.

The most critical step is to directly (and regularly) contact the legislative stakeholders to understand what issues they want to be studied.  We do this for all projects, contacting the member sponsor and key legislative staff (appropriations and substantive committees).  We will meet with them at the beginning of the project, at some point(s) during fieldwork, and when we have completed the project.  Our philosophy is that if our projects are responsive to legislators, the elected representatives of the public, then the projects are responsive to the public.  Also, during projects our teams must meet with our "Team Support Council," consisting of management and senior staff, to discuss project scope and methods.  These "TSC" meetings must occur shortly after project initiation to discuss scope, after a preliminary survey has been completed to formalize scope and methods, and towards the end of project fieldwork to discuss potential findings and recommendations.

Among other approaches, we try to keep projects scoped as narrowly as possible so that we can complete the work in a shorter time frame.  This helps ensure that audits are both timely and useful to legislators and members of the public.

We attempt to look at programs that have the most fiscal impact, most potential for abuse, most citizens affected.  We also meet with legislative staff, monitor committee meetings and legislative sessions, and monitor issues in the news.

To ensure we include relevant topics in our studies, staff conduct fairly extensive “pre-audit surveys”.  These “surveys” can take as little as several hours, but more typically take from 2 to 4 weeks. They involve some or all of the following activities:

• Discuss context of study request with members or sponsors, as well as with legislative staff;
• Review related state statutes and legislative history;
• Review outside evaluations and other relevant studies;
• Search for professional or academic papers on the subject;
• Estimate project schedule and staffing needs;
• Inform the auditee of the project and the schedule;
• Outline potential membership of a Legislative Advisory Group;
• Develop preliminary scope and objectives and study plan;
• Develop strategies for building an active “audience” for the study is a key part of this scoping stage, as well as for the operational stages of most JLARC studies.

Outside experts are sometimes engaged at the “front-end” of our work to offer a different perspective on how a given subject is perceived and handled beyond our state and who assist in developing analytical approaches.  In the course of conducting our studies, we often use consultants with particular expertise to handle portions of a study that can benefit from their expertise.

Helping New Staff Get “Up to Speed”:  How has your office helped new staff “learn the ropes”—for example, learning your office’s writing style or what constitutes a “finding”  Have you found ways to help new staff feel comfortable and “fit in”?  (Source:  June-July 2001 NLPES survey)

Michael J. Mayhew, Michigan Office of the Auditor General

1.  We provide 4 days of orientation prior to assigning new staff to an audit.  2.  During their first year with us we provide approximately 6 one day training seminars on basic audit skills including: a. Michigan's accounting and reporting systems, b. basic report writing, c. interviewing skills, d. automated workpaper preparation and documentation, and e. audit sampling.  During their first 6 months with us we charge only 50% of new staff hours to the projects to which they are assigned, the balance is charged to on the job training. We try to assign the new people to an audit in which they will be able to draw on past experience to feel self confident and contribute perspectives to the audit.

As I am sure most audit professionals have discovered, experience is the best teacher.  New staff training has always been problematic in balancing expectations/needs with ability/retention.  There is simply too much variation from audit to audit to expect a new staff member to grasp all the necessary tools immediately, or to not lose heart periodically.  Training for our new staff begins their first two days with a general overview of auditing and the office.  The significant confusion created by this training is addressed with ‘new auditor’ specific training over the next two to three months with formalized one day training for: surveys, programs, and report writing and informally with OJT training.  In addition, new staff are included in all other scheduled training as it arises.  The ‘fitting in’ process, hopefully, results from OJT and various social activities, both formal and informal.  We have been fortunate in the fact that our existing staff is very paternal toward new staff members.

We recently completed a procedural manual which covers everything from office attire to writing style.  It provides appendices that give examples of report formatting, writing style, etc.  The manual has sections that discuss how to conduct interviews, risk assessments, elements of a finding, audit plans, confidentiality requirements, etc.  In numerous places the manual refers to the Government Auditing Standards developed by the GAO so that personnel can see the consistency between our procedures and Generally Accepted Government Auditing Standards.

I wish we had some "best practice magic" here.  We have started bringing in some outside training help on writing, both to improve our style and to help with new and even somewhat experienced staff better understand and match our writing style.  Teaching people how to be appropriately analytical (assuming we correctly hired analytical types) and find problems and the associated "findings" is difficult at best.  We orient them by example and by anecdote.  Then we rely on project managers to guide and teach new analysts on their first projects.  Supervisors assist as potential findings are brought up for discussion at internal meetings.  We use these meetings not just to guide the project, but to help guide the analysts how to best accomplish their work.  As a relatively small agency (about 31 staff), we also use social functions to help new staff acclimate to us and the process.

We have developed a standard process for orienting new employees which begins with the use of a "buddy" and proceeds through formal training as well as OJT.  The OJT includes job shadowing such activities as entrance conferences, exit conferences, internal quality control reviews, etc. early after the new employee comes on board rather than waiting for these activities to occur through the course of the project.

When new staff start, we have a 5-day in-house orientation to all phases of our work and office "lifestyle."  We try to have as many of our staff as possible involved in conducting part of the orientation (one might do the policy manual, another explain how we keep time calendars, another explaining our referencing process, etc.).  All staff try to reinforce the idea that new staff may ask any question of any staff member and if that person don't know the answer, they will assist in locating the relevant staff member who does.  We also have a donut reception for new staff to meet everyone.  Although the new staff member's first supervisor is probably most responsible to assist in training, everyone on the staff is generally "watching out for" the new staff to assist them whenever possible.  This informal process works because of our small staff size; it is not likely that the new person will be overlooked!

I don't think we've got answers on this one that could help anyone else; am looking forward to learning from others.

For staff level auditors we instituted a 6 month on-the-job training program that puts new staff in small groups with a trained and experienced on-the-job training instructor.  This group goes through a series of formal training programs and then is assigned to a project as one FTE.  Upon graduation, each auditor is then assigned one on one to projects.  The OJI program has been very successful at helping staff "fit in" and "learn the ropes."  We also assign a mentor manager to every employee so that there is one person to go to regarding their professional development and career growth at the SAO.

Assign a buddy to each new staff person, have a team lunch with new staff on their first day, new staff training within 3 - 4 weeks of being hired, frequent team meetings, brainstorming-style meeting to formulate fieldwork objectives (free-flow exchange of ideas where everyone is comfortable to provide input), just-in-time training for key skills (e.g. writing and indexing) whereby the staff person receives formal training on the skill a week or so before having to perform that skill.

Our office recently implemented a mentoring program for new employees in which each new staff is assigned a supervisor to review the audit process, office procedures, etc.

We have an extensive orientation program, including a seven page checklist that guides new staff (as well as those doing the orienting) through their first week.  We have also found the NCSL Skills Development Seminar to be useful for new staff.  We allocate time during the employee's first week to study our Research Guides, Administrative Manual, Standards of Conduct, and other orientation materials.

We provide all new employees with two weeks of training during their first year.  This training covers topics such as interviewing, work paper preparation, performance auditing, and research techniques.  Although this training provides the basics, most of the learning takes place on the job.  In some instances we have allowed new employees to "shadow" an experienced employee on an audit.  In this situation the new employee is merely an observer and does not have any specific audit tasks for which he/she is responsible.  As required by the Yellow Book, we also provide 80 hours of continuing education every two years.  Aside from the standard classroom and on the job training, we have implemented other programs to develop new employees and help them fit in.  For example, we recently started to track the experience auditors gain on each assignment.  We will use this information to identify training and development needs.  To help auditors develop their writing skills, we have had staff work one-on-one with the editor.  This individualized tutoring allows the auditor to work on his/her specific writing problems.  Finally, we have established a mentor program for new employees.  This program is designed to help new employees fit in.

1)  Develop a detailed handbook that describes basic philosophy, research approach, methods, and procedures.  2)  Develop a "style manual" that outlines report writing and production policies and procedures.  3)  Develop a "checklist" of activities and subjects to be covered when each new employee comes on board.  4)  Assign a mentor to assist each new employee.

We try to be clear about certain stylistic conventions but learning how to sift massive amounts of data into key findings that can be read quickly and clearly without distortion is something that we find takes a long time (and a lot of tries) to learn.  Putting new staff on teams with experienced staff and having regular office-wide staff meetings, as well as celebrating birthdays and other key life events, helps somewhat in the "fitting in" process.

1)  A series of training courses, delivered in-house, over the course of an employee's first year on the job.  Shorter workshops are offered during "Orientation Week."  2)  On-the-job training with project supervisors, who have a check-list to use in ensuring a well-rounded set of experiences in the first year on the job.  3)  Staff developed a "mentoring" program that matches non-supervisory "mentors" with new-employee "proteges."  Proteges look to mentors for information not specific to their audit, but pertinent to workplace culture and state employment.

We have a very elaborate online orientation manual that we require all new staff to review within the first several days of employment.  In addition, our staff typically uses the "on the job" training approach by assigning newer staff to work with more seasoned staff.  Also, our publications coordinator works very closely with new staff to ensure their compliance with the PEER format and writing style.

We have a new employee's orientation program for all new staff in which they meet with several different staff to talk about their projects.  I also take each new employee to lunch on their first day so that I can get to know them some and also seem more "approachable."

New staff are assigned a "mentor", someone who is experienced and already knows the ropes, to help them adjust and answer questions along the way.  We have frequent team meetings where we discuss issues which may develop into findings and strategies for writing findings. New staff are given an opportunity to work with at least two different team leaders during their first year, which gives them an opportunity to learn from more than one teacher.  We have a small office and do our best to foster an atmosphere of teamwork.  We usually have an office-wide luncheon to welcome a new staff member.

We get them involved in the entire process from the start.  We don't just let them work on the initial review activities or labor intensive areas. They are involved in meetings - discussions with on-going quality control - writing memos - writing report sections - all under the supervision of a more senior staff member.  I guess talking things out works the best.

We have a 1-week orientation process, followed by another period ( a couple of days to a weeks or more) in which the rookie completes part of a series of self-paced training workbooks on topics like writing, interviewing, surveys, findings, etc.  We also assign a buddy/mentor to each new person to help them learn the ropes.  Usually, we take the new person out to lunch on his or her first day.

We have developed a comprehensive training program for new staff, consisting of mentoring, formal training, and on-the-job training.  Each new analyst is assigned to an experienced senior analyst who serves as a mentor (we have developed protocols and training for these mentors so that they understand their roles).  The mentor's job is to take the employee under their wing and make sure that they learn what they need to be successful.  This mentor is typically on the analyst's first project team.  In some cases, we also assign a second mentor who is responsible for taking the analyst under their wing for the first couple of days to show them where everything is in the office (we do this if the team mentor is unavailable when the new analyst arrives). Second, we have developed about a 40-hour series of classroom training courses that the new analyst will take during their first three months on the job.  These consist of a series of 3-4 hour workshops that the new analysts will take (we offer this about 3 times a year).  The workshops are taught by senior staff and cover topics such as teamwork, interviewing, writing, fieldwork, research methods, and working with the Legislature.  Third, we have developed an orientation program for new analysts that is available through our intranet; new staff are expected to go through this material in their first week on the job.

We have an extensive in-house training program that touches upon most of the areas of responsibility that new staff will have.  It is our goal to ensure that all new staff attend certain basic training classes as quickly as they can post-hire.  This is sometimes challenging, however, since we hire staff continuously and some classes are taught only once a year.  We also have an internal committee that organizes pot lucks, charity drives, and other types of events that are held throughout the year.  These serve as a forum for new staff to expand their network and meet people in the organization other than their immediate teammates.

We have a "buddy" system.  New staff are paired off with staff who have at least six months or more experience.  Also, due to limited space, our staff are seated in 2-person cubicles.  They get familiar real fast!

Staff members work in project teams, typically on one or more studies at a time.  New staff members are typically assigned to work on projects with at least one experienced auditor.  And, other experienced staff not working directly with new persons assist in their on-going orientation.  Training resources are available within the larger framework of Washington State government, as well as through professional conferences and meetings on specialized topics.  The Committee actively supports professional development and training for its staff.

Staffing Project Teams:  What lessons have you learned about how to properly staff an evaluation project? (Source:  June-July 2001 NLPES survey)

Michael J. Mayhew, Michigan Office of the Auditor General

We try to establish audit teams that are diverse in terms of experience, gender and race.  We track work histories and assign staff who have particular interests in a given program. We periodically survey the audit staff to determine such interests and we encourage staff to volunteer if they are interested in a particular project.  We start out with conservative time budgets and will increase them if the need is justified.

Based on the size and complexity of the agency under Sunset review, as well as the number of agencies set for Sunset, we set a schedule for the 14 months we have to get the 25 to 30 agencies reviewed.  For example, a large complex agency may be assigned a team of five analysts for six months.  A small licensing agency may have two analysts for two or three months.  We attempt to place the most experienced project managers and team members on the complex reviews.  We also look at prior experience and compatibility of the team members.  We will sometimes make adjustments during the 14-month review cycle based on changing circumstances such as resignations or a significant change in the complexity of a review (such as when major problems are found beyond original expectations).

A few years ago, we acknowledged the need to account for a "fudge factor" when estimating calendar time needed to complete a project. It is the sum of all non-project related time (leave, special projects, training, etc.) that does not add into project hours, but can influence the date of report completion.

The primary lesson that we learn over and over is that our project staffing is much more of a reaction than a weighed decision.  Legislative needs far outstrip our staff size and ability to quickly initiate and turn over audits.  Our inability to initiate audits immediately upon request can also be an asset.  The time between request/prioritization and actually beginning fieldwork can be spent planning.  Management can better define legislative needs, fit in the best supervisor for the job, and, in some cases, free up staff for major jobs.  Staffing decisions are often based on manager/supervisor knowledge of the subject, familiarity with agency staff, agency activities, audit methodology, and auditor abilities.  We are often not only concerned with getting the best people on the audit but also getting the best, all-round training for audit staff.  Often staffing is determined by time constraints - how many hours will be required.  Unfortunately, the decision of staff resource allocation is determined by asking the questions:  who is available and when is the audit information required?  Audit scope becomes the critical question.

Generally, smaller is better. When teams are greater than 3 (project leader and two analysts), costs go up and timelines tend to get extended, not reduced.

We have found that smaller teams (1-2 people) work more efficiently than large teams, and we work at limiting the scope of our projects to facilitate the timely completion by a smaller team.  If additional significant issues are identified during the project we will initiate a second project rather than extend the initial project.

The number assigned to a team generally varies from 2 to 4, based on the scope of the project.  All of our staff are generalists and should be able to undertake any project, but we try to assign auditors with the most relevant experience and background to a project (someone experienced in auditing colleges to a college audit or someone experienced in economic development to an audit related to economic development, our CPA to an audit that will require financial analysis, etc.).  At the end of the planning phase of a project, staff determine the audit budget based on how long they think it will take to do the plan (often team members independently estimate the days it will take, and get together as a team to arrive at a team estimate).  If there is a deadline externally imposed on the audit (such as the requesters needing to have it at the beginning of the legislative session), the plan may need to be adjusted to meet this deadline.

It's terribly important to rotate assignments so that no two people have to work with each other project after project, without a break.

We have found that staffing projects is best handled by the managers responsible for subject matter area or area being reviewed.  These managers have the best understanding of what the engagement will take and which staff are best suited for the work that is planned.  Regarding estimated hours, we put a base number of hours we believe the work will take in our workplan.  Once the teams have begun planning and scoping is complete, hours are adjusted as necessary.  Hours can also be changed at any time throughout the process as long as it is approved by the manager and director. We expect projects to be completed within 5% of budget so adjusting project hours is very important.

Over 4 on a project seems to end up counterproductive.  Each project is assigned a senior/team leader who is overseen by a manager, and additional staff members are assigned based on the audit hours budgeted and staff availability.  Two- to three-person teams seem most effective but again it depends on the audit budget.  Estimating hours per project is based on historical hours budgeted for same or similar projects and whether those hours were appropriate and our goal to keep overall average hours for all audits at 2,500 or fewer hours.

Three major criteria guide staff assignments:  1) technical needs of the audit must be met;  2) rotation of staff is pursued (working with different team members; and  3) consideration of preferences/interests of staff.

This is more an area of art than science.  Consequently we do not estimate hours per se.

We always make a preliminary assessment of the resources needed on the audit based on prior experience and what we know about the job at the time.  However, the key to success is flexibility--being able to add or delete resources when circumstances dictate. -- As for staffing the job, we attempt to get the proper mix of experience levels.  Because our office is relatively small compared to the need for our services, we do not have the luxury of specialization.  Instead, when we staff an audit, we assess the audit's complexities and match the auditor's skills and experience to the job.

1) Consult with staff individually about possible assignments to get project and team member preferences.  2) Most projects require 2-4 staff, the more complex and data intensive, the more staff are generally required.

We know we need a mix of people (e.g., those who see the big picture and those who focus on details; those who have experience with the topic and with the methodology; those who like to talk with stakeholders; etc.)

Proper staffing is a calculated balancing act that considers:  1) staff availability - It's typically desirable to assign all staff as early as possible, without removing staff from other teams.  2) skills necessary to complete project tasks - Sometimes specific skills are recognized as essential very early in the project.  3) supervisory experience available - Supervisory skills must be well-suited to the complexity of the audit and to the development needs of the staff assigned to the project.  4) speed with which the project must be completed - For staff development purposes, it may sometimes be advantageous to allow a relatively small team work for a longer period of time than to assign relatively more staff to complete the project more rapidly.

Because PEER employs a multidisciplinary staff, we typically have staff with the appropriate skills for various projects.  Over the recent past, we have assigned projects to small teams--e.g., one to three people.  We usually base the time projection for projects on similar projects that we have completed in the past.

Although I tend to assign people according to their expertise and interest, I have found that sometimes assigning someone who isn't as knowledgeable sometimes helps provide a fresh approach.

We attempt to diversify as much as possible, mixing staff of different experience levels and backgrounds.  We also try to rotate staff so they are working for a different team leader on each project (as much as possible).  Sometimes making sure you have the right personality mix is nearly as important as the mix of expertise!  Working in close proximity to other staff members for several weeks at a time can create special considerations.  We typically work in teams of 3 to 4 evaluators.  We have found that larger teams are usually less efficient.  We have tracked time on previous projects and can estimate staff hours with reasonable accuracy.  It is important to build staff leave time, training and time for agencies to respond to the draft report into the total time estimate.

There is no good answer for this one. The only lesson is know your staff abilities and knowledge before you staff a project - then follow through on the design you implement.  With less experienced staff you will need more supervision - account for it.  For those staff who struggle with writing - provide more contact at the end of the project. The most important lesson is stay flexible.  Let the managers and supervisors be managers and supervisors.  Allow them to exchange resources - coordinate with each other - and make sure they explain why they are making changes.  If staff have multiple projects, let them fill in the "time gaps" in one project with work in another.  Having a major project and smaller projects on the table for each person helps.

Nothing beats experience here.  In most cases, the project deadline is fixed (must be completed before session, etc.).  We believe that most projects can be done by 2-3 people, with 2 the preferred number.  We have learned that putting more people on projects rapidly reaches diminishing returns; the amount of meeting time and internal team supervision needed on large teams increases exponentially with team size, and not much productive capacity is added after about 3 people.  We will at times assign more people to assist a team in tasks such as surveys; these people will then be released to their other projects when the task is done.  Once the team size is determined (and the deadline is already fixed), then hours readily fall into place.  We ensure that each team has the skills needed to do the job.  This is easier in a large office such as OPPAGA.  We plan our work out a year in advance, and let staff "bid" on projects, believing that staff work better on projects that they care about.  To the extent possible we honor these bids, but will not put more people than needed on a project and will ensure that all assigned teams have the required skills.

Staffing projects is always a challenge.  I don't know that we've learned any particularly insightful lessons here, we just try to stay flexible.

We attempt to ensure we have the necessary skills on each team.  Primarily, we try to have at least one "numbers" person on each team so that we can be sure that the financial aspects get covered.  It is often difficult to estimate the number of staff hours, however, we do try to set time periods for completion (i.e., 6 months from start to end) depending on the complexity and scope of the audit.

JLARC is one of the smaller legislative performance audit functions in NLPES.  Consequently, staff are sought who are “generalists” and can switch their study topics with ease from one complex area of state government to another.  Project teams are assembled largely based on the timelines of the assignment, matched to availability, sensitivity and skills of individual JLARC staff.  Recently, JLARC has been successful in attracting to its staff individuals with extensive experience in working for other policy and fiscal standing committees within our Legislature.

Measuring your agency’s performance:  How does your office assess the impact of its reports and how does it report this to legislators?  What are the most important measures of your agency’s performance?  (Source:  June-July 2001 NLPES survey)

Michael J. Mayhew, Michigan Office of the Auditor General

We estimate economic impact of our audits.  We track number of recommendations and percent agreed to and complied with by the auditee.  We establish goals and track related results in terms of efficiency of the audit projects and timeliness of the audit reports.

We have what is called an "Update" process that requires us to revisit the agency and report the agency's compliance with our recommendations.  When serious problems have been identified, the update report shows the extent to which the problems have been resolved.  This in turn measures our office's effectiveness.  However, a limited number of legislators (those on our committee) are aware of our success. There is more that we can do in this area and I am interested in knowing about other state's practices. We are working on developing a scrap book of newspaper articles that reported on our audits.  We also want to list the agencies that have been terminated due to our audits.  However, we need to do better in compiling the monetary savings that result from our audits.

We track the number of staff recommendations that are approved by the Sunset Commission, and then track the success of the Commission's recommendations in the Legislative session.  We also measure and report the fiscal impact of Sunset recommendations.  For non-statutory recommendations we track their implementation over the following two years and report on the success (or not) of those recommendations.  In a non-formal measurement, we talk with key staff and legislators after each biennium about what worked in the Sunset process and what they felt did not.  We use this information to make adjustments to our process to try to achieve greater future success.

The committee monitors the impact of reports through six-month progress reports from agencies.  Staff compile these reports and track the number of recommendations implemented or in progress and the resultant cost-savings and revenue enhancement.

Agency performance is measured formally in an end of year report that identifies estimated savings per dollar spent, general demographic information, and report recommendation implementation.  We have also used questionnaires to determine if we are appropriately addressing legislative needs and performing at acceptable levels.  The most important measure of the office’s performance/success is informal legislative feedback and the most important feedback is the number and quality of legislative requests.

Most important measure:  unsolicited feedback from legislators.

We measure our success in terms of the percentage of recommendations that are implemented.  We are aware of the specific dollar impacts of our reports when there are hard dollars (actual savings or avoided costs) involved but do not formally track them.  We do not report the implementation information to the legislature.

All audit requesters are surveyed post-audit to determine whether the audit met their expectations.  The results of these surveys and other performance measures are reported to legislators in the agency's annual accountability report.  The two major measures of our results have been Financial Benefits Identified and Number of Recommendations (both are reported using a four-year average).  Beginning in FY 00-01 we will also report Financial Benefits Realized and Recommendations Implemented (based on data that will become available from our follow-up process).  Our other performance measures are discussed in detail in our annual accountability report, which is available at our website:  www.state.sc.us/sclac.

Two years after the release of a report, we do a follow-up with the agency to see what if any effect our recommendations and the committee's actions may have had.  This information goes to the committee and based on it, they decide whether to assign new work in that area or not.  The committee has been largely satisfied with agencies' efforts to implement and comply with the recommendations in our reports.

The mission of the SAO is to provide useful information to the legislature.   To gage the usefulness of reports, the directors and state auditor review monthly the reports issued and rate the usefulness of each by discussing who has used it, what impact it has had or may have in the future.   This information is plugged into our agency-wide balanced scorecard which is the tool we use to manage and measure the achievement of our mission and strategies. This approach links strategy to performance measurement and is used at the agency as well as individual project level.   For each strategy (agency-wide level) or project (project level) we develop objectives, measures, and targets and then manage toward those on an ongoing basis.  The project level scorecards roll-up and provide information about how our agency is doing at the agency level.  This gives us critical information about what is working, what is not, and if done timely, gives us the means to detect this early and make the appropriate changes.

Our office maintains a strategic plan which contains specific goals and objectives.  Each year we publish an annual report which details the audits completed.

We perform follow-up work every 6 months until all recommendations have been implemented or 2 years, whichever comes first.  We report whether the agency has implemented, is in the process of implementing, or has not implemented and report this to the legislative committees.  We also have corresponding performance measures in our strategic plan.

Every two years, our agency produces a report to the General Assembly that measures the impact of studies.  The report is prepared based on systematic follow-up contacts with affected agencies.

We measure our performance in a variety of ways.  Most importantly, we determine the dollar savings generated by our recommendations.  In addition, we keep track of the number of recommendations made and implemented each year.  We also keep tabs on the number of hours we spend on audits each year; this is most meaningful when the same audit is performed year after year.  We track legislation introduced and chaptered.  Also, we track the number of times that staff provide testimony to various legislative committees.  Finally, we monitor media coverage.

We write "impact memos" at the end of each project summarizing presentations, hearings, briefings, press coverage, bills introduced and passed, and other legislative and non-legislative action that results from each study.

[We examine] how and when our reports get used (e.g., our recommendations are adopted by agencies or are put into legislation) and how often, by whom, and under what circumstances our reports get cited.

The key indicator of our agency's performance is legislative demand for audits that significantly out-strips our agency capacity to complete all the requests.  Other measure of agency performance include:  the range of testimony and discussion at public hearings of our reports; reference in legislative discussions and debates to our audit reports; and inclusion of our audit findings and recommendations in our state's omnibus budget bill.

The Mississippi Joint Legislative Budget Committee only requires PEER to report on three types of outputs--number of reports released, number of legislative assistance requests, and number of background investigation requests.  PEER assesses its impact (at least anecdotally) through our Director's contact with the legislative leadership, committee chairs, and individual legislators.

Our agency has not measured our performance with respect to program evaluation work.  We do this work part-time (6 months a year) and have concentrated our efforts on using all available staff hours to complete as many projects as possible.  It is something we would like to work toward and would be interested in other responses to this question.

We perform annual follow-ups of each audit.  This provides a view of the response to recommendations.  These follow-ups are presented to the Audit Committee and are available to all legislators.  We discontinued our annual report years ago.  For its marketing impact it took up too much time.  In house we track whether or not recommendations are implemented -- why they were or why they were not -- are there successful or unsuccessful trends in certain types of recommendations - managers maintain contacts with agency management to gauge our impact and acceptance at the agency.  We are trying to concentrate on outcomes not output.

We prepare a follow-up report each year, updating our committee on the extent to which agencies have implemented the previous year's audit recommendations.

This is critical!  Offices must be able to readily demonstrate their value to their legislature or at some point they will get into political trouble.  OPPAGA maintains data on several performance measures.  As outputs, we track the number of reports we issue, the number of Florida Government Accountability Report (FGAR) profiles of state government programs maintained, the number of visits (not hits) to our website, and the number of subscribers to the Florida Monitor Weekly, our weekly electronic newletter that identifies and summarizes good policy research reports issued during the past week.  As outcomes, we measure the potential fiscal impact of recommendations we issue, the fiscal impact of recommendations that the legislature and agencies have implemented, and OPPAGA's operating cost per hour compared to that charged by private consultants.  We report this information to the Legislature twice a year, as part of our operating budget and as part of our annual report to the Legislature.

Our office tracks its performance against three goals. The measures associated with these goals are: percentage of recommendations accepted by audited agencies, percentage of recommendations implemented by audited agencies, legislation resulting from audits, and dollar savings identified in audit reports.  We report our performance on each of these measures through our Annual Report, which is available to both legislators and members of the general public.

We do follow-up studies periodically to determine the number of recommendations that have been implemented.  We report this to legislators in 2 ways:  (1) our annual report; (2) via the follow studies that are done about every 3-5 years.  We measure success by the number of recommendations implemented; dollar savings/revenue enhancements realized.

JLARC is the only legislative entity in Washington that reports performance measures through our state’s system for such reporting.  We use two measures:  (1) the proportion of recommendations made that are implemented; and (2) one-time and annual avoided costs (savings) resulting from recommendations implemented.

Evaluating staff performance:  What lessons have you learned about useful ways to evaluate the performance of your agency’s staff and give feedback to employees?  (Source:  June-July 2001 NLPES survey)

Michael J. Mayhew, Michigan Office of the Auditor General

Each supervisor evaluates each staff auditor at the completion of each audit project.  We encourage staff to evaluate the supervisors on an informal basis (just between the two parties).  However, we solicit input from the staff concerning people to whom they have reported only to be used in our promotion process.

We evaluate staff performance through a rating system on seven components of the auditing process.  These components are:  1) Survey Phase;  2) Management of Workpapers;  3) Writing;  4) Indexing Reports;  5) Referencing Reports;  6) Oral Presentations; and  7) Attitude.

We are presently in the process of revising our evaluation and feedback system.  We keep making adjustments but do not feel we have nailed down a system that meets both legal/statutory requirements, yet provides appropriate and useful feedback for analysts and managers to improve their skills and approaches.  We've tried elements of the 360 degree feedback methodology with mixed results.  We try to separate the feedback from the compensation question but in a small agency that is quite difficult (and everybody knows it).  So, I will be interested in any suggestions arising from this survey.

We use a Performance Expectations and Review form to establish expectations and provide feedback on a project basis.  Expectations are agreed to at the start of a project and performance is reviewed at the completion, starting with self-assessment and followed by project manager review.  In our small office, the director also reviews all Performance Expectations and Reviews.

The overriding lessons we have learned about evaluating staff performance is that there is no correct system and that, to work at all, it must be on-going.  Evaluating an individual’s performance is a necessary task that is equally difficult for the evaluator and the evaluated.  As a result of the disagreeable nature of evaluation, we utilize a multi-level approach that attempts to defray the negative nature of evaluation on any single individual.  Our staff evaluation system begins with supervisors reviewing performance expectations with each staff member prior to beginning each audit.  The evaluation tool itself is the result of office-wide discussions.  The close nature of our work should result in continuous and timely feedback from supervisor to staff.  Each audit assignment ends with an evaluation assessment using the above mentioned tool that is completed by the supervisor, discussed with the staff member, and reviewed by all members of the management team.  Each year closes out with a summary review of staff where supervisors are asked to identify the strengths of each staff member and any areas needing improvements.  All staff are interviewed and provided the results of this information by the Auditor General.

Our employee appraisal form/process consists of a lengthy listing of skills, abilities, knowledge, etc. which the employee's supervisor comments on in terms of how the employee can enhance their skill, ability, knowledge, etc.  The form does not include rankings and does not focus on failings.  It focuses on what the employees can do to improve regardless of what level of competency they are at.

Currently we have an internal quality improvement team working on developing a new staff performance evaluation system.  This team has obtained instruments from many NLPES offices and also surveyed the evaluation literature.  We expect to implement a new system sometime during FY 01-02.

Again, we're looking for guidance from other states.

We have found that performance appraisal works best when handled consistently and routinely and through a system that is easy to use and understand. We require a formal performance appraisal for staff at the completion of each project where staff work in excess of 180 hours.  Staff are rated consistently throughout the office using a standard appraisal tool for their responsibility level.  They are rated on a system of 1 to 4 with 1 being poor performance and a 4 being exceptional.  A 2 rating is performing to expectation.  Project managers are also required to complete informal performance notes either verbally or in writing throughout the project. These notes are intended to help the staff know how their doing on a more frequent basis and they help the project manager complete the project evaluation at the end of the project.  PMs get into difficult situations when performance notes are not completed when needed and poor performance becomes an issue.

We have developed new performance evaluation forms (a separate form for each level) that clearly tie performance to skills needed for promotion.  We are seeking feedback on these forms to continue to revise and improve them as needed.  Each employee also meets with his/her manager annually to discuss goals and needed development opportunities.  Timely, direct feedback with examples to support proves to be the most beneficial -- both positive and negative feedback.

Don't wait for an evaluation.  Instead, give feedback on an ongoing basis and don't forget to focus on the positives as well as the deficiencies.

JLARC has a comprehensive evaluation process and annual reviews of all staff.  One practice that has tended to dampen rater inflation is our requirement that evaluations be reviewed by the rater's rater prior to giving the evaluation to the staff member being evaluated.  We also query evaluators on bottom line indicators.  For example, "would you seek to have this analyst work on your next project?"  If the answer is "no" or a qualified yes, we go over the evaluation very carefully to determine if it is a good measure of the employee's performance.

Although informal feedback in the form of team meetings, supervisory review notes, and one-on-one discussions is constant and on going, evaluations are the primary form of formal feedback.  We use both upward and downward evaluations.  A downward evaluation is presented to the person being evaluated by the evaluator at the end of each job.  Although prepared at the end of each job, upward evaluations are summarized and presented periodically to ensure anonymity.  In addition to end of job evaluations, we periodically conduct counseling sessions with each employee.  During these sessions, the employee receives a written evaluation, which is a composite of the employee's downward evaluations and management's perception of the employee's progress and standing, and develops his/her annual training and development plan.

We do individual performance reviews at the end of each project.  Each review includes a rating on each aspect of the employee's job description, plus an open ended memo on strengths and weaknesses.

1)  Tie performance evaluation to opportunities for skill development on future assignments.  2)  Ask staff to draft a self-assessment to start the annual performance evaluation process.  3)  A careful assessment of performance requires dialogue between the evaluator and the staff person.  4)  Work from a performance evaluation form that calls for thoughtful articulation of: expectations, demonstrated skills, and future assignments.  Our form calls for text to be drafted on the following topics:  Duties Assigned and Expectations; Planning; Documentation; Analysis; Written Communication; Oral Communication; Follow-Through; Supervision; Work Habits; Summary; Future Activities; and Employee Comments.

Employee appraisals have always been a challenge for PEER.  Because the nature of our work is team- and project-driven, it is difficult to objectively appraise each employee.

Feedback needs to be frequent and immediate whether it's positive or negative. In all of our evaluations we focus first on what has gone well and the employee's positive performance and then focus on improvements that may be needed.

We still use an annual written appraisal form which ranks employees on specific job expectations.  However, we have recently considered going to a verbal employee evaluation with a follow-up memo for the file.  New employees are evaluated at their first six-month and annual anniversary dates.  I have found an annual written evaluation on the same form for long-term employees is not exceptionally valuable.

Do it on an everyday basis.  This needs to be a team leader/manager role.  Don't use formal evaluations as an excuse to put off talking to staff until a particular time.  Make sure there are discussions of review topics and progress. No real surprises.

We evaluate at the end of each audit AND annually.  The after-audit evaluations provide more chances for formal feedback.  We tell supervisors that nothing on the evaluation form should come as a surprise:  there should have been verbal feedback throughout the audit, and the written document should only summarize and document that verbal feedback.

We have done a lot of work on this and have developed a strong evaluation system.  At the end of each project, teams get together with a senior analyst and conduct an end of project debriefing, which includes reviews of each person's contributions to the project.  Annually, each employee undergoes a formal evaluation.  This includes 360 feedback from the analysts that they have worked with as well as management feedback.  The key is to be honest and open.  If someone has a problem it has to be discussed; trying to hedge feedback to avoid hurting someone's feelings doesn't do anyone a favor.

Our formal internal evaluation process is designed to give staff feedback on their performance at least twice a year.  We evaluate staff on a number of benchmarks including their writing skills, analytical abilities, oral communication skills, and teamwork.  The evaluation instruments we use were designed specifically for our audit environment.  As such, they are highly relevant and useful for helping staff understand the areas where they are doing well and those where they need improvement.  We also encourage supervisors to provide daily feedback to staff via various other mechanisms (e.g., team meetings, workpaper reviews).

Honest and direct.  We provide staff with daily feedback during on-the-job training; end of job written appraisals are done; and complete annual appraisals.  The audit manager and the auditor-in-charge discuss appraisals before they are given to staff auditors so that the audit manager and AIC are on the same page.  Managers can force honest appraisal ratings before the document is given to the staff.

Staff performance reviews are conducted on an annual basis, with the Legislative Auditor participating in all reviews of audit staff.  The results of these annual reviews are used as part of the decision around merit-based salary increases.

Program Evaluation “Hot Topics”:  What are some good, timely topics that other states’ legislative program evaluation agencies may wish to consider?  (Source:  June-July 2001 NLPES survey)

Michael J. Mayhew, Michigan Office of the Auditor General

(1)  Day Care—This is an area in which we recently completed an audit which had very high impact in terms of legislative, public, and agency personnel resulting in material improvements to the programs.  (2)  Home Health Care!  We recently completed an audit of this program as part of the NSAA joint audit process.  Our audit contained three material audit findings and we believe that this area is ripe for future legislative interest.  (3)  Law Enforcement Information Network (LEIN)—We recently completed a performance audit in this area which had high impact in terms of identifying significant risk to the public and generating legislative interest and action.

(1)  Child Protective Services—This is an area where sigificant problems can exist.  We found significant problems.  (2)  Child Support Enforcement—We found a variety of problems in enforcement of child support.  It is a complicated process because mistakes can occur at the agency level, the court system, and with the parents themselves.

(1)  E-government—We find this issue crossing many of the agencies under review and have recently made recommendations to centralize certain elements of e-government, such as e-procurement.  (2)  Competitive government (i.e., privatization)—While certainly not a new topic, budget constraints will likely lead to a new round of private sector competition initiatives.  Using best practices from the last few years, the efforts to evaluate appropriate functions for competition should be more accurate and quickly accomplished.  (3) Technological impacts on government operations—This can range from privacy issues to reducing the need for warehouses to improving telecommuting opportunities.  Technology is changing the way agencies do or should operate.  Evaluating agencies' ability to optimize technology and recommending ways to improve is an area I see us getting into.

(1)  Tax "gap"—A tax gap exists in every state and is generally always of interest.  This study, while methodologically challenging, also results in large potential savings and revenue enhancements for the state.

(1)  Performance-based budgeting—What is its effect on the budget process and agency performance?  Also, performance-based budgeting is expensive to implement—is the expense justified?  (2)  How is the Internet being used to improve performance?  What factors are impeding progress?  The Internet has the potential to improve performance, and impediments should be identified and overcome.  (3)  The effect of outsourcing governmental functions—outsourcing or privatizing governmental functions is supposed to cut costs.  Has it really cut costs, and has agency accountability been affected?  (4)  Computerized communication between evaluators/auditors and legislators—computerized communications with legislators is now possible, but is it being used, and with what effect?

(1)  Quasi-government organizations and special districts—these organizations often lack oversight.  (2) Public education—differences in state and local operations can mean confused programs and misspent funding.  (3)  Tourism promotion—different goals and objectives for state and local spending can effect programs.

(1)  Child Support Collections—lots of mistakes are made.  (2)  Result of state investments in high-tech venture capital funds.  (3) County Voting Systems—the feds appear to have dropped the ball on this one.

(1)  Contract Management (how well does an agency manage a contract to ensure that it receives the level and quality of services it contracted for.  A very significant amount of funding goes towards the purchase of client services from both non-profit and for-profit organizations.  In many cases this funding is inappropriately treated as a grant rather than a contract.  Program personnel do not monitor or evaluate the services received.  (2)  Internal audit resources available and utilized by agencies.  We have found that very few resources are provided for internal audit functions without which management cannot ensure that its program are operating efficiently or effectively.  They therefore end up relying on external auditors such as ourselves to find problems.  Management should be identifying and correcting problems before we arrive.

(1)  State Medicaid policy and management—an area of high costs with potential for improved management.  We have recently completed an audit that was well received and identified cost savings and needed improvements.  (2)  Performance funding/performance measurement system—topic of ongoing relevance in all areas of government.  We recently completed an audit of higher education performance funding, but the topic is relevant to all areas of government.  (3)  Evaluating state information technology use—effective use of information technology is key to improving government performance.  (4) Economic development—an area where accountability has been lacking and is important.

(1)  Archiving Government Records in an Electronic Age—While not a headline grabber or the most expensive item in a state's budget, this issue is emerging as one with monstrous proportions as well as far-reaching and lasting implications.  When we did a study in 2000, we found little to go on from other states.

(1)  Contracting (every aspect from negotiation, administration, to monitoring)—In Texas we have found that many of our state services are now shifting to contractors to perform.  We have also found that we do not do a good job of negotiating, administering, and monitoring our contracts so state dollars are at risk.  We have developed a great contracting model that helps us to understand the risk and focus our work appropriately.  It also helps agencies understand what good contracting should look like.  (2) Risk mapping—Like the U.S. General Accounting Office, we have begun evaluating risk at the statewide level and are preparing a statewide risk map. We believe this to be a valuable tool for discussion with the state leadership of where the state is at the most risk and how that risk is being addressed.  (3)  IT security and privacy in e-government—This area is becoming increasingly more important as states move to providing many of their services on-line.  The seriousness and difficulty of the issues surrounding security and privacy are ones that will have to be addressed by every state very soon.  Many states still have not adequately studied or passed legislation in these critical areas.  (4)  Human capital—A state's human resources are its most significant capital and yet little attention has been placed in this area.  Do we understand this area?  What are the risks we face today and what about the reality of tomorrow as much of our workforce is aging?  This is an area overdue for study and evaluation in the states.

(1)  Human Services Providers.  (2) Unemployment Compensation.  (3)  Education Grants.  (4)  State Childrens Health Insurance Program (SCHIP).

(1)  Interagency coordination for human services—An issue our legislature is concerned with, an even more costly issue if there is poor coordination or duplication, and people won't be adequately served.  (2)  Planning for the elderly population—Nationwide and especially in Arizona, this is the reality and have we adequately planned for it?  Costs to provide the services is high, and can states afford it?  (3)  Electric deregulation as its being planned for or along the way of being implemented.  An area our office will be auditing in the next few years.  An area that has to be done right, the first time, or there are huge consequences.  (4)  Health regulatory boards—Consistency of available disciplinary options, are the boards disciplining violators, is adequate information on licensees being released to the public, are sound investigative and adjudicative practices being used.  Arizona consistently finds problems when reviewing these boards and has found their statutes to be inconsistent.

(1)  Are performance measures delivering what they promise?  For the past decade, fairly substantial claims have been made about the utility of performance measures.  Off the record, however, most people working with performance measures will acknowledge that they are used as background information to "inform decisions" and are not used as decision drivers.  (2)  Is any state using performance measures in the budget process, other than as background information?  With very few exceptions (e.g. certain child support enforcement measures) I see little evidence of performance measures being used meaningfully in the budget process.  (3)  When will all NLPES members' reports be available in a subject searchable manner?  While the web has greatly increased our access to the work of peer agencies, the ability to search other states' work meaningfully by subject area/specific topic would be a killer application.  (4) Legislators are often interested in comparative information on how other states are addressing common problems.  Available data do not always tell the entire story, however.  Certain cross cutting data appear sound but do not withstand close scrutiny.

(1)  Energy availability and conservation—Energy shortages are not quickly or easily resolved.  Consequently, early identification of shortages, evaluation of generator licensing processes and conservation programs is important.  (2)  Welfare Reform—As the economy begins to slow, it will be interesting to see if welfare reform was a success or merely a product of good economic times.  (3)  Education Reform—Are recently implemented reforms, such as statewide testing and reduced class sizes making a difference at the classroom level?

(1)  Affordable housing—High level of legislative and public interest.  (2)  Welfare eligibility and program design—Federal limits are approaching.  (3)  Computer system management and security—Growing dependence on computer systems and growing vulnerability.  (4) Juvenile out-of-home placement—Costs have grown rapidly, and there is a need to look at placement patterns, costs, and outcomes across multiple agencies (corrections, human services).

(1)  Contract management—Seems to surface as a weakness of the executive branch in many of our reports.  (2)  Managing IT—An area of significant expenditures for many executive branch agencies and one in which labor force issues have been of significance in recent years.

(1)  Budget reduction ideas.  (2)  Evaluating or auditing technology and/or computer programs

(1)  Electric Deregulation—Pros and cons; protections that should be in place, etc.  (2)  Services for the elderly—Elderly population is increasing. Industry is suffering from staffing shortages, regulatory issues, etc.  (3)  Development of state data systems—Our state has had several large IT projects that ultimately failed.

(1)  Medicaid prescription drug costs—Our state is looking for ways to curb the rise in Medicaid's prescription drug expenses.  I know Florida did a report on this issue, which I read and shared with our legislature.  (It was GREAT)  It appears to be an area which could result in substantial findings and recommendations for major savings in state costs.  (2)  State budget procedures—We are currently beginning an evaluation of state budget procedures, in particular, lapsed appropriations.  Our state is facing a budget shortfall both this year and next.  Our legislators are looking for areas they can take money from to support their priorities.  Due to federal tax changes and economic conditions, many other states may be facing similar situations.  It is important for legislators to understand how the state budget is administered after it has been approved by the legislature.  (3)  Community Initiatives—In our state more and more social programs have been outsourced to community-based non-profit organizations.  The legislature has appropriated funds to be used in flexible, creative ways to curb welfare dependence, improve outcomes for children, and a host of other objectives.  In so doing, many decisions regarding how the funds will be spent have been made at the local level.  The state has lost some control and accountability over the expenditure of these funds.  Funding sources for programs have over-lapped and it has become difficult to determine the total cost of some initiatives as well as the effectiveness of them.  This may not be a trend nationwide, but it does seem to be a trend within our own state.  (4)  Large IT projects—Our state has encountered a few very large IT projects which have gone considerably beyond the estimated time-frame and considerably exceeded the original budget.  By the time agencies come back to the legislature seeking more funding, they are mid-stream in the project and it is difficult to say no.  Evaluating some of the projects could lead to more caution in the future by the legislature in approving the initial funding and by state agencies who plan projects.

(1)  Management of Geographic Information System (GIS)—An area that has potential to provide agency managers and the public with useful data.  Who's involved?  It should be more than natural resource-based agencies.  An area that has potential to provide agency managers and the public with useful data.  (2)  University Physical Plant Operations—Rising costs of energy and maintenance costs.

(1)  Medicaid efforts to collect from third-party insurers—We found that in many cases Medicaid was being tapped to pay bills that private insurers should have paid.  (2)  Weights and Measures—We found that our weights and measures program was potentially allowing consumers to be cheated in some cases.

(1)  Impact of privatization on state government programs—Privatization is occurring at an increasing pace, but there has been little study of the actual impact on service quality and costs.  (2)  Use and maintenance of state vehicle fleets—These studies typically find that major cost savings can be achieved through better controls on who gets cars and volume purchasing of vehicles and maintenance services.  These studies are always hot topics because they are "good government" studies whose recommendations are politically easy to implement.  (3)  Growth management—A messy subject, but a hot topic as many states are struggling with how to pay for and control growth.  (4)  Cost controls in Medicaid.  Medicaid costs are starting to rise nationwide, creating large budget problems for states.  Analysis of reasons for these cost increases and cost control strategies can have huge impact on states.  OPPAGA issued a report on prescription drugs usage this spring; the Legislature adopted our recommendation for a drug formulary and saved $225 million.

(1)  Contracted Services—What value is added by contracted services?  Is it REALLY cheaper?  (2) Information Technology—Does current use of technology improve communication and information sharing among agencies?  (3)  Economic Development—What do states receive in exchang