2002 NLPES Surveys

2002 NLPES Surveys

Internal review of findings: What procedures has your office found to be effective for ensuring that draft reports are accurate and fair before they are sent to the agency being reviewed? How is the work of individual staff reviewed and verified prior to its inclusion in a report?

(Editor’s note: Some of the approaches cited by our respondents included: review of an evaluation’s documentation and presentation by audit managers, legal staff, quality assurance staff, editors, and/or peers from the audit agency who did not work on the evaluation; cross-referencing of factual statements in the audit report to working papers; ongoing management reviews of research plans, audit team progress, and preliminary summaries of findings and recommendations; creation of legislative advisory groups for projects (Washington);and self-monitoring by the staff who prepared the draft audit.)

Ken Levine, Texas Sunset Advisory Commission

We have a multi-layer review process at Sunset. First, before the analyst writes a first draft, the analyst pulls together an "issue results document," essentially an outline of the findings and recommendations that will be made. The analyst, project manager, supervisor, and often the director will meet to discuss the results and provide guidance into the content of the upcoming draft. Project managers then are the first, and perhaps the most important layer in the review process. The project manager performs a developmental edit of a first draft, focused on content, sufficiency of support, logical presentation of information etc. Grammar and style are not the focus of this edit. The project manager then reviews the revised draft with more of a copy edit approach, checking format, style, grammar etc. The supervisor reviews the draft once the project manager feels it is ready for publication (to the agency) from a content standpoint. The supervisor looks again for adequacy of the support for problems found, logic and workability of the recommendations. Essentially the supervisors apply their experience to the issue to make sure it still works, and works well. Finally, the director reads and provides comments on the draft before it is sent to the agency.

Christine Chavez, New Mexico Legislative Finance Committee

The New Mexico LFC performance audit manual requires that:

The audit team leader is responsible for ensuring that the working papers support the draft report. Findings included in the draft are cross-referenced to the supporting documentation in the working papers. The auditor team leader is also responsible for supervising the fieldwork, which includes verifying that the working papers contain sufficient, relevant and competent evidence to support the work performed and the findings issued.
The audit manager work closely with the team leader. The audit manager is responsible for ensuring findings are accurate and the working papers support the draft report.
The director and/or the deputy director are responsible for final review of the draft report to ensure it is objective, convincing, clear and concise.

John Sylvia, West Virginia Legislative Auditor’s Office

Before any report is presented to our legislative committee, it goes through an extensive review process that includes the project manager, an independent analyst, the Division Director, the Legislative Auditor, and the auditee. Every report is reviewed for style, flow, tone, content, etc. by the research manager over the project, then by the Division Director, and then by the Legislative Auditor. The accuracy of the report is verified by a process we call indexing and referencing. Every draft report is indexed by the analyst so that every statement of fact in the report is given an index number indicating the evidence that supports the factual statement. The report is then referenced by an analyst who was not involved in the audit. The referencer looks at the indexed evidence and compares it with the report's statements to see if the report is accurate and consistent with evidence. We ideally would like for all reports to be referenced before the draft is sent to the agency for its review. However, in most cases reports are referenced while the agency is reviewing the draft report. We would prefer that we catch any factual mistakes before the agency does. This is prevented because our time schedule is tight. The agency normally points out errors at an exit conference. The bottom line is that errors, content and tone are reviewed several times to ensure that the report is to our satisfaction before it is presented to the Legislature and to the public.

Kirk Jonas, Virginia Joint Legislative Audit and Review Commission

The work of individual staff members is reviewed by their team leaders and supervisors. Draft reports are subject to an "extended review process." Each draft report is reviewed by the extended review team, which consists of the Director, Deputy Director, both Division Chiefs, the Chief Methodologist and the staff editor. The extended review team meets and provides directions to the study team for the revision of the report. Revised reports drafts are reviewed again by the Director and division chief prior to sending the report to the agency.

Tom Sykes, Washington Joint Legislative Audit Review Commission

We place extensive responsibilities on each of our analysts for thoroughness, accuracy, replicability, and rigor in their analyses and the findings generated from them. Analysts usually work in teams and team members have an equal and reciprocal responsibility for the accuracy of the work of the team. We also use group discussions to bring in the perspectives of those analysts working on other projects, so we get additional perspectives. With most of our projects, we also create a legislative advisory group, composed of key legislators (including those outside of our own Committee) and legislative staff (from policy and fiscal committees as well as research staff from legislative caucuses. We also work with the Governor’s policy and fiscal staff as well as with staff of the agency(ies) under review. Projects are supervised, with all work papers, draft reports, preliminary reports, presentations, etc. receiving substantive critiques before they are transmitted to agencies. We often share completed work papers on technical aspects of our projects with staff of the agency(ies) under review for accuracy. Agencies also get copies of draft reports, absent the reports’ recommendations, to review for accuracy on technical points. We try to do this before we finalize a preliminary report to send to our Committee members. All of these efforts in combination work to ensure accuracy and fairness.

Heather Moritz, Colorado Office of the State Auditor

We have a multi-level review process for ensuring draft reports are accurate and fair prior to sending them to the audited agency. This includes reviewing the work of individual staff well before the report writing stage. Supervisors are expected to meet with staff periodically and also review their workpapers as fieldwork progresses to ensure conclusions are solid and findings are well-documented. We also have periodic meetings with upper management to brief them on findings prior to starting the report writing process. Once a team writes a draft audit report, it is reviewed by the team leader, an audit manager (if applicable), a deputy state auditor, and the state auditor herself before it is sent to the agency. Oftentimes, before the report is sent to the agency, we will also have a "cold" review by someone who wasn't assigned to the audit to ensure the report presentation is objective and compelling. Once the draft has been disseminated to the audited agency, we have one or more meetings with agency staff to get their input about the report's contents and tone. This is also a good way of ensuring the report is presented in a fair and accurate manner prior to its finalization.

Kate Wade, Wisconsin Legislative Audit Bureau

The accuracy of drafts, and indeed the work of individual analysts, is subject to fairly intensive internal review and discussion. The work of individual team members is subject to supervisory review of working papers to ensure findings and conclusions are well-documented; peer-group review of analytical efforts occurs in team meeting discussions, which allow for consideration of the work of individual analysts in a broader project context. During supervisory review and team discussion, the strength of the analysis is tested by asking questions of the analyst—for example, what response the auditee is likely to offer, or whether there are conflicting data that the auditee will present to discredit the audit conclusion. Draft reports are subject to several layers of internal review before they are sent to the agency. First, report drafts are read and critiqued within the team. Second, each draft is shared with a small number of senior staff for review and discussion in a formal meeting. The State Auditor then reads all drafts and meets with each team to test and confirm the strength of each analysis.

John Norris, Alabama Department of Examiners of Public Accounts

Use of style and content manual to encourage standardization (cuts down on errors), supervisory review and editing, review of supervisory editing by preparers.

Jon Wise, Michigan Office of the Auditor General

Our office has a structured report processing track for each audit, and all reports (and related working papers) are reviewed by audit operations management, Quality Assurance staff, and Audit Report Review staff prior to issuance. The work of individual staff auditors is reviewed by the audit supervisor during fieldwork, and is also subject to review (e.g., Quality Assurance review) during report processing.

Philip Durgin, Pennsylvania Legislative Budget and Finance Committee

The director reads each report. Most work is double-checked by another member of the team.

Sylvia Hensley, California Bureau of State Audits

Generally the team leader reviews all staff work during fieldwork; the audit manager may also review some or all of the staff work at this time. In addition staff is encouraged to discuss the results of their audit work with the auditee to ensure they have considered all the facts as well as the auditee’s point of view. Once the report is drafted, but before it is submitted to the auditee, the audit manager risk reviews the report. During this process the manager reviews the evidence underlying the report to ensure the report is accurate and adequately supported. In addition, executive management reviews the draft report up to four times (twice before and twice after risk review) before it is submitted to the auditee. As a final check, the audit team normally holds its formal exit conference before the draft report is submitted to the auditee. This again provides the auditee an opportunity to raise concerns, provide additional facts, and clarify its point of view.

Cynthia Johnson, Nebraska Legislative Research Division, Program Evaluation Unit

We have a very small staff, so the responsibility for assuring accuracy and fairness rests primarily with the analysts who write the reports. This self-monitoring is reinforced by a very rigorous editing process involving two editors, both of whom look for accuracy and fairness in addition to fine-tuning the written product.

Rick Riggs, Kansas Legislative Division of Post Audit

Each workpaper is reviewed by the audit supervisor, and signed off on. The review ensures that the workpaper does what it was designed to do, that audit standards are met, and that the work is accurate and logical. Then the audit manager conducts a similar, but higher-level, review, further ensuring that the work is logical and accurate. The audit team, including the supervisor, manager, and Post Auditor, also have a series of meetings throughout the audit to ensure that the work is focused and on track, that the message of the audit report is fair and balanced, etc. Management review of the draft report (usually consisting of the assigned manager, the Post Auditor, and a cold reader) ensures that the analysis is sound and the tone is fair.

Jim Pellegrini, Montana Legislative Audit Division

All work of the team is reviewed by the team leader prior to inclusion in the report. Major issues have already been formally discussed with the agency/program using memos. The Project Oversight person has reviewed the work of the team leader. The off-site QC person has reviewed conclusions and recommendations and the draft report. Meetings are held with the deputy, in-house legal counsel and the legislative auditor.

Roger Brooks, Minnesota Office of the Legislative Auditor

An outside review team, consisting of the Legislative Auditor, Deputy, and another senior staff person, critique the study's workplan, preliminary statement of findings, and report drafts. Except in unusual cases, this does not involve data verification. (The statement of findings is a summary of the main findings and conclusions, and it is developed prior to report drafting.) For the most part, each team is responsible for ensuring the accuracy of its own report. Often this is accomplished by team members checking each other's work.

Jim Henderson, Idaho Office of Performance Evaluations

The work of individual staff (workpapers) are reviewed by the team lead or another team member. Report drafts are reviewed by the team lead, other team members, and the director. Additionally, we strive to have them reviewed by a "cold reader," which can be difficult in a small office because, at times, the whole staff may be involved on a project. We have also hired and external editor and had our drafts reviewed by a deputy attorney general.

Jane Thesing, South Carolina Legislative Audit Council

Our two main quality control tools are supervisory review and our referencing process. Each auditor's work products are reviewed by her supervisor, which includes a supervisor sign-off of each workpaper. When a report is in draft, the auditors index every fact in the report to their supporting documentation. An independent auditor (on a different audit team) "references" the file, which includes checking each word for accuracy. The referencer also comments on methodology and writing as appropriate. Each manager and the director also review each report (for content, style, fairness, etc.) before the report goes to the agency and our council (governing board). While time-consuming, we have found this process to be an effective control over the accuracy and fairness of our reports.

Gary VanLandingham, Florida Office of Program Policy Analysis and Government Accountability

In accordance with evaluation standards (the "Red Book" as well as the "Yellow Book", our teams cross-reference all factual statements in our reports to work papers. These "doc-cites" are then checked by an independent reviewer to ensure that the report is factually accurate. Report drafts are reviewed at several levels (by the team, the project's staff director, by an independent reviewer, and by management) to ensure that they are well-written and supportable. During the course of studies, teams must also periodically meet with our Team Support Council (consisting of highly experienced staff and management) to discuss their projects, including research plans, preliminary hypotheses, and finding logic.

Stephen Fox, New Hampshire Office of Legislative Budget Assistant, Audit Division

We have an internal review structure that ensures all reports at the draft stage have been reviewed by the supervisor and the director of audits. We also have a panel of "cold readers" (audit staff that were not assigned to the audit) read each draft report. Individual staff work is reviewed by the in-charge auditor for the audit as well as by the audit supervisor.

Barbara Rogers, Wyoming Legislative Service Office

The section manager reviews and edits all draft reports before they go to the director of the Legislative Service Office. The director reviews them before they go to the agency and he often requests other LSO legal staff with experience in the subject matter to review the drafts. Their reviews usually result in some suggestions to fine tune language or fill in the logic, rather than requiring that a finding be re-worked. The director and those staff members are attorneys with long experience in state government, so the feedback they provide is reliable and generally useful. Within our section, every workpaper is reviewed and signed off on by all team members on a given project, as well as by the manager. At weekly team meetings, we incorporate information from newly developed workpapers as we discuss our emerging understanding of the evidence, and how it fulfills (or sometimes, unfortunately, misses the point of) a project objective. Prior to report writing, we sometimes verify by sending certain newly-compiled data to the agency for their review, -- especially if it looks to be sensitive information (i.e. not consistent with what the agency has been telling us). Other than that, and some spot-checking of calculations and assumptions, we expect that if something goes astray, the group review process will catch it. The section manager does a thorough review of the cross-referenced, or sourced, document, but this occurs after the draft has gone to the agency.

Wayne Welsh, Utah Office of the Legislative Auditor

Findings are reviewed by the Audit Supervisor and the Audit Manager. The Audit Manager also checks the work paper references used to support the findings included in the audit report.

Michael Nauer, Connecticut Legislative Program Review and Investigations

Project teams review each other's projects at critical points during the work. Senior staff oversee and review each product deliverable during development.

Joel Barrera, Massachusetts Senate Post Audit and Oversight Bureau

The Director and Executive Director of the Bureau are responsible for all fact checking. We go through the reports with a fine-tooth comb. We put staff members through basically a panel, challenging every fact in the report and making them show the documentation.

Paul Bernard, Georgia Department of Audits and Accounts

We have an on-going review process for our projects. The project team's work (work papers) is reviewed by the team leader and the Supervisor/Audit Manager during the field work portion of the project. Once the report is prepared, a referenced copy is subjected to QC review internally, it is then reviewed by the Audit Manager and subsequently by the Director.

David Greer, Louisiana Office of the Legislative Auditor

All audit work is reviewed by the in-charge auditor and an audit manager. Usually a report conference is held with the director and the report draft is again reviewed. Prior to report issuance, we conduct a selective referencing of the report. This is performed by a senior level staff person who has not been associated with the audit.

Ethel Detch, Tennessee Offices of Research and Education Accountability, Comptroller of the Treasury

Most reports are reviewed by peers for accuracy and also by management. Because much of our work is policy analysis rather than evaluation, it is less directly critical of agencies, so we may not be as focused on fairness as other shops might be. I also see the review by the agency as part of the assurance that we've been accurate.

James Barber, Mississippi Joint Legislative Program Evaluation and Expenditure Review Committee

PEER evaluators (some of whom serve as project managers) are responsible for determining the accuracy and quality of their own workpapers. In addition, PEER has two Evaluation Division Managers who serve as a final check on the documentation associated with each project.

Mitzi Ferguson, Arkansas Division of Legislative Audit

Draft reports are first reviewed by the managers. Then, the Agency is given an opportunity to review findings and inform us of any errors or inaccuracies they note. After the manager is confident the report is accurate, it is forwarded to the Deputy Legislative Auditor. The Deputy Legislative Auditor will review the report and consult with other appropriate management staff on issues pertaining to their sections. For example, the Staff Attorney may be asked to review the report to confirm the accuracy of statements on issues involving state law. After the Deputy is satisfied with the report, a final review is made by the Legislative Auditor.

Dena Winningham, Tennessee Division of Post Audit, Performance Audit Section

Team leaders review supporting documentation of team members' work. Audit managers review supporting documentation for the draft report. Occasionally team members check other team members' work. In addition, top management reviews reports for fairness and clarity.

Maria Chun, Hawaii Office of the Auditor

During the entire course of a project, each team member's work is reviewed by the immediate supervisor. The review is initialed and dated on the working papers. A staff person, who is not a member of the audit team, conducts an independent verification. The review consists of a detailed examination of an indexed office draft of the report to ensure contents are accurate and supported. Also, we sometimes use our exit conference with the agency head as an opportunity to assess with the agency the usefulness of possible recommendations by asking "what if" questions.

Greg Hager, Kentucky Legislative Research Commission, Program Review and Investigations Committee

All audit work is reviewed by the in-charge auditor and an audit manager. Usually a report conference is held with the director and the report draft is again reviewed. Prior to report issuance, we conduct a selective referencing of the report. This is performed by a senior level staff person who has not been assovcaited with the audit.

Craig Kinton, Texas State Auditor’s Office

The SAO uses technical writers to ensure that reports are clear and accurately convey audit findings. Prior to the release of audit reports, SAO staff meet with management of the agency being reviewed to discuss the audit findings. Once a draft of the report is ready, the agency has the opportunity to respond to SAO findings and the agency’s responses are published in the report. The SAO Quality Control Team (QC Team) works with auditors throughout the course of an audit to ensure the quality of SAO products and services. The QC Team ensures that audit findings/conclusions have proper supporting documentation/evidence and comply with all applicable standards.

Alicia Mitchell, Ohio Legislative Office of Education Oversight

Our office conducts its research in teams of two or more staff. Each team has a designated project manager, who is responsible for the timely and satisfactory completion of all stages of the research process for the project, in collaboration with the Chief of Research, and subject to the Director’s overall responsibility for the office.

The research teams develop three types of documents that help ensure the accuracy and fairness of the final reports.

Working papers. LOEO staff use a variety of methods, such as reviewing research literature, conducting interviews and surveys, and searching agency databases to obtain qualitative and quantitative data for studies. These data are analyzed, summarized, interpreted, and placed into working papers. Working papers are the source of documentation and verification for all facts and findings contained in the full report. Working papers are reviewed by the project manager, and in some instances, the chief of research.
Study plan. For each project, a research team, led by a project manager, develops a plan outlining the study questions and how the evaluation will be conducted. The plan outlines detailed explanations of and timelines for completing the initial background research, research design and methodology, instrument development, data collection, analysis and draft report preparation, and final review and publication of the report.
Draft reports. The team produces multiple drafts of each report for internal review before the draft is read by anyone outside of the office. In addition, although the final reports do not contain footnotes or endnotes, an annotated version links every fact, finding, and idea to a working paper. This annotated version is filed in the office so that all facts and findings can be traced to their source.

Our office uses the following five-step review process to review study plans and evaluation reports.

Research team development and review. Each member of the research team reviews the study plan and draft report for accuracy, clarity, and coherence. The project manager incorporates the changes into a “team” draft of a study plan or evaluation report. The research team may have as many as three drafts of each of these documents.

Chief of Research review. The Chief of Research, who is responsible for overseeing all research projects in the office, reviews the team drafts of the study plan or evaluation reports for all aspects of research including the clarity and logic of arguments. The chief of research may review as many as three additional drafts of each of these documents.

Director review. The director reviews study plans and draft evaluation reports after the project manager has incorporated all of the changes the team and the chief of research have agreed upon. The director reviews these documents for all aspects of research, but primarily for the clarity of ideas. The director and the team may discussion two additional drafts of these documents at this stage. The chief of research also participates in this step of the review process.

All staff review. All staff members that are not members of the research team review a draft of the study plan and the evaluation report for all aspects of research, including design, methodology, clarity of ideas, and logic of arguments.

External review. LOEO shares draft reports with the agencies that administer the program being evaluated. LOEO may determine that other stakeholders should participate in reviewing the report as well. As many as ten separate organizations or individuals may participate in this step of the review process. Sometimes, a meeting is held with the agency responsible for administering the program to discuss their written feedback and to clarify and correct any inaccuracies or misleading statements. Occasionally, study plans are subject to external review as well.

After LOEO incorporates all of the necessary changes resulting from the external review process, the report is given to the ten-member Legislative Committee Education on Oversight (LCEO), to whom the office serves as staff. The project manager also gives an oral presentation to the committee. After the presentation of the report to the LCEO, the agency responsible for administering the program and legislators on the committee are permitted to submit comments that are appended to the report.

Other Surveys