NLPES Conference Notes: Verifying the Accuracy of Computerized Data

NLPES Conference Notes

Verifying the Accuracy of Computerized Data
Friday, September 7, 2001

Moderator: Shan Hays, Performance Audit Manager, Office of the Auditor General, Arizona

Barbara Johnson, Senior Analyst for Data Reliability Issues
U.S. General Accounting Office (GAO)

Ms. Johnson has found that auditors will avoid using data if it's too complicated to verify the data's accuracy. However, the Yellow Book, Section 6.62 states that "Auditors should obtain sufficient, competent, and relevant evidence that computer-processed data are valid and reliable when those data are significant to the auditor's findings." The bottom line here: We do not want to issue a product that contains incorrect information or the wrong message.

Ms. Johnson defined data reliability as it relates to computer-processed data. Generally, computer-processed data are reliable enough for our purposes if:

the input data are reasonably accurate and complete, and
any computer processing performed on that data is reasonably accurate and complete.

Because the information world has changed dramatically-more data are collected and collected in a variety of ways-and because audit teams were struggling with using the GAO guidance "Assessing the Reliability of Computer-Processed Data-the 'Gray Book' ," the GAO decided to develop new guidance for assessing the reliability of computer-processed data.

To develop the new guidance, a GAO-wide task team was formed about 1 year ago. It developed the new guidance, "Assessing the Reliability of Computer-Processed Data." This guidance is consistent with the Yellow Book and will replace the Gray Book (GAO/OP-8.1.3, September 1990).

The GAO task team developed a flexible framework (flowcharts) within the new guidance so it would be easy to use. Auditors will use the first flowchart to help them decided whether they need to use the computer-processed data. If the auditors decided they need to use the data, then they should go on to the second flowchart. The second flowchart helps auditors decide what testing should be done on the data that's used.

Throughout her presentation, Ms. Johnson emphasized two points: First, that auditors should only need to test the data that they use, and secondly, that any reliability testing of computer-processed data should only be done to determine if the data is reliable "for our [the auditors'] purposes," NOT as a way to bless the accuracy of the data system.

Ms. Johnson provided an overview of the GAO's new guidance, "Assessing the Reliability of Computer-Processed Data." The new guidance is very well-outlined in Ms. Johnson's slide presentation.

She noted that auditors shouldn't forget to do quick, common-sense-based testing on preliminary computer-processed data to find obvious problems. For example, auditors should look at existing audit reports, including financial, GAO and Inspector General reports. Auditors should also check for data quality studies done by the auditee or contractors, and should do a total record count to identify missing data, check highs/lows, and should get the data dictionary for the data. All this can be done quickly and early in the audit process.

Ms. Johnson noted that if auditors know what to do to verify the data-they'll do it quicker-the decisions on whether to use computer-processed data will be made quicker. How much testing, and how long it takes depends on how critical the data is to your audit product. However, she emphasized that even computer-processed data used for background data should be reliability tested.

The GAO will release an exposure draft of the new guidance the week of September 10, 2001.

Walt Smiley, Section Manager for Fiscal Analysis and Expenditure Forecasting
Virginia Joint Legislative Audit and Review Commission (JLARC)

Mr. Smiley noted 3 things that his office worries about in this area: access, quality of data, and disclosure. Access issues usually come up when dealing with death penalty cases. As for data quality, it's rare to find data in as "good" of condition as the agency reports it to be. Disclosure-personnel with Virginia Joint Legislative Audit and Review Commission (JLARC) make sure that they document what they do with the data from agencies. They have found that this helps diffuse some of the problems that may come up after audits are issued. For example, often, with computerized-data people will try to use the data to make a point. As long as auditors document what they did with the data-their methodology-it's less-likely that non-audit entities will abuse the data.

In Virginia, it's up to the audit teams to judge the validity of the data. Teams use common sense and healthy skepticism to make this judgment. Because they can't be in error, teams are often slow to draw conclusions. They must verify the data-so they go to the data sources. Mr. Smiley gave an example of checking computerized-data with the State Highway Commission, only to find out that the data-list was "made up" by the agency.

Often, audit teams start out trying to think about where data may already be available, but then end up getting their own data through surveys and interviews. Mr. Smiley agreed with Ms. Johnson of the GAO, that data should be reliability-tested in light of using it for our purposes-for the auditors' purposes, not to bless the accuracy of the data source system.

Mr. Smiley noted that Alan Greenspan's has said "It is better to be roughly right than precisely wrong." In bringing up this quotation, Mr. Smiley warned that it is easy to blow audit deadlines by "getting to know and love the data," because this often leads to verifying too much data-much more than is sometimes necessary.

How long should it take to verify data? Mr. Smiley said, this, of course depends. Verification work might consist of interviews, or some statistical testing. The bottom line however is that, auditors will never be finished testing the data and that when reporting they should fully disclose how they collected the data and who generated the data. Mr Smiley warned of shifting data to answer legislators' questions-this can lead to inaccuracies. He also noted that auditors are responsible to the legislature, but not for the legislators and again emphasized disclosure.