NLPES Question of the Month

May-June 2005:

How can geographic information system (GIS) tools be used in program evaluation work?

We have used GIS on a couple of our reports, where we have found it provides a great visual tool as well as a way to measure and present variables that might be difficult to quantify otherwise. In 1997, we used GIS to analyze whether urban transit (bus) routes adequately served low income and minority residents. We did this by overlaying transit routes with census data by zip code.

http://www.comptroller.state.tn.us/orea/reports/transit.pdf

We recently used GIS to determine the exact locations of asphalt plants and their potential market areas. The study was to determine the feasibility of local governments owning and operating their own asphalt plants for paving their local roads, and the degree of potential competition among private asphalt plants throughout the state. GIS was first used to locate the exact locations of existing asphalt plants throughout the state, and then 30 mile rings were drawn around the plants (again using GIS) to estimate their market area. Because it must be kept hot, asphalt can only be hauled so far. Thirty miles was a conservative estimate. The goal was to determine whether there were adequate asphalt plants in the state in geographic proximity to all areas to provide adequte competition on bids.

http://www.comptroller.state.tn.us/orea/reports/asphalt.pdf

Benjamin Wilson, Oregon

We have found that GIS software can be a powerful analysis tool in identifying risks and displaying audit results. First an example of risk identification is a an audit we are completing that we used GIS software to match up school districts with the number of students they were claiming in their alternative education programs. Instead of analyzing the school districts on a spreadsheet and analyzing the risks this way we linked it to GIS data and found that the top school districts were all located in one corner of the state. This led us to understand that an education service district under investigation for misreporting was also in the geographic area and had set up programs in all of these school districts. This enabled the narrowing of the audit scope and a greater understanding of the data than can be easily determined by conventional analysis methods.

For reporting purposes we have also found that it is a nice tool to display the audit results in an easily understood format. For the above example we plan to include a map of the above situation in the report to show how these school districts stood out from the general population of school districts. The old adage that a picture can say a thousand words is definitely true for this audit.

We have also used the GIS software in an audit of heavy equipment and looking for inter- and intra-agency sharing opportunities by mapping all heavy equipment owned by state agencies. Another audit idea was to analyze agencies that conduct compliance visits and determine if there are any patterns or if there are areas that do not receive adequate oversight.

Angus Maciver, Montana

The Montana Legislative Audit Division has used Geographic Information Systems (GIS) applications in performance audit work. We are currently using ESRI ArcView 8, which is a standard desktop GIS application and meets our needs.

GIS applications can be used as a graphics production tool to design maps illustrating many different features of government programs. We have used GIS to produce maps showing the administrative boundaries and distribution of state facilities used in forest fire fighting and in the administration of the state’s university system. Although these maps could be produced using conventional graphics software, GIS applications have enhanced capabilities to easily quantify and depict different administrative boundaries and units, variations in conditions, and the distribution and density of state service recipients and facilities.

However, the real advantage in using GIS in auditing is the added analytical power. The general approach we have been using is to combine ‘layers’ of geographic data related to government operations or programs with other ‘layers’ showing general or specialized administrative boundaries. It is this ability to make connections and quantify relationships between seemingly unrelated sets of data that makes GIS such a valuable analytical tool.

A practical example of using GIS was demonstrated in our recent performance audit of the Montana Youth Challenge program. Challenge is an intervention program for at-risk youth, which targets high school dropouts. We used GIS to assess whether the program was effectively targeting the state’s population of high school dropouts. We produced one geographic layer showing point data for all program participants (based on individual addresses from the program’s database). Another layer was formed using the geographic boundaries of high school districts (this data set also included district enrollment and dropout data). Using the tools in the GIS application, we were able to ‘cut’ a new layer showing how many Challenge participants were drawn from each district. We could then determine if there was any significant variation between a district’s drop out rate and the program’s recruitment efforts. This analysis showed graphically that the program could improve recruitment efforts in some of the state’s urban areas and Indian reservations.
The program had not made the connection directly and GIS analysis helped to quantify the relationship and provided specific data for improving recruitment efforts.

As the volume and quality of GIS-compatible data (both inside and outside of government) continues to grow, opportunities for using GIS applications in auditing will also increase. The volume and potential uses of socio-economic data available from the U.S. Census Bureau could provide many opportunities for evaluating the efficiency and effectiveness of government. GIS applications allow users to attach or join base data (such as that provided by the Census), to any geographic coverage and perform analysis that was previously difficult and time-consuming, if not impossible.

Jeff Ripp, Wisconsin

In Wisconsin, we have used geographic information system (GIS) software for a variety of projects and applications since 2001. One of the primary uses of GIS in our office is the creation of presentation-quality maps for our reports. We have developed generalized data layers that provide clean outlines of state, county, and municipal government boundaries. These layers can be easily customized to depict information such as state agency regional boundaries, population, delivery areas for program services, location of facilities, or other information. Many of our reports contain maps that depict county-by-county differences that can help the Legislature and the public visualize findings that are relevant to the audit in a way that tables and graphs cannot.

Although GIS provides program evaluators with a great tool for presenting information in the form of maps, the true power of GIS is in its analytical and data management capabilities. Using GIS, program evaluators can identify geographic patterns or trends during audit planning or fieldwork that may require further investigation or lead to significant new findings. Some examples of how we have used GIS analytically in Wisconsin include:

mapping the location and density of air pollution permit holders as part of our effort to evaluate staffing and workload needs in Wisconsin’s Air Pollution Permit program;
analyzing the distribution of manufacturing property throughout the state as part of an audit of Wisconsin’s manufacturing property tax assessment process;
evaluating state employee purchasing card spending patterns by city and zip code to identify trends and potential geographic areas with problems; and
quantifying the effect of sewer overflows in Milwaukee on water quality, including analyzing the location of overflow points and other sewer infrastructure.

Despite the usefulness of GIS for program evaluation work, implementation of a GIS as an evaluation tool differs from enterprise GIS applications at other government agencies. Many other agencies tend to use and update the same geographic data layers consistently, such as the Department of Transportation’s need for updated and accurate information on roads or bridges. In contrast, as program evaluators, we require access to new and varied information for each project. Creating new geographic data is a time consuming and technically demanding process, and as a result we do very little geoprocessing in our office. This means that we must rely heavily on other federal, state, and local government agencies to provide us with geographic data. Fortunately, much of these data are available free-of-charge or for a minimal fee over the Internet.

Because we must rely on other organizations, it is critically important that we evaluate the quality, accuracy, and currency of the data we use in our projects. Most organizations are able to provide standardized information about their data, known as “metadata,” which adequately describes its content, quality, and accuracy. We have built a reasonably large database of geographic data by collecting data layers from other state and local government agencies throughout Wisconsin, and have customized some data layers for our specific purposes.

In closing, we have found that GIS has enhanced our program evaluation work in Wisconsin. Like any other software tool, users need to be trained before the technology can be fully implemented. For those who may be interested, we are currently using ESRI software (ArcView 9) on Microsoft Windows XP workstations. We rely on the Wisconsin Legislative Technology Services Bureau, the information technology agency for the State Legislature, for product support and licensing.

Heather Moss, Washington

Regarding GIS, we've used it in a couple of different ways. On a basic level, we've used GIS to plot data on a statewide map and then used the map in our reports and presentations to show expenditures, projects, or clients across the state at the county level. Our members are always very interested to see how their local jurisdictions compare to their neighbors, and they like to understand statewide patterns. In one project (Investing in the Environment, Report 01-1), we took a slightly more sophisticated approach to using GIS: we identified various local environmental and municipal projects that occurred across the state and arrayed them all together on a map to highlight where coordination needs to happen between environmental goals and efforts and other public projects. The maps we created showed that, while we may have good programs that work vertically (state to local), there is room to improve how various state and local programs work together to coordinate efforts on the ground. We imagine there is much more potential to use GIS and are anxious to learn how other audit shops have used the tool.

Jan Sandberg, Minnesota

Possible uses of GIS include:

(1) Calculation of distance. For example, in a study of Minnesota’s tuition reciprocity agreements with neighboring states, we used an internet tool to determine the distance between students’ hometowns and the schools in which they enrolled, based on zip codes.

(2) Capturing information within an area. For example, you could calculate the percentage of state residents within X miles of a point, such as a vehicle registration office. Or you could determine the percentage of persons within other types of areas, such as proposed legislative districts (MapInfo promotes its tool for redistricting uses).

MapInfo focuses on four broad uses:

1. Locate (finding service users, tracking mobile resources)

2. Visualize (mapping industries or population density by geographic area, locating specific users). We typically use MapInfo to display location information.

3. Analyze (using thematic mapping to shade areas and reflect numbers from a database, identifying clusters and other geographic patterns, asking questions using location as criterion)

4. Plan (e.g., locating service centers or redistricting).