The Working Paper



Chair's Corner | Greg Fugate (Colorado)


Picture of NLPES Chair Greg FugateAutumn is a time of transition. In nature, the trees are taking on vibrant colors as they prepare for the cold winter weather ahead. At home, the kids are back in school and weekends are now filled with sporting events. At work, we are wrapping up months-long projects and beginning preparations for the upcoming legislative session. Well, autumn is a time of transition for NLPES too. As the incoming chair of the NLPES Executive Committee for the 2016-2017 term, I’m looking forward to working with the group of talented and dedicated colleagues who serve on the Executive Committee. I would like to congratulate Linda Triplett (Mississippi) on being elected as our new  vice chair, and Shunti Taylor (Georgia) on being elected as our new secretary. I would also like to welcome our three newest members—Melinda Hamilton (Michigan), Emily Johnson (Texas) and Kiernan McGorty (North Carolina)—and thank our recent departures—Dale Carlson (California), Rachel Hibbard (formerly with Hawaii), and Charles Sallee (New Mexico)—for their years of service on the Executive Committee. Finally, I would like to welcome Jon Courtney (New Mexico) to the Executive Committee. In accordance with the NLPES Bylaws, Jon will fill the seat left vacant by Nathalie Molliet-Ribet (Virginia), who recently took a new position with Virginia's Department of Education.

This year the Executive Committee will continue its efforts to keep the membership informed, provide opportunities for professional development and recognize superior work. Additionally, the Executive Committee will complete development of a document outlining data access principles that we hope will help member offices communicate with their legislatures and other key constituents about the importance of auditors and evaluators access to data as a critical component to our ability to provide the thorough, independent, objective and fact-based assessments and analyses on which legislators rely for decision making. Finally, the Executive Committee will take a comprehensive look at the NLPES bylaws to identify changes that are needed for clarification and to ensure that the bylaws remain current as NLPES’ needs evolve. 

I would like to congratulate the Mississippi PEER Committee staff on recently hosting a very successful fall professional development seminar (PDS) in Jackson, Miss. We had about 130 registered attendees from 26 states and Washington D.C. I would like to thank office directors for investing in employees’ career development by sending them to attend the PDS. We’re already planning next year’s PDS, which will take place Sept. 17-20, 2017, in Madison, Wisconsin! 

During my remarks at the NLPES Awards Luncheon in Jackson, I talked about the fact that many of us didn’t necessarily plan our career paths to become auditors and evaluators. For me, organizations like NLPES and NCSL have been critical to my career and professional growth because they allowed me to see that what I was doing was more than just a job—it was a career in public service. This change in perspective helped me to see the bigger picture, which was key to my job satisfaction, my retention, and, ultimately, my ability to understand the value of doing what I do. 

Through NLPES and NCSL, each of us is exposed to a network of individuals and resources where we can learn ideas and share insights. We’re reminded that, as legislative staff, what we do is essential to the functioning of the legislature as an institution and carrying out its responsibility to provide effective oversight as part of the policy-making process. Second, we’re reminded that, as auditors and evaluators, we’re members of a profession that serves the public interest by working to ensure accountability for the use of public resources—this is critical to our democratic system of government. 

So, regardless of whether you are new to the profession, or have been working in the trenches for many years, I encourage you to remain involved in NLPES and NCSL and to keep this bigger picture in mind the next time someone asks you what you do. 

Greg Fugate is the 2016–2017 NLPES Executive Committee Chair. He can be reached at 

BIG Changes in Store


SpotlightBeginning in 2017, NLPES’ official newsletter, The Working Paper, will be distributed on a semi-annual basis. The newsletter’s purpose is to help legislative evaluators and auditors get a better feel for what’s going on in the field of legislative program evaluation and performance auditing around the nation. But, we need YOUR help to ensure the newsletter remains relevant and meaningful. 

Please take a few minutes to provide feedback on the newsletter by completing a brief survey (link below).  If you have never read the newsletter or didn't know it existed until now, that's OK, we want your opinion, too!

Please complete the survey by Dec. 23, 2016. We appreciate your feedback!

See the survey


Report Radar—Roundup of Evaluations from Across the Country | Chris Latta (Pennsylvania)


Magnifying glass over reportThat time of year thou mayst in me behold when yellow leaves, or none, or few, do hang upon those boughs which shake against the cold, bare ruin’d choirs, where late the sweet birds sang.”  - William Shakespeare, Sonnet 73

Good day, fellow seekers and welcome to the fall edition of the Report Radar, your one-stop shop for interesting evaluations from across the country.  While these studies may not match up to the poetry of The Bard of Avon, they do make for some fascinating prose. Featured in this edition are reports on education, natural resources, and economic development.

First in the hopper, we have a report on water resources from Old Dominion where Virginia’s Joint Legislative Audit and Review Commission (JLARC) found the effectiveness of the commonwealth’s planning to be lacking.  The commission took up the study in response to “concerns about the sustainability of water supply and demand especially in eastern Virginia.”  Specifically, the report found that while groundwater supplies are insufficient to meet current demand, the Department of Environmental Quality could bring current consumption to sustainable levels. Given the probable growth of permitted and unpermitted uses in the region, however, these successes may be fleeting.

The commission also found that state and local water plans are vague and not aligned with water location and use. Current plans fail to set out the state’s water challenges and are deficient in comprehensive and actionable approaches to address known issues. JLARC also found that local water plans, while useful, do not take into account regional concerns. The report in its entirely can be found here.

Next we have an interesting report from our brothers and sisters in the Copper State. The Arizona Auditor General’s office found in its October special investigation that a former Northern Arizona University postal services manager used fraudulent schemes to embezzle public monies totaling $354,902. The university employee managed this by faking computer and purchasing card records and submitting fabricated bills to hide his pilfering. 

The good folks in the Arizona AG’s office also found that university officials failed to safeguard and control university funds. Specifically, “management did not monitor or take steps to ensure employees were appropriately following policies and procedures designed to protect public monies.” They also found that management failed to properly train employees to investigate suspicious vendor statements or reconcile purchasing card reports to bank software.  The primary recommendations of the Auditor General’s office include conducting random and periodic reviews to make sure goods are physically accounted for and regularly assess whether or not financial controls are performing appropriately.  The full report can be found here.

Moving right along, we have a report from the Florida Legislature Office of Program Policy Analysis & Government Accountability (OPPAGA).  Apparently not all is bright in the Sunshine State’s nursing programs where around 42 percent had licensure exam passage rates below the required legislative standard in 2015. Most of the programs were established since 2009 and are not accredited. 

In 2009, the Florida Legislature made changes to the law to increase the number of approved nursing schools in an effort to deal with the state’s nursing shortage. The OPPAGA report found that passage rates vary by program type. Across Florida, 83 percent of graduates from licensed practical nurse programs passed the exam in 2015 (national average was 82 percent).  Students graduating from bachelor of science in nursing programs passed at a rate of 88 percent compared to 87 percent nationally.  However, graduates from associate degree programs passed at 75 percent compared to 82 percent nationally. Read the Florida report.

Finally we bring to you a performance audit from the Utah Legislative Auditor General on the Economic Development Corporation of Utah (EDCU). While the AG’s office reports the EDCU has provided a valuable service to the state, that success is coupled with poor “practices in its financial management and governance.”

The audit found that improper expenditures occurred at EDCU. Specifically, the report found that a former EDCU executive made credit card purchases totaling $2,783 for items that were not business related. The EDCU also failed to produce documentation for its executive’s other questionable credit card purchases. As such, the purchases known to be improper may be low compared to the actual. 

The auditors also found that $3,005 of computer hardware was missing and lacked receipts. The EDCU executive was not required to reimburse economic development corporation. Instead the funds were deducted from his severance package. That package “included a vehicle, multiple months of pay, health insurance and electronic equipment.”

Other credit card purchases were found to be questionable because they were not properly documented. Half of the executive’s credit card transactions lacked receipts—totaling $46,496. These transactions included travel purchases and other questionable expenses. 

During negotiations with the Utah Governor’s Office of Economic Development, the EDCU provided 76 meals worth $9,514 ($125 per meal) to governor’s office employees. The former executive of EDCU also took governor’s office employees to lunch an additional 70 times. Those lunches were worth approximately $2,000. The meals occurred during the time when the Governor’s Office of Economic Development “was in the process of soliciting bids and renewing its contract with EDCU.” The Utah reported can be found by clicking here.

If there’s a report you feel merits inclusion in the next edition of Report Radar, please feel free to send it my way. Until then, have a great fall/winter. I’ll see you back here in the spring.

Chris Latta is a project manager with Pennsylvania’s Legislative Budget and Finance Committee. He can be reached at 

State Profile—Wisconsin Legislative Audit Bureau | Joe Chrisman (Wisconsin) | Click image for larger view


Photo of Wisconsin Legislative Audit Bureau staffAbout Us: The Legislative Audit Bureau is a nonpartisan legislative service agency created to assist the Legislature in maintaining effective oversight of state operations. The bureau’s 87 staff conduct objective financial audits and performance evaluations of state agency operations to ensure financial transactions have been made in a legal and proper manner and to determine whether programs are administered effectively, efficiently and in accordance with the policies of the Legislature and the Governor. The results of the bureau’s work, including recommendations for improvements in agency operations, are addressed to the 10-member Joint Legislative Audit Committee and provided to the 132 members of the Legislature. The bureau is strictly nonpartisan. 

The head of the bureau is the Wisconsin state auditor, who is appointed by and serves at the pleasure of the Joint Committee on Legislative Organization. The current state auditor, Joe Chrisman, is only the fourth person to serve in this capacity in the bureau’s 50-year history.

The bureau has recently received Certificate of Impact awards from NLPES for its evaluations of the Government Accountability Board, Supervised Release Program Expenditures, and the Wisconsin Economic Development Corporation.

New Initiatives: In 2016, the bureau celebrated its 50th year of service to the Wisconsin Legislature by hosting a two-day professional development conference for all staff (at which Brenda Erickson graciously shared her NCSL wisdom), launching a newly designed website, joining the Twitterverse (@WILEGAUDIT to announce report releases, introducing a new one-page Briefing Sheet navigational tool to accompany each report, and converting to electronic publication of all reports.

2017 Professional Development Seminar: The bureau is pleased to have been selected by the NLPES Executive Committee to host the 2017 Professional Development Seminar in Madison. Please mark your calendars for three days of training on Sept. 18‑20, 2017. The venue is the recently renovated and modernized Concourse Hotel, where the nightly room charge for the conference will be $165 plus tax.

Travel to Madison is particularly convenient, with many daily flights to the Dane County Regional Airport. The conference hotel has free shuttle service to and from this airport. Alternatively, Madison is a short 90 minutes from Milwaukee’s General Mitchell Airport or two hours from Chicago’s O’Hare Airport. Both Milwaukee and Chicago are connected to Madison by bus service at $60 roundtrip ( or 

There is no need to rent a car during your visit to Madison. The conference hotel is conveniently located one block off the bustling Capitol Square and is adjacent to the many great restaurants and shops on Madison’s famous State Street. 

Madison has been reported to have more restaurants per capita than any other U.S. city.  In addition to classic American fare, Madison’s diverse dining scene has many ethnic restaurants and many food carts, which serve delicious cuisine from all around the world. In fact, Bureau staff are eager to share the following recommendations for your consideration.

While visiting Madison for the 2017 Professional Development Seminar, the one place you must eat is:

  • The Old Fashioned because of its mouth-watering, award-winning cheese curds. Few things are more Wisconsin than good cheese curds!
  • The Tipsy Cow because they have the best fried avocado tacos!
  • Graze because you don’t know how good mac and cheese can be until you’ve tried Graze’s mac and cheese!
  • Dotty Dumpling’s Dowry because they have the best burgers and fried cheese curds!
  • Salvatore’s Tomato Pies because the pizza will likely be the best you have ever had!
  • Mickey’s Tavern because they have the best lamb burger!
  • Paisans because you can enjoy delicious pizza from their patio overlooking Lake Monona!
  • Mickie’s Dairy Bar because it is most authentic Madison experience around and their chili cheese omelet will leave you satisfied for days!
  • Madison Sourdough because everything I’ve had there has been delicious, including the fresh homemade bakery, strata and salads, and its central location!

We look forward to welcoming you all to Wisconsin in September 2017! Come and learn along with us!

Joe Chrisman is Wisconsin’s state auditor. He can be reached at Photo: Wisconsin Legislative Audit Bureau, 50th Anniversary.

Research and Methodology—Strategies for Comparison in Performance Auditing: A Brief Methodological Guide to State Selection | R. Scott Swagerty, Ph.D. (Arizona)


Graphic of person trying to choose a pathComparison, or evaluating the similarities and differences between the traits of two or more entities, is a common method used in performance auditing. When auditing the performance of state government agencies, comparisons are usually between the agency being audited and similar agencies in other states. However, it is not usually clear which states to use as comparative peers. Thus, auditors may select comparative peers without methodological rigor, potentially depriving the comparison of its value in making and supporting audit conclusions. 

Using a group of states for comparisons regardless of what is being compared is one common practice for selecting comparative peers. For example, geographic, political or demographic attributes can generate a group of states that are similar to each other (i.e., Western states, states with Republican-controlled legislatures, or states with populations of more than 5 million). However, this approach does not consider the comparison’s purpose and may lead to selecting inappropriate comparative peers. Whether or not a state is located in the West may be useful in comparing water usage, for example, but would not by default be useful in comparing state lotteries. So when auditors select comparative peers using a “general peer states” approach, they miss opportunities to capitalize on variation that could change their conclusions. Selecting comparative peers with methodological rigor is essential to making useful, informative and reliable comparisons.

To select methodologically appropriate comparative peers, first identify the comparative dimension, answering the question “Comparable in what respect?” Conceptually speaking, a comparative dimension should answer an if-then question—if dimension X were structured differently (relative to how it is currently structured), then outcome Y should differ in this predictable way.

For example, in a recent audit of the Arizona State Retirement System (ASRS), our office evaluated how the ASRS performed relative to other public employee retirement systems. Auditors determined that performance was best measured by the ASRS pension plan’s funded status, or the ratio of the pension plan’s assets to its estimated obligations to current and future retirees. After identifying this outcome measure, auditors determined how various features of the pension plan might affect the outcome measure. For example, auditors identified that ASRS’ investment portfolio, advisor fees, contribution rates and benefit increases were related to the ASRS pension plan’s funded status. The dimension(s) of interest for comparison’s sake should come from this list. In other words, if contribution rates were higher, then funded status would increase, and so on. These observations led to a natural ordering of the areas where the ASRS pension plan’s performance should be compared to peer states’ pension plans.

Once the comparative dimensions have been identified, there are several strategies for actually selecting comparative peers:

  • Best-practice comparison: Auditors select peer states that a national organization or another independent agency has identified as best-practice. In this type of comparison, auditors compare their agency’s performance to a best-practice state on the comparative dimension as established by criteria. This is the simplest state selection strategy when a state can be identified as a best-practice model. 
  • Most-similar comparison:  Auditors select peer states most similar to their own on the comparative dimension and compare the states’ respective outcomes of interest. Importantly, the similar states do not have to be similar on every dimension. Auditors should establish that, to the extent practicable, comparative peer states are similar on dimensions relevant to the outcome of interest. Further, auditors should be reasonably sure that the ways their agency is different from its comparative peers are not meaningfully related to the outcome measure of interest.
  • Most-different comparison: Auditors select peer states most dissimilar to their own on the comparative dimension and compare states’ respective outcomes of interest. As with the most-similar design, a most-different comparison need not be different on every dimension, just those that are substantively related to the outcome of interest. These comparisons are useful when there are strong criteria to support that different structures lead to different outcomes. For example, if an auditor found criteria to support that decentralized child welfare agencies generally conduct more timely initial investigations than centralized child welfare agencies, the auditor could select a state with a decentralized agency as a peer to their own state’s centralized agency and compare time to initial investigation between the two.
  • A typological comparison: This comparison strategy is useful when there are two or more discrete types of systems/programs/features and an audit question requires providing information about each type. Comparative peers are selected to ensure that each discrete type of system/program/feature is represented in the comparison. To continue with the example of child welfare systems, a typological comparison would select a state-administered system, a county-administered system, and a hybrid system with shared administrative responsibilities between the two levels of government. Typological comparisons provide information to stakeholders about the known variation with regard to the comparative dimension and where the entity being audited falls on that spectrum.
  • A temporal comparison: Occasionally, the relevant comparative peer is not another state but the same agency at a different point in time. For example, if the state child welfare agency recently decentralized some administrative responsibilities to the county level, a comparison of outcome of interest before the change and after the change would provide relevant information regarding the overall effect of the change.

One or more of these comparative strategies may be appropriate for audit work, and all are useful to consider when planning comparative work. In many cases, one of the approaches will be the clear methodological winner; in other cases, strategies must be combined to answer the audit question effectively. However, by clearly articulating the comparative dimension and adopting a clear comparative strategy, the appropriate comparative peers become more easily identifiable, and the conclusions drawn from the comparative work are more reliable. 

R. Scott Swagerty, Ph.D., serves as methodologist for the Arizona Office of the Auditor General. He can be reached at 

Blast from the Past—Team Problems


Blast from the Past(Editor’s Note:  This article appeared in the April 1989 issue of the LPES News, now known as The Working Paper.  Discuss these case studies with your co-workers to arrive at a resolution to the situations.)

One of the great joys, and great challenges, of working in program evaluation/performance auditing is the team concept.  While most in the profession believe the team approach produces good reports, most of us have ruined many a dinner with our spouses and significant others by complaining about the occasional [expletive deleted] on our team.

Listed below are classic case studies regarding working on a team.  While we all recognize these situations, do we know what to do?  As with many things in program evaluation, the questions come to mind more readily than the answers (that is, the printable ones).

If you have faced some of these situations and they got your creative juices flowing, why not share your solutions, be they effective, humorous or outrageous, with your co-workers.

Case Studies

  1. Alice is an employee who was hired three months ago. She seems to be having more trouble than usual learning to do audit work. You have trouble getting her to let you see workpapers. When you do see them, you feel that there is a lack of attention to detail and a lack of understanding of the intent of the procedures. In addition, Alice is going over budget consistently on assigned tasks.  As her in-charge, you realize that it will be up to you to do this person’s last probationary evaluation. What steps should you take to deal with this situation?
  2. You are about to become an in-charge for the first time and will be supervising Joe, who is a close friend. You recent got promoted to auditor II even though others, including Joe, have been at State Audit a few months longer.  What difficulties might you expect to encounter? How might you address them?
  3. You are about halfway through preliminary survey of a large audit. You are finding that your manager has very different ideas from yours on how to proceed with the audit and what issues are important. You have had a few conversations that bordered on being heated. You also sense that the manager is “talking behind your back” to the assistant director and only giving one side of the story. What steps might you take to address this situation?
  4. You are supervising a large team in a small workspace. One of your team members talks a lot, making cracks about the auditee, and talking about subjects not related to work. You feel that it causes everyone to be distracted and too often causes the whole team to get into conversations. You are approaching being over budget and wish that your team members seemed a little more concerned about meeting deadlines. What steps could you take to improve the use of work time and still maintain team harmony?


This is the seventh in a series of ‘oldie but goodie’ articles culled from past issues of the NLPES newsletter. 

Re-energizing Our Supervisor Training Program | Yvonne Benn (Michigan)


What We DoDuring fiscal year 2015, the Michigan Office of the Auditor General issued auditor's opinions on 19 sets of financial statements and schedules, including the State of Michigan Comprehensive Annual Financial Report. In addition, we also conducted the Statewide Single Audit to assess compliance for approximately $21 billion in federal funds, and we completed 38 performance audits covering a multitude of programs and functions. These projects were all directly overseen by a team of 36 audit supervisors who, in most cases, supervise both financial and/or the single audit and performance projects and have experience levels ranging from almost none to 30+ years. As you can imagine, effectively determining and meeting the training needs of this group of supervisors has been an ongoing challenge.  

Old times gone badly: Historically, each member of our staff completed an annual training needs assessment form to indicate any desired training topics.  We then used the completed assessments to formulate an annual office-wide training plan based on the most commonly requested topics and available resources. Predictably, this often led to training sessions that had a one-size-fits-all approach that left our supervisors with a somewhat lukewarm level of satisfaction and wanting more. The problem was determining exactly what our supervisors wanted more of and how we would deliver it.  

Bagging it: In the summer of 2015, we began pursuing a few different avenues to determine our supervisory training needs. Our first endeavor was a series of brown bag lunches for our supervisors and training coordinators. These brown bag lunches were easy, low-cost and provided a valuable starting point for gathering information to better understand our supervisors' current challenges and their related training needs. We offered an optional invitation to all supervisors to join in one of three brown bag lunches that provided an opportunity for open and free-flowing dialogue within the targeted group.  Almost all of our supervisors actively participated, and we had an overwhelmingly positive response.  And wow, did we get information!!  From those three brown bag sessions, we gathered hundreds of suggestions for training directly related to technical areas; project management; and soft skills, such as providing feedback, resolving conflict and improving communication. Through the brown bag experiences, we also quickly recognized the value of bringing the supervisors together and providing the opportunity to have an open conversation about their training needs. This experiment was so fruitful and well-received that our office now uses brown bag lunch sessions for a variety of groups and topics.

Into the Future:  Ironically, it was at one of the brown bag lunches that a supervisor introduced the idea of the World Café Model as a potential training delivery method for our office. The World Café Model is based on the idea that the power of conversation should not be overlooked and that a group of individuals has the wisdom and creativity to solve the group's challenges through open and free-flowing conversations that link and connect ideas from small subgroup conversations to the larger group. At first, we were somewhat skeptical that this format would work successfully with our group of supervisors. We did our research, however, and, in June 2016, we held our first café.  We structured our café so that the group could connect and share supervisory best practices for several topics gleaned from our brown bag sessions. The café was a resounding success! Many of our supervisors expressed that they gained more valuable information in this four-hour session than they ever thought possible, and many have asked for more café style training sessions in the future. In addition, the café produced numerous supervisory training requests and also provided valuable information related to the training needs of our staff auditors. Since the café in June, we have held supervisory training sessions on a variety of requested topics, such as time management and budgeting specifically tailored to our environment, preparing value-added staff evaluations, identifying teachable moments and giving constructive feedback, resolving conflict, and managing a multigenerational team.  We plan to expand our café forums in the future.

Feedback, feedback, feedback: Each month, our auditor general also holds a monthly luncheon with a different group of staff members. At these lunches, staff are encouraged to share any ideas, questions, or concerns that they may have directly with him. Staff really look forward to these opportunities to connect directly with the auditor general in a casual atmosphere, and these discussions have also brought forth several training ideas for our supervisors. 

Reenergized: Developing and maintaining a dynamic and engaging training program that addresses the broad spectrum of needs of our supervisors while remaining cost-effective is an ongoing challenge. We now have a storehouse of training topics for consideration specifically requested by our supervisors, which has greatly helped us in developing our current and future training sessions and our supervisors feel more invested and enthusiastic about their training. In addition, we can now consider a multitude of requested topics, more effectively prioritize training needs, and identify subjects that are best presented by external experts and those suited for in-house training development. This has helped us present the most desired subjects timely, keep costs down, and provided opportunities for in-house experts on a variety of topics.  

Yvonne Benn is an audit manager with the Michigan Office of the Auditor General.  She can be reached at 

Mapping Tennessee Education—Legislative Profiles | Russell Moore (Tennessee)


Last summer, a state legislator called the Tennessee Comptroller’s Office of Research and Education Accountability (OREA) with a question: Where can I obtain a list of the K-12 schools in my state House district? After some quick research, we determined that such a list was not readily available from any state or local government agency, but OREA had an idea for how to produce one using ArcGIS software.

In 2014, OREA launched Mapping Tennessee Education, an interactive mapping tool on our website that provided demographic and financial information on Tennessee’s public school districts using ArcGIS. That project, however, focused on school district-level information. We had not yet explored the possibilities of including school-level data. To create a map of schools by state legislative district we would have to cross-reference several datasets from multiple sources. OREA ultimately went on to produce an interactive map of all the public schools in Tennessee, overlaid with maps of state legislative district and school district boundaries, using the Arc software suite – Desktop and ArcGIS Online. This is how we did it.

OREA already had a shapefile created in ArcMap of Tennessee school districts, but we had to locate an accurate file of all the public schools in the state. We recognized early on that obtaining a current list of schools would require some extra steps on our part because the school-level data published each fall by the Tennessee Department of Education is based on the previous school year’s data, which would not include records for certain schools. or, for example, charter schools that had opened or closed and schools that were transferred from traditional school districts to Tennessee’s Achievement School District (ASD) since the previous school year would not be included in the data.

Because we wanted our maps to reflect the most current list of open schools, we cross-referenced the dataset from the previous year against the department's school directory, a database of all K-12 schools in the state. Closed schools would have data but would not appear in the directory; new schools would appear in the directory but would not have data. For example, the dataset included a school that did not appear in the department’s K-12 schools database. After speaking with the district, we learned that the school had closed. After completing such cross-referencing, we had a comprehensive Excel file of all schools in the state as of the fall of the 2015-16 school year.

This comprehensive Excel file also included demographic data for each school (e.g., percentage of students classified as English language learners, with disabilities, enrolled in the federal free and reduced price meals program, etc.), as well as academic measures, such as ACT scores and state accountability designations. The Tennessee Department of Education maintains several comprehensive data files of student demographics and academic performance by school that we were able to use as the core file of school-level data. We ran a v-lookup function to combine all of these files into one final spreadsheet that could be uploaded into ArcMap.

Once we confirmed the data for each school, we had to convert our excel file into a format that would allow us to cross-reference it with the geographical boundaries of legislators’ districts. Similar to how Google Maps will pin an address on a map, we geocoded the addresses for all of the schools and ran an intersect function to generate an Excel-type report that assigned each school to a unique House and Senate district. With this information, we created an individual PDF profile for each member of the House and Senate education committees that listed applicable data for all of the schools and school districts in their legislative district. 

First example of Mapping Tennessee EducationIn addition to the profile created for education committee members, we developed an interactive map where users could view data for an individual school or school district. We took the shapefiles for the geocoded school locations, the school districts, and the legislative districts, and uploaded them all to ArcGIS Online. This tool allows users to customize shapefiles into interactive maps, and we used it to create popup boxes that displayed each layer of data—schools, school districts, and legislative districts. Click image for larger view.

Second example of Mapping Tennessee EducationThe map application feature allowed us to create a map that was both “searchable and clickable.” Users could view information by selecting a feature on the map, or they could type in a particular school, school district, or legislator, and the map would then zoom to that spot. Click image for larger view.

Third Example of Mapping Tennessee EducationThis project was well received by members of the Tennessee General Assembly, especially by those with positions on the education committees. Although much of the information on the maps was already publicly available, no agency had combined the multiple datasets into one comprehensive and searchable map.

Our efforts to create this mapping tool took us far beyond answering that state legislator’s question last summer about which K-12 schools are located within which state legislative districts. OREA plans to develop our mapping skills further still so that we can better inform important policy discussions, debates, and decisions in the Tennessee General Assembly. Click image for larger view.



Defining Tennessee Education: A Glossary of Education Terms

In addition to our primary policy research and evaluation activities, the Tennessee Comptroller’s OREA staff help monitor state House and Senate committees during the legislative session, tracking activity on bills of general comptroller interest (state and local government finance and operations, property assessments, state and local audits, open records) as well as almost all education bills of specific interest to our office. During our committee monitoring duties, staff noticed legislators and staff asking questions about basic education program terminology. With elections and new committee assignments, there are always legislators rotating onto the education committees who may not have much background in education policy. Policy discussions got bogged down with folks trying to get common agreement on definitions or trying to separate state requirements from federal requirements. 

We decided the time had come to produce a glossary of education terms. This was an idea that had been floating around for a while and other nationally-produced education glossaries were out there. But we determined to focus ours on how programs and policies were implemented in Tennessee so that it would hopefully meet the needs of legislators and other policy decision makers in understanding the specific context of policies in Tennessee.

We developed a list of terms based on perennial topics of interest, recent items in legislation, questions that we have heard in committees, “hot” topics in the media, and basic policy areas addressed by the Tennessee Department of Education, State Board of Education, and Tennessee Higher Education Commission. The terms included ranged from accountability to English learner students to municipal school district to Title I of the Every Student Succeeds Act. Where relevant, we included pertinent facts related to Tennessee, as well as more generic definitions, to give legislators needed context. So entries include facts such as:

  • In Tennessee, accountability for student performance is currently administered under the state’s No Child Left Behind waiver, approved by the U.S. Department of Education in July 2015. The passage of the federal Every Student Succeeds Act in December 2015 . . . may prompt revisions to Tennessee’s accountability system.
  • In the 2014-15 school year . . . Tennessee school districts serve 45,739 EL students, about 4.6 percent of the state’s total student population.  According to the Tennessee Department of Education, Tennessee students speak about 140 different languages.
  • Tennessee has 33 municipal school districts across 22 counties.
  • All school districts in Tennessee receive some Title I funding. In fiscal year 2014, the state’s school districts received approximately $280 million from Title I.

We also included a list of education-related acronyms and terms; acronyms were divided into three sections: federal and state laws, key terms, and organizations and agencies.

After developing a few style and content guidelines to ensure consistency—e.g., no footnotes or sourcing, a tough change for some of us—we divided the 180+ entries among eight staff. Despite the decision not to include sources within the finished glossary entries, we did reference every entry and put them through our standard “technical review” process to check every item for accuracy. Every entry also went through management and editorial review to check for consistency across entries and to ensure objectivity, as well as to continue to whittle their length down to the most salient facts. The sheer number of entries made assigning technical reviews a challenge to make sure every entry was reviewed by someone who had not done the research or writing. During the several months of work, we found some additional entries we needed to include, and then, as we neared our deadline, raced to update the No Child Left Behind-related entries to reflect the passage of the Every Student Succeeds Act.

Early on in the production process we identified the need for both an online version and a small, paperback print version. We came up with a simple design we liked for the print version and discussed paper, binding and cover choices. We also worked with the new web designer in the Comptroller’s Division of Technology Services to develop the online version with the type of functionality we wanted.

After requesting the relevant state education agencies to review a sampling of entries for accuracy (typically items that had potential to be politically sensitive or items related to policies that were in the process of changing), our draft finally headed to the state’s General Services Print Shop.

Initially, print copies of "Defining Tennessee Education: A Glossary of Education Terms" were distributed to legislators and key staff, directors of schools in all 140 local school districts, and staff of the state education agencies. And then we started receiving requests from a variety of education stakeholders, state school board members, and local school board members. We found it interesting that despite the availability of the online version, there were numerous requests for the print version, and one superintendent even offered to buy additional copies for his school board members!

Our plans are to update the online version annually: new laws affecting education have been passed this session; suggestions for new terms to add have already been received.  And the legislative response to the glossary has been so positive that the comptroller has decided to have other divisions produce glossaries for their areas. Next year, the state and local audit divisions will produce a glossary of audit-related terms, to be followed by a property assessments-themed glossary in 2018 and a version focused on state and local finance terminology in 2019. 

See a print version of "Defining Tennessee Education: A Glossary of Education Terms."

See the interactive online version.  

Russell Moore is director of OREA. He can be reached at


News & Snippets


Calendar imageDon’t get caught flatfooted! Put these NLPES events on your calendar for 2017—Executive Committee elections, NLPES Awards, and the professional development seminar. Their deadlines—April, May, and September, respectively—will be here before you know it! Also, be on the lookout for planned webinars on data access and another highlighting the reports from Louisiana and Colorado that won the 2016 NLPES Excellence in Research Methods Award.

  • April 15, 2017 is the deadline for the 2017-2018 Executive Committee elections. Be on the lookout for the call for candidates in early 2017.
  • The 2017 NLPES award submissions are due May 12, 2017. Please visit the NLPES awards webpage to review criteria and consider submitting an application to recognize your office’s work. 
  • The dates for next year’s PDS are Sept. 18‑20, 2017. For a sneak peek of what’s in store, check out the State Profile on the Wisconsin Legislative Audit Bureau.

2016 NLPES Awards News

Blue RibbonAt the PDS in Jackson Mississippi, NLPES presented 27 awards to member offices for the exceptional work they have done to further the program evaluation and performance auditing profession. 

The Excellence in Evaluation Award is given annually to the office that made significant contributions to the field of program evaluation over the last four years (2012-2015). The Mississippi Joint Committee on Performance Evaluation and Expenditure Review is the 2016 winner of this prestigious award.

Picture of James Barber and Greg FugateAn Excellence in Research Methods Award is given to the office or up to three offices that have demonstrated the outstanding use of evaluation methods in a prior year’s (2015) report. The 2016 award winners are the Colorado Office of the State Auditor for its Local Sales Taxes performance audit report and the Louisiana Legislative Auditor for its Louisiana Lottery Corporation performance audit report. 

Certificates of Impact are awarded to offices that released a report in the prior two years (2014 or 2015) that had a documented public policy impact. The 2016 winners are Arizona, California, Colorado, Florida, Georgia, Hawaii, Idaho, Kansas, Louisiana, Michigan, Minnesota, Mississippi, Montana, New Mexico, North Carolina, Oregon, Pennsylvania, Tennessee, Texas, Utah, Washington, West Virginia, Wisconsin and Wyoming.

Congratulations to all winners! Details about the winning offices and reports are posted on the NLPES website.

The NLPES Executive Committee thanks the judges for their help in making the 2016 award program a success. Thank you!

Also, we extend our sincere gratitude and appreciation to Florida’s OPPAGA for their ongoing assistance in announcing our award winners. Hear the podcast announcing this year’s winners. 

Staff Happenings


Hawai‘i’s Office of the Auditor is pleased to announce the addition of two new staff members:

  • Jennifer Ueki (deputy auditor) was the deputy program manager of administration for Hawaii Energy, the state’s energy efficiency and conservation program. Prior to that, she had served as an Assistant Auditor in our office from 2008–2012. Jennifer is a licensed attorney and CPA, and previously practiced employment and labor law and worked as an audit associate at KPMG LLP.
  • Jen Catanzariti (assistant editor) was the creative manager at Pacific Basin Communications, a Honolulu-based publisher of business and lifestyle magazines. Before moving back to Hawai‘i in 2006, Jen was a production manager at Harper's Magazine in New York for six years.

Rakesh MohanImage of ASPA awardRakesh Mohan, the director of the Idaho Office of Performance Evaluations, was presented with the Donald C. and Alice M. Stone Outstanding Practitioner Award from the American Society for Public Administration’s Section on Intergovernmental Administration and Management for professional contributions in the field of intergovernmental management.

After more than 37 years of service, assistant to the deputy auditor general, Gerry Schwandt, will retire Dec. 30 from Michigan’s Office of the Auditor General. Gerry started with the Office in September 1979 as an entry level auditor. He supervised and managed performance audits for the majority of his career before appointment to assist the deputy auditor general in 2011. Gerry was the office's primary performance audit writing trainer and mentor for many years. Gerry was involved in NLPES membership for most of his career and served on the NLPES Executive Committee for two terms (2005-2006 and 2006-2007). Gerry said he "knew it was time to retire when he had worked for one-quarter of the auditors general pictured in the conference room!" Gerry plans to enjoy his retirement by spending more time with his three grandchildren; boating, fishing, and hunting in northern Michigan; and completing a large list of "honey-do" remodeling projects. CONGRATULATIONS GERRY!

Stephen Fox retired from the Audit Division of New Hampshire’s Office of Legislative Budget Assistant (LBA) this past July. Steve obtained a Ph.D. in social policy and social work and was employed for six years at Virginia’s Joint Legislative Audit & Review Commission. In 1992, he moved back to New England where he grew up and became a senior audit manager and then the performance audit supervisor at the LBA. In retirement, he plans to continue riding his bike more than 2,000 miles a year, making furniture and fly fishing. Jay Henry has replaced Steve as performance audit supervisor. Jay started with GAO in New York City and moved back to NH 20 years ago as a senior audit manager with the LBA.

Please let us know if you have staff happenings to share! Email

Stop the Presses! Performance Reports in the News | Dale Carlson (California)

Picture of boy holding newspaper

♫ ♪ “Summertime … and the livin’ is easy.” ♪ ♫

Billie, Ella, Janis, and yes, even Willie, have crooned this classic 1935 lyric by George Gershwin and DuBose Heyward.

Based on recent media reports regarding their work, our member offices certainly weren’t “livin’ easy” this past summer. Below are summaries of the media coverage for some the reports issued by our member offices from roughly the end of June through a large chunk of October 2016.

Arizona Department of Child Safety rejects key provisions of audit
Oct. 6, 2016 – AZ Central – State lawmakers are puzzled at the Department of Child Safety's rejection of a number of recommendations intended to get children into permanent homes more quickly.
Office of the Auditor General  [Click for full report]

Audit slams state agencies for drugging foster kids
Oct. 20, 2016 – ABC 10 – During a recent hearing at the Capitol, State Auditor Elaine Howle revealed the findings on her department’s audit regarding the prescribing and oversight of psychotropic medications to children in foster care. “The State authorized medications in dosages that exceed what’s in the guidance – multiple meds – excessive meds – than the guidance. Key overarching issues,” Howle said.
State Auditor  [Click for full report]

9 Investigates: Are leaders doing enough to help child trafficking victims in foster care?

Aug. 25, 2016 – WFTV 9 – 9 Investigates a new report that shows state leaders are not doing enough to help the most vulnerable children in the foster care system. In 2015, 264 commercial sexual exploitation child victims were in the state’s foster care system. Florida paid $3 million to nonprofit agencies that manage the foster care system. However, state records show the cost for caring for the children is almost $4 million. Members of the Florida Legislature just received a new report from the Office of Program Policy Analysis and Government Accountability, which outlines the shortfalls in the system. 
Office of Program Policy Analysis and Government Accountability  [Click for full report]

Audit reveals 19 points for improvement for College of DuPage 
Oct. 5, 2016 – The Courier – The Illinois auditor general released its performance audit on College of DuPage on Wednesday Sept. 28, offering 19 recommendations for the Board of Trustees to improve oversight, operations and transparency. The college agrees with all 19 points
Auditor General  [Click for full report]

Restrict asset seizure, forfeiture 
Aug. 10, 2016 – The Wichita Eagle – A recent legislative audit found a lack of internal controls regarding property seized by law enforcement in Kansas – including in Sedgwick County. But the bigger concern is whether the state grants too much authority to seize and keep or sell assets, especially when someone hasn’t been convicted of a crime. 
Legislative Division of Post Audit  [Click for full report]

Louisiana could save money with prison alternatives, auditor says  
Sept. 6, 2016 – – The Louisiana Legislative Auditor is urging the state Louisiana could realize hundreds of millions of dollars in savings over the years by finding alternatives to imprisoning nonviolent offenders, a report from the Legislative Auditor said. "While incarceration is necessary for offenders who pose a threat to public safety, implementing strategies to reduce Louisiana's incarceration rate, especially for nonviolent offenders, could reduce costs and still keep the public safe," the report said.
Legislative Auditor  [Click for full report]

New audit of Baltimore liquor board finds recurring problems  
July 20, 2016 – The Baltimore Sun – A state audit released Wednesday found that Baltimore’s Board of Liquor License Commissioners had failed to correct a number of problems unearthed in a 2013 audit. However, the board’s chairman said most of the recurring problems have been addressed since auditors closed their investigation in September. The 49-page report by the state Office of Legislative Audits spells out 18 major problems that the board identified between January and September 2015.
Office of Legislative Audits  [Click for full report]

Report: Michigan school buses not being properly inspected 
Sept. 9, 2016 – – We trust them with the safety of our children. But how safe are school buses in Michigan? A new report accuses Michigan State Police of not properly following up on bus inspections. Inspectors are required to inspect all of Michigan’s 17,000 school buses every year. A recent report from the Michigan Office of the Auditor General says MSP wasn’t following up with school districts to make sure problems were being fixed.  
Office of the Auditor General  [Click for full report]

Audit slams Nevada dental board’s system for handling records and complaints 
July 1, 2016 – Las Vegas Review Journal -- A legislative audit of the state dental board revealed systemic issues with the board’s operations, including poor record keeping and questions about how the public’s complaints are investigated. The report highlighted the Nevada State Board of Dental Examiners’ lack of an independent committee to review outcomes of investigations into misconduct allegations. The review process complaint was just one of several issues investigated by the auditors, who also found the board lacked “an effective process for accurately determining the amount of investigative costs” to dental professionals under scrutiny.  
Legislative Counsel Bureau – Audit Division  [Click for full report]

School funding not based on actual enrollment, audit says 
Sept. 30, 2016 – -- As Gov. Chris Christie trumpets his proposal to change how public schools are funded in New Jersey, an audit shows the state still failing to properly fund schools under the existing law. The audit shows districts aren't receiving state money based on their student population and demographics as dictated by the current school funding formula, according to a report from State Auditor Stephen Eells. That's left hundreds of districts with either overpayments or underpayments from the state. The report also found problems in special education and preschool funding. Money isn't being tied to actual enrollment numbers.  
Office of the State Auditor  [Click for full report]

Audit report on COFT reveals savings for the state  
June 18, 2016 – -- A state audit requested by Governor Mary Fallin takes a close look at the Council on Firefighter Training, finding a way to save the state thousands of dollars by possibly consolidating it with another similar program. “Council on Firefighter Training is a commission that`s set up, the money flows through the fire marshall`s office and then they`re in charge of working with different fire departments and training groups and stuff like that, recommended training,” State Auditor and Inspector Gary Jones said.  
Office of the State Auditor and Inspector  [Click for full report]

Tennessee doesn’t pay for enough teachers to meet its class size mandates, says state report 
Oct. 5, 2016 – Chalkbeat -- The state is considerably underestimating the number of educators needed to run Tennessee schools according to its own requirements, says a state comptroller’s report released Wednesday. And local governments are paying the difference. Statewide, districts employ about 12,700 more educators than the state funds, according to the comptroller’s Office of Research and Education Accountability, or OREA.  
Comptroller of the Treasury, Offices of Research and Education Accountability  [Click for full report]

Audit: State health department didn’t double-check $7 million in bills 
June 30, 2016 – Austin American-Statesman – The Texas Department of State Health Services failed to verify the prices on $7.6 million worth of drugs before paying a pharmaceutical company for them, according to an audit by the State Auditor’s Office. The audit released this month found that the agency did not verify that prices for drugs paid to pharmaceutical wholesaler Morris & Dickson Company were correct and complied with their contract’s pricing, terms and conditions.  
State Auditor's Office  [Click for full report]

Audit reveals UDOT needs better oversight of contractors, consultants 
Aug. 2, 2016 – Deseret News – Utah Department of Transportation contractors incorrectly installed 109 signs on a state road in Tooele and another dozen unsafe signs on Bangerter Highway, according to a new legislative audit. The Office of the Legislative Auditor General cited those mistakes as examples of UDOT needing to better oversee contractors and consultants on road projects.  
Office of the Legislative Auditor General  [Click for full report]

Amidst Abundant Rain, Eastern Virginia Still Faces Water Shortages  
Oct. 12, 2016 – Bacon’s Rebellion – After getting soaked with rain over the past two weeks, most Virginians would find it difficult to imagine that the Old Dominion could ever face a water shortage. But the Joint Legislative Audit and Review Commission (JLARC) has been thinking beyond next week’s weather forecast, and while there are no immediate threats to the viability of Virginia’s water supply, the commission say that the long-term outlook in eastern Virginia is problematic.  
Joint Legislative Audit and Review Commission  [Click for full report]

Audit calls for restructuring higher-ed panels
Sept. 20, 2016 – The – After a January 2016 audit by the Performance Evaluation and Research Division (PERD), research analysts are recommending the restructure of the West Virginia Higher Education Policy Commission (HEPC) and the West Virginia Council for Community and Technical College Education (CCTCE). PERD research analyst Christopher Carney said the audit found HEPC and CCTCE are functioning in an advisory capacity only and are not holding institutions accountable.  
Performance Evaluation and Research Division  [Click for full report]

Share your coverage with us! If you would like an article highlighted in our next newsletter, send a hyperlink to

NLPES Executive Committee News

The last several months have been busy ones. The 2015-2016 Executive Committee Committee met on July 18, 2016, (via conference call) and on Aug. 9, 2016, during NCSL’s Legislative Summit in Chicago, IL. Members provided updates on subcommittee work and the 2016 PDS. These meeting minutes have been approved and are available on the website.  

The 2016-2017 Executive Committee held its first meeting on Sept. 25, 2016, prior to the start of the PDS held in Jackson, Miss. Discussion included last minute details for the PDS; election of new officers (Linda Triplett—vice-chair; Shunti Taylor—secretary); and subcommittee work. Minutes will be posted after they are approved at the next EC meeting.  

Check It Out!

Check It Out!NLPES websiteSpend a few moments touring our NLPES website to learn more about NLPES and see what we do.  You’ll find general information about NLPES, including our bylaws, Executive Committee membership and subcommittees, state contacts, awards and information on peer reviews.  We also have a training library and resources including past meeting minutes, newsletters and more.  Check it out!

NLPES listservThe NLPES listserv is an email discussion group for NLPES members. By sending a message to, you can reach all listserv subscribers simultaneously.  Listserv members can query other states about evaluation work similar to their own projects, receive announcements about performance evaluation reports and job opportunities from other states, and are notified when the latest edition of this newsletter is available!  To join the listserv, send an email to Brenda Erickson, NCSL liaison to NLPES, with the subject “SUBSCRIBE to NLPES Listserv.”  Include your name, job title, audit agency/organization name, mailing address (including city, state, ZIP code), phone number and email address.  A “Welcome” message will be sent to you once you are successfully added to the listserv.  See the listserv link on the NLPES website for additional information on how to post messages and “netiquette” niceties, like not hitting “Reply All” when answering a query posted to the listserv.

Email symbolAre you receiving our listserv emails?  Some states’ systems block NLPES listserv emails. If you think you are not receiving our emails, please check your state’s security system and spam filters, and/or contact Brenda Erickson.

Legislative careers websiteKnow someone thinking about pursuing a career with a state legislature? Point them to the opportunities posted on NCSL’s legislative careers website. Job seekers can explore the types of work legislative staffers perform, including performance evaluation, budgeting, fiscal analysis, legal and policy research and opinions, bill drafting, public relations, librarians, building security and information technology support. Opportunities are posted by states offering positions under Legislative Jobs.  Attracting young people to work as legislative staff will be increasingly important in the coming years. And even though baby boomers make up about a third of the national workforce, nearly half of legislative staff responding to a survey were 50 years old or older. Replacing those staffers will present challenges. Check out the welcome video, “A Day at the Capitol,” and learn more about the opportunities and rewards of working for state legislatures. Watch the videos under Career Paths to hear from staffers in different states describing their jobs and the satisfaction they gain from a public service career.  

Stack of booksNLPES’ Professional Development ResourcesVisit our NLPES online training library for a variety of refresher and training materials!  There are nearly two dozen resources on planning and scoping, fieldwork, writing and publication, and management topics. Most are PowerPoint slides; some are narrated; a few are webinars or podcasts. Check them out!  

JLARC presentation on web reportingIn 2014, the state of Washington’s Joint Legislative Audit and Review Committee (JLARC) began issuing audit reports as webpages rather than PDF documents. Reports are now readily accessible on computers, mobile devices, and tablets. JLARC has received favorable comments on the change from legislators, legislative and executive branch staff and the public. Writing for the web requires a change in perspective, in addition to changes in technology. Visit JLARC’s website to see samples of their web reports. Contact Valerie Whitener, audit coordinator, JLARC, with questions. 

Ask GAO LiveAsk GAO LiveAskGAOLive is a 30-minute interface where GAO staff chat about a specific report and research, and answer questions that are emailed or tweeted in. Sessions are recorded and archived on the website. You can also “follow” GAOLive to receive advance notice of chat sessions. Topics include veterans and higher education, prescription drug shortages, prison overcrowding, state and local fiscal outlook, and government contracting.

Image of a capitol domeEnsuring the Public Trust—What’s the most common internal performance measure for evaluation shops? How many offices tweet? What percentage of staff has fewer than 10 years of experience? How can you contact a sister office in another state?  "Ensuring the Public Trust" summarizes information about legislative offices conducting program evaluations, policy analyses and performance audits across the country.



The Working Paper

The Working Paper  is published two times a year by the National Legislative Program Evaluation Society, a staff section of the National Conference of State Legislatures. NLPES serves the professionals of state legislative agencies engaged in government program evaluation. The purposes of NLPES are to promote the art and science of legislative program evaluation; to enhance professionalism and training in legislative program evaluation; and to promote the exchange of ideas and information about legislative program evaluation.

The Working Paper is produced by the NLPES Communications Subcommittee.
Patricia Berger, 2016-2017 chair (CA)
Shunti Taylor, newsletter editor (GA)
Emily Johnson (TX)

NCSL Liaison to NLPES
Brenda Erickson, 303-856-1391
NCSL Denver Office • 303-364-7700