NLPES logo

 

 

Contents

Related Sites

NLPES News

National Legislative Program Evaluation Society

A Staff Section of the National Conference of State Legislatures

July 2010

print version 

Chair’s Corner 
James Barber (Mississippi)

As you know, my term as chair of the NLPES Executive Committee ends at the close of the NCSL Legislative Summit in Louisville, Kentucky. I wanted to take advantage of this last Chair’s Corner to make some parting comments about the past year and activities of NLPES. 

First, I want to extend my thanks to all of you for allowing me the opportunity to serve as NLPES Chair. (Actually, this was my second turn as Chair.) The experience has been extremely rewarding both professionally and personally. In particular, I have enjoyed working with the other members of the NLPES Executive Committee as well as our NCSL staff liaison, Bob Boerner. It is obvious that there are many people scattered throughout the country who care deeply about the field of legislative program evaluation and auditing and are committed to the further advancement of our profession. 

Second, I would like to encourage all of you to become involved or remain involved with the activities of NLPES. As an organization, we have accomplished a lot over the years because of your interest and support. I realize that all of us face challenges because of shrinking state budgets and travel restrictions, but it is important for everyone to stay involved with NLPES. 

Third, I encourage all of you to consider attending the upcoming NLPES fall professional development seminar.  The NLPES Executive Committee’s Professional Development Subcommittee and the Performance Audit Division staff of the Louisiana Legislative Auditor’s Office have been working tirelessly to plan the 2010 fall professional development seminar (commonly known as the fall training conference). The seminar will be held October 4 – 7 in Baton Rouge, Louisiana. NLPES will co-sponsor the seminar with the National Legislative Services and Security Association (NLSSA), another NCSL staff section. The NLPES portion of the seminar will conclude on October 6, while NLSSA will continue to meet until October 7. Potential concurrent and plenary sessions include topics such as dealing with budget cuts, ARRA funding, effective audit planning and correctional cost savings. One highlight of the seminar will be a plenary session during which Commander Mark McGinnis, U.S. Navy SEAL, will speak. Commander McGinnis truly understands the term “leadership in challenging times” and how to translate his SEAL training and experiences to his civilian career. 

Fourth, I want to wish the new NLPES Executive Committee, chaired by Kathy McGuire (FL), a productive year. We all owe our gratitude to Executive Committee members who give of their time to keep the wheels of NLPES turning. Those serving on the “new” Executive Committee include: Dale Carlson (CA); Mitzi Ferguson (AR); Greg Fugate (CO); Patrick Goldsmith (LA); Lisa Kieffer (GA); Angus Maciver (MT); Tricia Oftana (HI); Scott Sager (WI); Karl Spock (TX); and, Andrea Truitt (SC). 

Last, be sure and read the “From the Editor” column on page six of the newsletter to learn of changes in the near future for NLPES. 

Again, thanks for the opportunity to serve you and good luck in the future!

Know Your Data!
Scott Sager (Wisconsin Legislative Audit Bureau)

“You guys are going to snicker, but I’m serious when I say ‘Be intimate with your data’,” said one of my statistics professors. Yes, we all snickered. However, we also learned how to systematically dismantle our data before and while conducting analyses to ensure that we were using the correct analytical methods with adequate controls and providing caveats to our findings to contextualize the scope of our findings. 

All datasets (even those you create) contain issues, errors, and irregularities that if not understood compromise conducting accurate analyses and drawing reasonable conclusions. Data validation should be done systematically and ideally, it should be ongoing throughout an audit. However, initial data validation should include tests to understand the information contained in a dataset, identify missing information, and identify outliers or other anomalies. Here are some basic tests you can consider doing when you first obtain a set of data: 

  • Consistency test, which is used to both identify instances of inconsistency in data input, e.g. analyzing whether a vendor’s name is spelled the same throughout a dataset, or whether a “date of birth” field and “age” field correspond correctly.

  • Limit test, which are similar to range tests, but focus on upper or lower bounds of data, e.g. conducting a test to identify all expenditures over a statutory threshold. 

  • Missing values test, which identify both the presence and number of missing values within a field. While missing values may not always indicate poor data entry, it can limit the types of analyses that can be performed and the conclusions drawn from them. 

  • Range test, which includes determining the range of values in a field, such as birthdates, and identifying whether the range is consistent with common sense or as defined by an agency’s data dictionary. This test is also useful in identifying outliers. 

  • Referential integrity test, which is used to determine whether the link between two or more data tables is constrained, e.g. does not reference all applicable rows. 

  • Unique values test, which is used to determine both the presence and number of unique values in a field. This can be useful in attempting to determine whether duplicate values, such as case ID numbers or social security numbers, are valid within a field or an indication of problematic duplication. 

During the course of an audit, your findings from these initial tests will allow you to create more narrow tests that can be useful in identifying new findings or developing data problems into findings.

NLPES Award Winners  

NLPES recently announced 2010 award recipients. NLPES chose the Florida Office of Program Policy Analysis and Government Accountability as the recipient of the 2010 Excellence in Evaluation Award. The award was presented to the office that had contributed the most to the field of legislative program evaluation during the four-year period beginning January 1, 2006 and ending on December 31, 2009.  

NLPES chose the Idaho Office of Performance Evaluations and Utah Office of the Legislative Auditor General as recipients of the 2010 Outstanding Evaluation Methodology Award. 

NLPES chose George Schroeder as the recipient of the 2010 Outstanding Achievement Award. George began his career in program evaluation in 1973 when he was appointed director of the Joint Legislative Program Review and Investigations Committee of the Connecticut General Assembly. In November 1975, he was appointed director of the newly formed South Carolina Legislative Audit Council. He served for 33 years as the director of the SCLAC until his retirement in February 2009. George was a founding member of the National Legislative Program Evaluation Society. 

The following states received Certificates of Impact: Arizona, Arkansas, California, Colorado, Florida, Georgia, Hawaii, Idaho, Illinois, Louisiana, Michigan, Minnesota, Mississippi, Montana, New Mexico, North Carolina, Tennessee, Texas (State Auditor), Texas (Sunset Commission), Utah, Washington, West Virginia, and Wisconsin.


Louisiana Capitol Building

Fall Professional Development Seminar 

Effectively Meeting the Demands of Our Customers: the Legislature, Local Government and Public

Co-Sponsored by the National Legislative Program Evaluation Society (NLPES) and National
Legislative Services and Security Association (NLSSA)  

 

October 4 – 7, 2010
Hilton Hotel
Baton Rouge, Louisiana
The NLPES portion of the seminar will conclude on October 6, while NLSSA will meet until October 7.

 

Picture of people in group meetingAt the fall professional development seminar, you will build your skills, expand your understanding of current legislative issues, and network with colleagues from across the country.

Session Topics (Sample list)

  • Information Technology
  • Dealing With Budget Cuts
  • Vehicle Fleets
  • Tools for Communication
  • ARRA Funds
  • Charter Schools
  • Lotteries
  • Correctional Cost Savings
  • Effective Audit Planning
 

Mark McGinnis, Commander, U.S. Navy SEAL Navy SEALS Plenary Speaker:  Mark McGinnis, Commander
U.S. Navy SEAL Navy SEALS

are the elite of the Special Operations Forces. Mark successfully completed this training and went on to lead more than 400 SEAL operators through real world operations and training evolutions with zero mishaps. He truly understands the term leadership in challenging times and how to translate his training and experiences to his civilian career.

National Conference of State Legislatures
7700 E. First Place Denver, CO 80230
(303) 364-7700
www.ncsl.org/nlpes

 NLPES Training Products Matrix

Over the past year or so the NLPES Professional Development Subcommittee has been developing and/or collecting training modules for the benefit of NLPES member offices. Listed below is a matrix of modules that are presently posted on the NLPES website. Some are designed to be classroom-style presentations, while others are designed for individual use. Please review the listing on the NLPES webpage frequently as other modules are posted.

 Critical Thinking

For the most part, the work of legislative program evaluators involves solving problems.  To identify problems, staff must be able to think critically and solve problems.  This PowerPoint describes the "critical thinking" concept and provides classroom exercises for applying the concept. (PowerPoint, 51 slides

 Finding Savings

Identifying opportunities for cost savings and cost avoidance are always important, but particularly so in times of budget shortfalls.  This PowerPoint offers concrete suggestions on there to look to find savings or additional revenues. (Narrated PowerPoint, 10 slides)

 

Interviews

Interviewing is a critical part audit and evaluation work.  Conducting good interviews takes time and practice.  This PowerPoint provides information on basic interviewing skills. (PowerPoint, 20 slides)

 Quantitative Methods

Hard numbers can provide compelling evidence.  This PowerPoint provides an overview of four quantitative methods: central tendency, regression, survival analysis, and path analysis.  (PowerPoint, 32 slides)

 Samples

Properly executed samples allow us to collect less information but reach reliable conclusions.  This PowerPoint provides an overview of types of samples, explains when each is appropriate, and discusses how to determine sample size. (PowerPoint, 14 slides) 

 Basic Survey Techniques

Getting good survey data starts with survey development.  This PowerPoint discusses determining your survey goals and objectives, sample selection, methodology, survey layout, and question and response sets.  (PowerPoint, 29 slides) 

 Survey
Research

Developing a good survey is harder than it looks.  This PowerPoint explains what is survey is, when to use a survey, how to write questions, how to format the questionnaire, and how to administer the survey and assess the response rate. (PowerPoint, 36 slides) 

 Rapid Response Assistance

More evaluations offices are being asked to provide information quickly to meet Legislative research needs.  This narrated PowerPoint describes how to quickly organize and execute rapid response assistance requested by the legislature.  (Narrated PowerPoint, 5 minutes in length) 

 Report Writing

Every report’s usefulness hinges on its ability to communicate effectively.  This PowerPoint discusses keys to crisp writing, including tips at the sentence, paragraph, and overall structure levels, as well as how to provide a good final polish. (PowerPoint, 48 slides) 

Legislative Code of Conduct

Legislative staff must sometimes make difficult judgments to guide their service to their state legislatures.  This PowerPoint provides basic concepts to help staff develop a sense of responsibility for their actions. (PowerPoint, 21 slides) 

 Managing Millennials

Integrating new employees is an important part of maintaining an effective staff.  This pod cast offers insights into the working style of millennials, as well as other generations of employees, with the objective of helping managers understand employee needs and preferences. (Podcast, 10 minutes in length) 

 Workplace
Mentoring

Mentors can help new staff understand your work environment and expectations.  This narrated PowerPoint addresses three questions: what is mentoring, what are the different types of mentors, and how can mentors be successful?  (Narrated PowerPoint, 8 minutes in length) 

Succession Planning

Most legislative program evaluation and audit offices face a future in which senior staff will be retiring.  This narrated PowerPoint outlines basic concepts for the transfer of knowledge and responsibilities from senior to newer staff. (Narrated PowerPoint, 8 minutes in length) 

Getting Along With the Auditee    

Editor’s Note: This article originally appeared in the December 1984 issue of the LPES Report, the predecessor to the NLPES News. The article was written by Douglas West, a former manager with the Utah Legislative Auditor General’s Office.
 

Many factors exist that can easily spoil our relationships with the auditee. At the start of most audits, we usually must deal with suspicious agency managers who are absolutely convinced that the audit is politically motivated. If they are not concerned about politics, they can feel strongly that audit groups like ours do not have the expertise or skills to evaluate their programs. As the audit progresses, we will, at some point have to defuse a potentially dangerous situation caused by a personality conflict between an audit team member and an agency manager or employee.

There is no simple formula or prescription that can be used to eliminate problems, but there are some positive steps we can take. Probably the most important is to recruit evaluators and auditors with good interpersonal skills. During the interview process, we look for candidates who can relate to a broad range of individuals. We want them to be able to deal with the DOT construction worker and the university president both on the same day. These skills are not easily taught, so recruitment interviews are crucial in identifying staff who have them. 

Even the most competent evaluators and auditors are going to have problems with the auditee sometime during an audit. To promote cooperation, evaluators and auditors must be willing to share information. In our office, we encourage the audit team leader to meet regularly with agency management. They are told to discuss all findings with agency staff and not put off giving them negative or critical information until the end of an audit. Our goal is no surprises at the exit conference. A successful technique we have used for years is a pre-exit meeting. This meeting gives agency management a chance to review the report informally with the audit team before involving the Auditor General. 

The process of regularly sharing information with the auditee produces some positive benefits that make a difficult job somewhat easier. Information is much easier to obtain when the evaluator or auditor is trusted and has credibility with the auditee. Credibility with the auditee gives the evaluator or auditor a better chance of seeing his or her recommendations implemented. Sharing information ultimately can create an audit environment free of conflict where the evaluator or auditor and auditee work cooperatively to improve program operations. 

Whatever action we take, there will always be problems, because audits and evaluators or auditors are never going to be popular with most agency managers. Our experience over the years, however, indicates that we improve our chance for positive program changes when we cooperate rather than fight with agency management. 

Office Happenings

Arizona 

The Arizona Office of the Auditor General announces the following retirement and promotions. Bill Thomson, long-time Deputy Auditor General, retired in June after dedicating 31 years to the Office. Melanie Chesney, Performance Audit Division Director, was promoted to Deputy Auditor General. Melanie began her career with the Office sixteen years ago and has served as the Director of the Performance Audit Division for the past nine years. Taking Melanie’s place as the Director of the Performance Audit Division is Dale Chapman. Dale began his career with the Office in the Performance Audit Division more than 18 years ago and has served as the manager in the Performance Audit Division for the past 12 years.

Colorado 

The Colorado Office of the State Auditor announces that Jenny Page has been promoted to Audit Manager and Jonathan Caldwell has been promoted to Project Leader/Independent Contributor. In addition, Jason Morrison has been hired as a Senior Legislative Auditor and Beatriz Bucher, Cassi Clark, Andrew Gleaves, and Karl Merida were hired as Legislative Audit Interns. Jenny Atchley has been promoted to Communication Analyst and Lesa Fisher has been hired as Executive Administrator to the State Auditor. 

Minnesota 

The Minnesota Office of the Legislative Auditor recently hired Sara Okos as a permanent program evaluator. Sara came to OLA from Virginia, where she was a policy analyst at the Commonwealth Institute for Fiscal Analysis. 

Texas (State Auditor) 

The State Auditor’s Office has promoted Anita D’Souza, CFE, to Chief of Staff and General Counsel; Kelly Linder, CIA, CGAP, to Assistant State Auditor; and, Jim Timberlake, CIA, to Audit Manager. As Chief of Staff, Anita will be the State Auditor’s official designee for all office related matters. Kelly will oversee ARRA audits and multiple performance audits and manage the SAO’s Federal Single Audit contract. Jim will primarily manage the Federal Single Audit. 

Wisconsin 

The Legislative Audit Bureau welcomes both Andrew McGuire and Jacob Schindler to the Program Evaluation Division as Legislative Program Analysts.

From the Editor

At the risk of sounding overly dramatic, “Elvis has left the building!” I thought I would use that well-worn statement to let you know that this is my last issue to serve as editor of the NLPES News. During its May meeting, the NLPES Executive Committee, with my encouragement and support, voted to make changes to the newsletter. Rather than having a single editor, the newsletter will become a responsibility of the Executive Committee’s Communication Subcommittee. This change will allow more individuals to have input into the design and content of the Society’s newsletter. Another change that will occur is that there will no longer be a hard-copy version of the newsletter. In the technology age in which we all live, many organizations have gravitated to online communication. NLPES will join that trend and post the newsletter on the NLPES page of the NCSL website. This will reduce printing and mailing costs associated with the newsletter and will allow us to issue the newsletter more quickly. 

In addition to changes in the newsletter, the Executive Committee will also experiment with Wikis, forums and other online discussion technology. This will expand the means by which the Executive Committee can share information and encourage “real-time” interaction among our members. 

Having said all of that, I want to thank those of you who have assisted me with the newsletter over the years. I have been newsletter editor since the spring of 2001 and have cajoled many of you into writing articles for inclusion in the newsletter. Rarely did anyone turn down my request for help. I also need to thank Dr. Max Arinder, PEER Committee Executive Director, for allowing me to take time away from my “day” job to work on the newsletter. It was not always easy to juggle many responsibilities, but things always seemed to come together just in time to send the newsletter to the printer. 

Thanks again for this opportunity to serve NLPES.

James Barber


NLPES-NEWS is published three times a year by the National Legislative Program Evaluation Society, a staff section of the National Conference of State Legislatures.  NLPES serves the professionals of state legislative agencies engaged in government program evaluation.  The purposes of NLPES are:

  • To promote the art and science of legislative program evaluation.
     
  • To enhance professionalism and training in legislative program evaluation.
  • To promote the exchange of ideas and information about legislative program evaluation.
     


NLPES News Editor: James Barber (MS)
601-359-1226
www.ncsl.org/nlpes/

National Conference of State Legislatures • 7700 East First Place • Denver, Colorado 80230 • 303-364-7700

NLPES Logo