The Working Paper



Lisa Kieffer (Georgia)
Picture of NLPES Chair, Lisa KiefferHere we are at the beginning of a new NLPES year! With the new year comes a new slate of committee officers. At the Professional Development Seminar in Austin, I became chair, Wayne Kidd (Utah) became vice chair, and Rachel Hibbard (Hawaii) became secretary. Along with all the members of the executive committee, we’re looking forward to serving you this year.

When I started thinking about the upcoming year, I was reminded of our accomplishments from last year—and the bar we’ve set for ourselves. Attendance at our Professional Development Seminar in Austin continued an upward trend, with 132 people from 30 states attending— including representative from some states we haven’t seen in a while. Many thanks, again, to our Texas colleagues for their warm hospitality and a strong PDS that was very well received. We also continued another upward trend last year with the number of applicants for our Excellence in Evaluation award showing an increase. Finally, we continued to push out more and more information and training content to our members through the NLPES website.

So with that as our starting point, we’re excited about the opportunities this year brings. Have you seen the new website? We have a new look! After much work on the part of NCSL and your Executive Committee, we have a fresh new look! We hope the new website will be easier to navigate and more accessible.

I’m also pleased to announce the location for our 2014 PDS will be Raleigh, N.C. We are looking forward to continuing our partnership with the National Legislative Services and Security Association (NLSSA) who will also be meeting in Raleigh. We are also working on additional training opportunities through webinars—so keep an eye out for announcements.

And finally, I’d like to make a push for your active involvement with NLPES. There are many ways you can take advantage of the resources this group has to offer, such as: contact a member of the Executive Committee with your ideas; brag on your office’s accomplishments through our awards process; collaborate with colleagues through our listserv; hoot about your office happenings in our newsletter; and use the resources, contacts, training materials, etc. available on our website. And finally, consider running for a place on the Executive Committee! We welcome your involvement and appreciate your commitment to continuing and sustaining NLPES, the only national organization dedicated to advancing legislative program evaluation.


Report Radar

Graphic magnifying glass on notepadAngus Maciver (Montana)
Greetings auditors and evaluators and welcome to another fun-packed edition of Report Radar. This tour of the government accountability landscape leads us to valuable reports addressing economic development incentive programs, port infrastructure districts, utility regulation, and state lottery operations. We also re-visit the always-popular “miscellaneous” category to highlight some familiar and some not-so-familiar topics.

Incentive programs aimed at fostering economic development are familiar territory for many of us. There is, however, a renewed push to subject various tax incentives and other economic development programs to more scrutiny. Four recent reports from member offices highlight this trend. In September, Kansas released the first of a multi-part review of economic development incentive programs. This initial report focuses on two programs that use tax breaks to encourage employment and workforce development and identified issues with data reporting, management oversight and claw-back provisions for incentive recipients. Meanwhile, neighbor state Nebraska also released a report in September looking at economic development incentives. The Nebraska report features a nine-state regional comparison of different tax incentives, as well as evaluation and reporting requirements. For any of us asked to undertake similar multi-state comparisons, this report is an excellent starting point. Our people in Utah are also busy in this area: in October they released a report on an incentive program aimed at promoting the state’s knowledge economy. The audit found issues with the way the program reported jobs created and return on investment, as well as other management and governance problems. Last, but not least, anyone interested in special taxing districts or similar local government financing programs for economic development should read a recent report from Washington. This work focused on a state program that provides funding for local infrastructure projects designed to promote economic development. Read the Washington report

One specific economic development activity common to many states is the development of port infrastructure and associated special districts. If port infrastructure and operations are on your radar, you should look at a recent report from our friends in Mississippi, which addresses restoration efforts underway at the port of Gulfport. The report details progress on restoring port facilities damaged by hurricane Katrina in 2005 and discusses job creation goals associated with the project. You should also read a July report from Illinois that discusses management and operational issues at the Illinois International Port District.
Read the Mississippi report

Moving on to utility regulation. Maine released a report in September looking at the state’s Public Utilities Commission with specific focus on how this regulatory agency handles consumer complaints about utility providers. Another interesting aspect of utility regulation is addressed in a June report from Delaware, which reviewed the operations of the state’s Sustainable Energy Utility. This organization was established as a nonprofit corporation and works with a state agency to promote sustainable energy. Read the Delaware report

State lotteries are a frequent subject matter for many of us, so you are sure to enjoy two reports from member offices discussing the agencies that administer lotteries. First up is Colorado, which released a report in October addressing how its state lottery could realize efficiencies and cost savings in its operations. Findings included discussion of how prize payouts should be evaluated to ensure they optimize sales. Another report worthy of your attention comes from Wisconsin, where a biennial program evaluation of the state lottery is a statutory requirement. The Wisconsin report discusses various aspects of lottery operations, with specific emphasis on licensing arrangements for instant games and retailer sales incentives. Read the Wisconsin report

Miscellaneous is one of our favorite categories here at Report Radar, mainly because it gives us an opportunity to showcase the amazing variety of issues our members address. These often involve issues many of us have never given much thought to before, such as a fascinating report from Minnesota on highway noise barriers. This report is full of interesting information on the construction of highway noise abatement barriers. Or maybe you want to know more about the use of digital textbooks in schools and how this might impact classrooms and school systems? Turns out there is a report addressing these issues from Tennessee, released in October, and is a great introduction to the subject. Or maybe you want to know more about an issue that has been covered previously in Report Radar, like state oversight of charter schools? Well, there is a new report from Arizona addressing the state’s Board of Charter Schools, which will certainly provide useful information. Read Arizona's report

Remember, Report Radar is always working on its next issue, so keep them coming. See you in 2014.

Staff Spotlight

Graphic magnifying glass on notepadRachel Mercer-Smith (New Mexico)
I came to the New Mexico Legislative Finance Committee (LFC) by way of the public school classroom. Before joining the LFC as a program evaluator in June 2012, I was a high school teacher and Teach for America Corps Member on the Navajo Nation in Shiprock, N.M. My interest in the education achievement gap developed during college, and I returned to New Mexico, my home state, to teach after completing a B.A. in political science and psychology from Rice University.

I planned to attend law school after teaching for several years, but my interest in the public school system led me to pursue opportunities in public policy and government. Since joining the LFC, I have worked on several projects, including an evaluation of teacher preparation programs in New Mexico, an assessment of the adequacy of the state’s healthcare workforce, and an evaluation of the impact of the state’s early childhood programs on student achievement.

I enjoy the analytical and research aspects of my work at the LFC, and feel that my role as a program evaluator supports the organization’s mission. As part of an evaluation team identifying opportunities to experience improved outcomes for state programs, our work can produce positive outcomes for New Mexico. For example, performance audits have helped the state better understand the preparation factors that enable teachers to lead students to significant academic growth and identify the elements of early childhood programs that are producing lasting impacts for children in New Mexico. Evaluation field work has also allowed me to gain a broad understanding of programs operating in New Mexico, and I have traveled to parts of the state that I have never before experienced, including Hobbs and Silver City!

I am currently working on a team evaluating the state’s funding of instructional materials for public education classrooms and completing a master’s in public administration at the University of New Mexico.

Rachel Mercer-Smith is a program evaluator with the New Mexico Legislative Finance Committee.

State Profile | Legislative Division of Post Audit

Picture Kansas State Capitol Dome Katrin Osterhaus (Kansas)
The Legislative Division of Post Audit (known as LPA) provides auditing services to the Legislature at the direction of the Legislative Post Audit Committee, a bipartisan committee of five senators and five representatives. LPA helps evaluate whether state government agencies and programs:


  • Are being run efficiently.
  • Are effective in accomplishing their goals.
  • Have adequate information technology security policies and procedures.

LPA also contracts with public accounting firms to perform a variety of financial audits required by state or federal law, and cover the state's accounting system and certain state agencies.

We have 19 auditors plus an information management technician and an administrative officer. Three audit staff are dedicated to conducting IT performance audits. Our financial auditor oversees roughly six contracted financial audits. An administrative auditor oversees our annual report, follow-up reports with agencies, and quality control reviews. Our audit staff have an average of 9.5 years with the office, with a healthy mix of newcomers (0-5 years on the job), longer-time staff (5-10 years), and those who keepsake the institutional memory of the office (i.e., “old farts” because they’ve been here 15+ years).

Our audit staff have a cross-section of JD’s, master's, or bachelor's degrees in business or political science; and undergraduate or graduate degrees in theater, history, and psychology. A number of our audit staff have special certifications including CIA, CGAP, CISSA, and CFE; and one is a certified professional dog trainer (yes, we have used that qualification in an audit setting)!

Most of our staff is homegrown right here in Kansas, though some of us come from as far away as California, South Dakota and even Germany.

Noteworthy changes
Over the past five years, we’ve adopted electronic work papers, marketed electronic reports rather than paper copies, and increased our flex hour and telecommunicating options to further improve our office’s productivity and efficiency.

The worst places we’ve had to audit
Our informal staff poll revealed the following as our worst audit experiences:

  • Spending days and days in a dusty, dingy warehouse to review corporate income tax documents.
  • Standing at the tray-return window in the chow hall at Kansas State Penitentiary asking inmates why they didn’t finish all their meals.
  • Descending into the Kansas underground salt mines used for record storage. We were greeted by rats along the 650-foot descent in a rickety old elevator (before it became a tourist attraction).
  • Getting stuck in the mud in the Kansas farm fields while trying to locate certain silos for a property taxation audit.
  • During an audit of dog tracks, watching trainers try to get urine samples from race dogs immediately following races with little Dixie cups on a long pole.
  • In a foster mother’s home, being bitten on the behind by a ferocious, man-eating wolf-dog (not really!).

Annual office traditions
We all look forward to our much anticipated United Way jeans week fundraiser, and every year we adopt a family for Christmas (we buy gifts and a holiday meal for them). Our December committee meeting generally involves the publication of several reports for the upcoming session and, more importantly, our holiday luncheon! We meet our legislative committee members on our (well decorated) turf to chat with them while savoring the food spread. We later hold a holiday party of a much different nature, which takes place at the home of a volunteering (and brave) staff and without committee members….

Research and Methodology Profile: Investing in Preschool Programs

Spelling blocks "Preschool"Jon Courtney (New Mexico)
Our Legislative Finance Committee recently asked us to evaluate whether higher spending on quality preschool programs results in better outcomes for participants who receive state subsidies.

Research shows that high quality early childhood programs can make a lasting difference. Evidence-based programs, such as the Perry Preschool Project and the Abecedarian Project, have lasting effects on education and life outcomes. Early literacy is a well-established predictor of outcomes including graduation, college attendance, future earning potential and later interaction with the justice system.

New Mexico has focused on improving school readiness and early literacy, as reflected in increased allocations to early childhood programs and investments in high quality child care. Since 1997, the state has paid differing child care subsidy reimbursement rates based on quality ratings. Providers with higher quality ratings receive higher subsidies, presumably leading to better outcomes.

The state is now implementing a third-generation quality rating improvement system for child care, even though it has never measured the previous two systems on outcomes such as student achievement, special education participation rates, or third-grade retention rates.

How we approached it: To address LFC’s request, we entered into a data sharing agreement with the state education department and the state child care agency to address privacy concerns and security of data. We merged early childhood program quarterly participation data, student demographics, third-grade reading and math standardized test scores, and school level information for about 260,000 students. From this, we analyzed third-grade outcomes for a cohort of three- and four-year old children, following approximately 50,000 students over a seven-year period. Of these students, 6,200 participated in child care and 3,700 participated in the state-funded prekindergarten program. 

There were a number of concerns with analyzing the data. For example, since it was not a randomized control study, there was a potential for selection bias. To minimize this impact, we used propensity score analysis. Another concern was that lay readers would want an “apples to apples” comparison of participants to non-participants; to address this, we drew a stratified sample from the population based on demographic information, such as poverty status. Child care providers were concerned that by the time a student reaches third-grade, his or her school or school district might account for a student’s progress, or lack thereof, more than his or her participation in an early childhood program. To address this, we used a hierarchical linear modeling approach, which was able to reflect the multilevel nature of the data—e.g., students are within classrooms, which are within schools, which are within school districts. This allowed us to understand the impact of contributing factors such as individual demographic characteristics, classroom organization, and school district policies on third-graders’ performance outcomes.

Our conclusions: When we evaluated state-subsidized child care, several of our statistical methodologies converged on similar results. They suggested that, regardless of program quality rating or frequency of attendance, participation in childcare is not associated with better outcomes on third-grade reading or math scores. A number of classroom- and individual-level variables were significantly related to student achievement, including differences in school composition and individual demographic variables. Interestingly, the state-funded prekindergarten program was significantly related to higher third-grade math and reading scores, higher proficiency rates, a 43 percent reduction in special education participation by the third-grade, and an 83 percent reduction in school retention. Thus, one year of participation in a $3,000 prekindergarten program has the potential to save a state tens of thousands of dollars in future costs.

When we compared program standards of high-quality child care versus state-funded prekindergarten, we found higher program standards in prekindergarten programs, including teacher qualifications and environmental ratings standards. This supported our statistical analysis and prompted our recommendation that the Legislature establish a statutory framework for high-quality child care modeled on prekindergarten standards. We also suggested generating, in future, a formal cost-benefit analysis through the Results First model based on the number of participants in prekindergarten.

Jon Courtney is a program evaluator with New Mexico’s Legislative Finance Committee. He has a Ph.D in experimental psychology and also plays in a rock band. He may be regretting having revealed that at LSMI. Read the report

News & blurbs

NLPES Professional Development Seminar (PDS) and Executive Committee News

Picture of children petting giraffThis year’s PDS was held Sept. 23-25 in Austin. The conference was (we think) very successful, and supplemented by the fantastic hospitality of our hosts, the Texas Sunset Advisory Commission, the Texas State Auditor’s Office, and the Texas Legislative Budget Board (thank you!). The seminar boasted 15 sessions over 2.5 days and was attended by 132 people from 30 states, the highest in several years. As has been done in the past and will be again in 2014, NLPES partnered with NLSSA (National Legislative Services and Security Association) for administrative aspects and social events to help leverage costs.

Aside from being kissed by a giraffe (Katrin Osterhaus in photo at left) at what is affectionately known as ‘The Dump’ (a private waste/recycling facility and game preserve outside Austin), highlights of the PDS included a roundtable session on keeping and retaining top talent; Peter Heinneccus’ highly engaging presentation on Washington JLARC’s research methods; and an illuminating discussion on communicating with legislators. Thank you also to the 50 people who submitted an evaluation summary; responses will be used to help guide and shape next year’s offerings.

The Executive Committee met on Sept. 22, prior to the start of the PDS. Discussion included last minute details for the PDS; election of new officers (Lisa Kieffer—chair; Wayne Kidd—vice chair; Rachel Hibbard—secretary); NCSL website redesign (new site went live in October); and subcommittee work by the Communications, Awards, Professional Development, Peer Review, and Elections subcommittees. Minutes will be posted after they are approved at the next EC meeting; prior minutes are available on the NLPES website. For more information on the EC or to contact a member or members, see our website.

Staff Happenings

Jennifer Jones has been promoted to deputy director of the Texas Sunset Commission—congratulations! The commission has also hired seven new policy analysts.

Please let us know if you have staff happenings to share! Email

Red graphic of words "Check it out" NLPES website: Learn more about NLPES and see what we do—spend a few moments touring our NLPES website on the new NCSL website. You’ll find general information about the NLPES, including by-laws, executive committee membership and subcommittees, state contacts, awards, and information on peer review. We also have a training library and resources including past meeting minutes, awards, newsletters, and more. Check out our new website

NLPES listserv

Join our NLPES listserv and:

  • Get our NLPES Newsletter delivered directly to your email box
  • Query other states about evaluation work similar to what you’ve just been assigned
  • Receive email announcements about performance evaluation reports from other states.

Simply send a blank email to with the subject area blank and the word “SUBSCRIBE” in the body. You will receive a welcome message if successful (the response may take a little while). See the listserv link on the NLPES website for additional information on how to post messages to the listserv and “netiquette” niceties, like not hitting “Reply All” when answering a query posted to the listserv. You’ll be glad you joined! NLPES listserv

Legislative careers website: Know a young professional thinking about pursuing a career with a state legislature? Point them to the opportunities posted NCSL’s legislative careers website. Job seekers can explore the types of work legislative staffers perform, including performance evaluation, budgeting, fiscal analysis, legal and policy research and opinions, bill drafting, public relations, librarians, building security, and information technology support. Opportunities are posted by states offering positions under Legislative Jobs.
Visit the career website 

Launched by NCSL in June 2012, this is a great website. According to NCSL, attracting young people to work as legislative staff will be increasingly important in the coming years. And even though baby boomers make up about a third of the national workforce, nearly half of legislative staff responding to a survey were 50 years old or older. Replacing those staffers will present challenges.

Check out the welcome video, “A Day at the Capitol,” and learn more about the opportunities and rewards of working for state legislatures. Watch the videos under Career Paths to hear from staffers in different states describing their jobs and the satisfaction they gain from a public service career. Visit the career website

Graphic of words "Upcoming Events"Upcoming Events

This is a quiet time of year for the newsletter as we all gear up to focus on our “day” jobs while legislatures are mostly in session over the winter. The NLPES Executive Committee will continue to work behind the scenes, though, and will meet briefly in the spring. We’ll keep you apprised of news and issues via the NLPES listserv, and look for the Spring 2014 edition of the The Working Paper in May 2014!

2014 Professional Development Seminar: will be held in Raleigh, N.C. in the fall. Details and dates to be shared when they are available.

The Working Paper is published three times a year by the National Legislative Program Evaluation Society, a staff section of the National Conference of State Legislatures. NLPES serves the professionals of state legislative agencies engaged in government program evaluation. The purposes of NLPES are to promote the art and science of legislative program evaluation; to enhance professionalism and training in legislative program evaluation; and to promote the exchange of ideas and information about legislative program evaluation.
Visit the NLPES website

2013–2014 NLPES Communications Subcommittee:

Dale Carlson (CA)
Charles Sallee (NM)
Rachel Hibbard, newsletter editor (HI)

NCSL Liaison to NLPES:
Brenda Erickson, (303) 856-1391
NCSL Denver Office • (303) 364-7700