The Working Paper is the official newsletter of the National Legislative Program Evaluation Society. NLPES serves the professionals of state legislative agencies engaged in government program evaluation. The purposes of NLPES are to promote the art and science of legislative program evaluation; to enhance professionalism and training in legislative program evaluation; and to promote the exchange of ideas and information about legislative program evaluation. Read the full newsletter in PDF form.
Inside the Issue
Chair’s Corner | Eric Thomas, Washington
Do you remember February? Spy balloons, Grammy Awards, Super Bowl, and of course, the last NLPES Working Paper. Seems like both yesterday and years ago.
A lot has changed in the NLPES world, and I’m happy to share the news!
Megan McClure is the new, permanent liaison for NLPES
With eight years at NCSL, Megan is well qualified to serve as our new liaison. She has been tremendously helpful in planning for the Professional Development Seminar in Santa Fe, N.M. (yes, that’s a shameless plug for the event). Filling Brenda Erickson’s shoes is bound to be a formidable task, but Megan is up to the challenge, bringing enthusiasm and a fresh perspective. We look forward to working with her and see the staff profile below to learn more about Megan!
Re: [nlpes-l] Re: Fwd: Re: Listserv replacement due this summer
NCSL informed us that a replacement for the ailing Listserv will be rolled out this summer. Our colleagues from the National Association of Legislative Information Technology (NALIT) have piloted the new platform, Breezio, and were very impressed with its functionality. This has perhaps taken longer than folks may have wished, but it should allow us to resume and perhaps improve communication and information sharing.
29 offices received 2023 NLPES awards
Congratulations to the winners! Take some time to review the reports that were recognized for Excellence in Evaluation and for the Certificates of Impact. The breadth and quality of work that our offices are completing is quite impressive. Thank you to Drew Dickinson, the awards team and everyone who volunteered to serve as a judge. It is a significant time commitment and much appreciated.
The NLPES Executive Committee has new members and leaders
Congratulations to the two new members joining the 2023-2024 Executive Committee: Mike Powell (Maryland) and Adam Fridley (West Virginia).
Congratulations and much appreciation to Darin “DRU” Underwood (Utah) and Jennifer Sebren (Mississippi) who will be the new Chair and Vice Chair, respectively. They are also beginning new terms on the committee. They bring a wealth of knowledge, experience, and enthusiasm that can serve us well.
Kristen Rottinghaus is leaving the Executive Committee
After all of the good news, some sad news. Among her many accomplishments, Kristen Rottinghaus served as Chair during the pandemic and was key to maintaining NLPES’s trainings during that time. Even more significantly, she increased the variety and frequency of trainings and began inter-office information sharing calls. She has been a great advocate for NLPES and the work that we all do. We hope that this move gives her more time to learn about whiskey (be sure to read her 2022 interview). Thank you!
Eric Thomas is legislative auditor for Washington state and the 2022-23 NLPES Executive Committee chair.
Previewing the 2023 PDS
Micaela Fischer, New Mexico
The Program Evaluation Unit of the New Mexico Legislative Finance Committee is proud to host this year’s NLPES Professional Development Seminar (PDS) from Oct. 31 to Nov. 2 in beautiful Santa Fe, N.M. As the premier gathering for legislative program evaluators, attendees of the PDS have come to expect high-caliber learning sessions, and the 2023 PDS will be no exception. In addition, as this will be the first in-person NLPES-specific gathering since before the pandemic, there will be a renewed focus on networking and facilitating the exchange of ideas between evaluation colleagues from around the nation. Stay tuned for more information soon about the PDS agenda.
The PDS will be held in the Drury Hotel, formerly the city’s old St. Vincent’s Hospital. Room 311 is allegedly haunted, so stay there if you want to double down on some Halloween spookiness. For down times, the Drury is near many Santa Fe sites, including the historic and beautiful Cathedral Basilica. Just across the street from the Basilica is the incredible Institute of American Indian Arts Museum of Contemporary Native Arts. One block further, you’ll be in the middle of Santa Fe’s Plaza, home to the oldest public building in America. It is the center of town and a lovely park, and simultaneously a place of a deep and complicated history with which many communities of Northern New Mexico are still actively reckoning.
Following the theme of connecting and networking, the PDS will include a reception in the New Mexico State Capitol. Known as the “Roundhouse,” the capitol is home to a world-class art collection featuring over 600 artists. All our staff have their favorite pieces, and I hope you’ll find one that speaks to you by the end of the trip as well. Between the art, the camaraderie, and the food (THE FOOD! — I need 350 more words to appropriately wax poetic about our food), I know we’re all going to come away from the PDS this year inspired for the year ahead.
Things to know before you come:
- Santa Fe sits at an elevation of 7,000 feet in the high desert so don’t forget your sunglasses and your #emotionalsupportwaterbottle to stay hydrated!
- Santa Fe is a small but bustling tourist town. If you have your heart set on eating at one of our many, many amazing dining establishments, it is probably best to make reservations a few weeks ahead of time. Hot tip: Know your answer to our state question before you dine.
- Events for the conference will be within walking distance of the hotel, but a vehicle will likely be required for those who want to venture out further. Ridesharing, taxis, and public transportation are very limited in Santa Fe.
Finally, there are plenty of other attractions worth seeing if you decide to stay a little longer. Some of my favorites are The Original Meow Wolf, our many hiking options, the Santa Fe Farmer’s Market (on Saturday mornings), or one of our standout museums such as the Georgia O’Keeffe.
Micaela Fischer is a program evaluation manager for the New Mexico Legislative Finance Committee and serves on the NLPES executive committee.
Virginia’s JLARC Evaluates the Income Tax System
Justin Brown, Virginia
Sometimes developing the right research methodology is only half the battle. Even the best research is not useful if those who need it to make decisions do not understand its meaning. Virginia’s Joint Legislative Audit and Review Commission (VaJLARC) was recently tasked with a project that required equal attention to designing and implementing complex research, but also being purposeful about explaining the results of the research in a relatively clear and understandable way.
The legislature asked VaJLARC to evaluate the state income tax and propose options to make it more progressive. This presented the staff with two distinct challenges. First, the research team needed to decide how to quantitatively measure how specific changes to the state income tax would affect tax progressivity. Second, the team needed to find a way to simply communicate to key legislative stakeholders how each change they could make would affect the after-tax income of their constituents.
The VaJLARC team decided to request from the state tax department filing data for the more than 4 million individuals who filed a state income tax return. The agency’s chief quantitative and economic development analyst used the file to create a baseline of the effective tax of all filers, by income decile. This baseline would allow a point of comparison for each option to change the income tax (e.g. raise the standard deduction, raise the filing threshold, additional tax brackets with higher rates, etc.).
Modeling the impact of each option to change the income tax was not possible without access to microsimulation software also used by the state tax department. The research team worked with the tax department to temporarily use laptops with standalone, direct access to the tax department network. This allowed the team to use the microsimulation software (“PolicyLinks”) to estimate how each filer’s effective tax rate would change under each option selected. The team received two full days of training from the vendor on the software.
The raw output of the microsimulation software was complex and not intuitive, requiring the team to develop standard measures of progressivity to show each option’s impact compared to the baseline. The agency’s chief methodologist researched more than a dozen different measures of tax progressivity; ultimately the team chose two different but complementary measures. The first measure showed the net change in tax progressivity across all filers, irrespective of income level. The second measure compared the effective tax rates of very high- and very low-income filers, allowing policy makers to see the relative change in tax burden between those two groups.
Rigorous methodologies and deliberate thinking about how to explain the results of the methodologies must go hand-in-hand. In hindsight, the team could not have successfully completed the project without access to all the tax filing data and the specialized microsimulation software. But identifying the relatively simple, standardized measures to communicate the results of what the software produced ended up being equally important.
Justin Brown is a senior associate director for the Virginia Joint Legislative Audit and Review Commission.
Did You Know?
As legislative evaluators, NLPES members are used to probing the effectiveness and efficiency—and sometimes even the ethics—of state government agencies and programs.
Sometimes we also find ourselves under scrutiny.
That’s what happened to the Mississippi Joint Legislative Committee on Performance Evaluation and Expenditure Review (PEER) right after its founding in 1973.
Apparently dismayed by informal references to Mississippi PEER as the “legislative audit committee,” the state Board of Public Accountancy investigated the staff’s qualifications because it believed that the word “audit” could apply only to the work of certified public accountants. The majority of Mississippi PEER’s staff members were not CPAs.
The board retreated under pressure from influential state legislators and ended its inquiry because PEER’s executive director and three of its supervisors were CPAs. Since 1977, however, the PEER executive director has not been a CPA—and the state Board of Public Accountancy has not raised any complaints.
Mississippi PEER has gone on to prosper, publishing more than 600 reports since its inception. In 2022, NLPES awarded its Outstanding Achievement Award to James Barber, who served as PEER’s executive director from 2015 to 2021 and was very active in NLPES.
Introducing Megan McClure!
I am so excited and honored to be the NCSL Liaison to NLPES!
I live in Denver with my 20-year-old daughter, Claire and partner, Sean. I’m a Denver native but have spent time in the Houston area too. I have a master’s in philosophy from the University of Colorado Denver. Being from here, surprisingly, I had never heard of NCSL before I applied for my position with the organization, but I couldn’t be happier that I did! I have been with NCSL for just about eight years and previously worked for an academic publisher and for a library.
When I started at NCSL, I was the senior staff assistant for the Legislative Staff Services Program. I’ve worked my way up and am now a policy specialist still with the program I consider my second family. With my library experience it made sense to step into the liaison role with the Legislative Research Librarians staff association (LRL) when the previous liaison retired. Then the Leadership Staff Professional Association (LSPA) came my way. LRL and LSPA have been a pleasure to work with - I’ve learned so much and met so many wonderful folks!
My favorite part of working at NCSL, besides my awesome colleagues, is that I get to work with legislative staff. A more devoted, hardworking, gracious and fun group of professionals does not exist! Knowing that my work supports some of the best folks I’ve ever met and so many more I haven’t, is what gets me out of bed on the hard days and gives me that extra umph needed to push through that last long day of Legislative Summit. NCSL’s unwavering commitment to supporting and strengthening American legislative institutions makes me proud to work where I do!
My proudest moment so far with NCSL was the conclusion of the very successful Staff Hub ATL 2022 staff seminar in Atlanta last
October. I was the lead NCSL staff member coordinating what became the largest gathering of legislative staff in NCSL history. With well over 400 staff in attendance, it was a huge task but one that was accomplished with the collaboration of countless NCSLers, the Georgia host team, legislative staff and association officers from across the nation!
I worked alongside my predecessor Brenda Erickson my entire time with NCSL. She is a fount of knowledge about NLPES and state legislatures; she made me, and so many others, better at what we do. I received a crash course on NLPES and the important work you all do while working closely with her before her retirement. A highlight in my onboarding was the opportunity to attend the peer review of the South Carolina Legislative Audit Committee with Brenda and a great team of reviewers. I learned the ropes of peer review and got to experience the friendly, thorough, and detail-oriented folks that compose NLPES offices. From the webinars, calls, and committee meetings I’ve worked on, NLPES members are dedicated to their work, their colleagues and the association.
When I’m not working and traveling with NCSL, I love to hike with my red heeler, Leto, do and teach yoga, and see the great comedians that live in and visit Denver. I enjoy traveling and am a total nerd for capitol buildings. So far, I have marked 14 off my list and should be adding another four or five this year!
Looking forward to working with and getting to know as many of you as I can!
Megan McClure is a policy specialist with NCLS and was named NLPES’s liaison in spring 2023.
NLPES Announces 2023 Award Winners
Drew Dickinson, Virginia
On behalf of the NLPES Awards Subcommittee, I would like to thank the 29 offices that applied for an award this year. It was amazing to see the impressive work being done by member offices across the country, and I know the judges for each award category found it challenging to pick the winners. Congratulations to our winners!
Also, thank you to the NLPES members who served as judges this year. We couldn’t have done this without you!
Outstanding Achievement Award
The Outstanding Achievement Award is presented to individuals who have made outstanding contributions to the fields of legislative program evaluation or performance auditing, both at the national and state levels. Leslie McGuire, who dedicated her entire professional career (nearly 29 years) to public service as an employee of the Performance Audit Division within the Georgia Department of Audits and Accounts, received the 2023 Outstanding Achievement Award. Congratulations, Leslie!
Excellence in Evaluation Award
The Excellence in Evaluation Award is presented to the legislative office determined to have made significant contributions to the fields of program evaluation or performance auditing during the most recent four-year period.
This year, Utah’s Office of the Legislative Auditor General (OLAG) won the Excellence in Evaluation Award. Congratulations to the leadership and staff of Utah’s OLAG!
Excellence in Research Methods Award
The NLPES Excellence in Research Methods Award is presented to a legislative office or offices that used exemplary research methods in an evaluation or audit report released during the most recent calendar year.
There were two winners of this year’s Excellence in Research Methods Award: the Louisiana Legislative Auditor for its report on Sufficiency Of The Transportation Trust Fund In Meeting The State’s Transportation Needs and Virginia’s Joint Legislative Audit and Review Commission for the report covering Options to Make Virginia's Individual Income Tax More Progressive as described on page 2. Congratulations to both offices!
Certificates of Impact
NLPES Certificates of Impact are presented to legislative offices that issued evaluations or audit reports during the prior three calendar years that resulted in documented public policy changes, program improvements, dollar savings or other public impacts. Twenty-six member offices were awarded a 2023 Certificate of Impact. Congratulations to each office that received a Certificate of Impact!
The full list of recipients is now available on the NLPES website.
Drew Dickinson is a chief legislative analyst at the Virginia Joint Legislative Audit and Review Commission and serves as the Chair of the Awards Subcommittee of the NLPES Executive Committee.
Recommendation Tracking: What Is Our Impact?
Jason Juffras, District of Columbia
You immersed yourself in a topic, conducted intensive field work, and published an insightful evaluation report. Is that enough? For many NLPES members, the answer is a resounding, “No!” We want our work to have an impact—to improve government policies and programs.
In our 2019 survey, NLPES members stated that their reports almost always include recommendations to agency officials and/or legislators. Many NLPES offices track the status of recommendations to hold themselves— and the agencies—accountable for improving government operations.
“That’s an important piece for us,” said Marcus Morgan, director of the Alabama Commission on the Evaluation of Services (ACES), noting that he is not satisfied with “work that gets done and put on the shelf.”
In our 2019 survey, 19 offices reported using “percentage of recommendations implemented” as a key performance indicator. Some offices (such as New Jersey) are required by law to track the status of recommendations, while others choose to do so, but each state’s approach has nuances.
The Kansas Legislative Division of Post Audit requires agency self-reporting every six months for recommendations that are between six months and two years old. Agency responses are posted online and presented to the division’s oversight committee. The division also proposes follow-up audits to the oversight committee to explore recommendation status in greater depth (see this example). For fiscal year 2022, Kansas reported a recommendation compliance rate of 75%.
Similarly, the Louisiana Legislative Auditor (LLA) asks agencies to report annually on the status of recommendations and provide relevant documentation. LLA reviews the agency response and revises the agency’s self-assessment, if necessary. LLA does not use recommendation compliance as a performance indicator.
The Texas Sunset Advisory Commission publishes reports evaluating agency compliance with actions required in sunset bills adopted by the legislature, while the Texas State Auditor reviews agency actions that did not require a change in statute. The Sunset reports (see this example) cover agencies evaluated in the previous biennium, and explain why some recommendations have not been carried out while providing timelines for implementation.
An interactive dashboard on the Utah Legislative Auditor General website shows the agency response and the legislature’s response to each audit recommendation issued in the past three years.
For example, the dashboard indicates that agencies have implemented 77 of 98 audit recommendations from 2020 and lets the viewer see each one. This year, the Utah legislature enacted 18 of 22 recommendations from a performance audit on the state’s election system.
The Utah legislature’s Audit Committee (co-chaired by the Senate President and Speaker of the House) helps ensure audits receive attention by designating a lead committee and a review committee to hear the audit and respond to the Audit Committee with recommended actions. Every audit report issued in 2022 was formally presented to a committee. “We’re trying to show audits aren’t one and done,” says Darin (Dru) Underwood, the deputy legislative auditor general. “This is not theoretical.”
Mississippi’s Joint Legislative Performance Evaluation and Expenditure Review (PEER) Committee also tracks legislative and administrative recommendations separately and helps turn recommendations into action by drafting legislation. The committee chair and vice-chair (who alternate between House and Senate) typically introduce the legislation in their respective chambers and PEER is often called on to provide briefings and testimony. Ted Booth, the executive director of Mississippi PEER, states that three bills implementing PEER recommendations were approved this year.
For recommendations to state agencies, Mississippi PEER does a six-month follow-up but may also be asked by the committee to do additional six-month checks on high-priority reports and recommendations. The PEER staff reports the status of recommendations to the committee but does not use recommendation compliance as a performance indicator.
Mississippi PEER’s Booth reminds us that implementing recommendations requires persistence. He jokingly cites a saying heard in the halls of the Mississippi Legislature: “That bill is such a good idea, we’ll see it next year.”
Jason Juffras is a senior analyst in the Office of the District of Columbia Auditor and serves on the NLPES executive committee.
Peer Exchange Program Takes the Next Steps
Marcus Morgan, Alabama
As NLPES members, we know how important professional development is to our staff and the organizations they serve. NLPES believes this too. Early this year, the executive committee decided to pilot a new professional development opportunity called the Peer Exchange Program (PEP).
This learning opportunity breaks from the traditional Professional Development Seminars and allows members from one state evaluation or audit office to spend one to three days with a similar organization from another state. The program aims to strategically tailor learning objectives to specific areas of expertise exhibited by the host site and desired by visiting participants.
Both the host site and visiting office collaborate to ensure a focused and intentional learning experience. This approach allows for a deeper exploration of the host site's expertise and addresses the learning needs and interests of the visitors.
The Utah Office of the Legislative Auditor General (OLAG) was one of a few states to express interest in the pilot program and the only office to formally apply, so the executive committee offered its support for a partnership between Utah and Washington State’s Joint Legislative Audit and Review Committee (JLARC). Later this year, two OLAG employees will visit JLARC to:
Take a closer look at the 2020 NLPES Excellence in Research Methods Award that allowed JLARC’s predictive model to streamline identification of 29,848 students who met eligibility requirements for a scholarship but did not receive one.
Explore JLARC’s performance measures and how they assess the quality or impact of recommendations implemented by agencies.
Review quantifiable cost saving and/or avoidance resulting from recommendations.
Discuss JLARC’s social media use and related policies.
Spurred by previous positive NLPES networking experiences, Utah expects the host site to benefit from sharing their audit methodology, policies, and culture through informal and formal conversations with JLARC leadership as well as the staff. This shared experience is at the heart of the program and will hopefully lead to expanding the program beyond the pilot.
Following the exchange, Utah plans to invite the staff to a manager’s meeting for a debrief where they will present lessons learned and information gained from the exchange. In addition, as a condition of the exchange, representatives from Utah will present a report to NLPES that captures their experience. They also plan to use the experience as internal professional development, extending the lessons learned beyond the one-time experience.
The NLPES executive committee would like to thank members for their interest and contributions to developing the PEP, and special thanks to Utah and Washington for participating in this inaugural program. Stay tuned for more information after the exchange!
Marcus Morgan is the Director of the Alabama Commission on the Evaluation of Services and serves on the NLPES executive committee.
Event Highlight: Legislative Summit 2023
NCSL’s Legislative Summit is the nation’s largest gathering of legislators and legislative staff, and provides a great opportunity for attendees to network, develop additional policy expertise, and stimulate new ideas. This year’s event is scheduled for Aug. 14-16 in Indianapolis. The three-day Summit will feature more than 80 sessions of policy discussion, inspiring keynote speakers, and numerous opportunities to link with colleagues and collaborate.
Legislative Summit 2023 will feature several sessions on a topic on all of our minds since 2020—the workforce paradox after the global disruption of the Covid-19 pandemic.
As such, Legislative Summit panelists will devote time to discuss the future of work, remote work, jobs of the future requiring more skills and training, and challenges in hiring. Furthermore, the event offers 14 tracks to group panels in themes, including specific policy areas, continuing legal education and professional network tracks.
Legislative Summit 2023 will also feature exhibits and the annual NCSL business meeting to set policy directives and resolutions that will guide NCSL’s advocacy efforts in the coming year.
Notable speakers range from sports figures Peyton Manning, Tamika Catchings and Al Unser Jr to inspirational individuals such as Cedric King and many more from the policy, legislative and business fields.
The event will be held at the Indiana Convention Center in the heart of downtown Indianapolis. Registration information is available on NCSL’s website and the deadline to book hotel rooms is July 17 with several options around the downtown area. A helpful FAQ is available here and contact email@example.com for more information or assistance.
Professional Development Update
Jennifer Sebren, Mississippi | Ryan Langrill, Idaho
The NLPES Executive Committee, and in particular the Professional Development subcommittee, has traditionally provided professional development resources to member offices and their staffs on a regular basis. NLPES’s goal is to provide trainings that are high-quality with minimal costs to its members. Thanks to our virtual world, the use of webinars has allowed NLPES to keep costs minimal and has resulted in strong participation across member offices. The most recent webinar recorded a whopping 117 attendees!
NLPES, with NCSL’s support, has hosted four free webinars in 2023, all of which were recorded and archived in NCSL’s Legislative Video Resource Center. NLPES kicked off its 2023 webinar series with two trainings by Curt Stedron, director of the NCLS Legislative Training Institute, and also hosted two other trainings led by staff from around the country:
- Collaborative Problem Solving (Curt Stedron, NCSL)
Description: Legislative staff can be part of situations where two sides are locked in a conflict, and they need to help resolve it. This webinar is designed to teach staff to apply the principles of negotiation to help explore creative ways of delivering solutions to conflicts in ways that can help all sides feel satisfied.
- Writing for a Legislative Audience (Curt Stedron, NCSL)
Description: Legislative staff create a wide variety of written products. From newsletters to emails to constituent correspondence—the list goes on and on. But regardless of the purpose, certain tactics are essential to crafting concise, efficient, and easy-to-digest content. In this workshop, you will learn how to effectively organize your thoughts and put them across in a form that maximizes the impact of your information.
- Recruiting and Retaining Legislative Performance Evaluation Staff (Matt Etzel (Kansas), Lori Leaming (Michigan), and Russell Moore (Tennessee)
Description: Panelists share their strategies for reaching potential performance evaluation and auditing candidates and for screening and evaluating applicants. Get ideas for keeping new and existing employees engaged and motivated to stay with your organization.
- Staffing Strategies for Strong Data Analytics—the first of a three-part webinar series on data analytics (Ana Prokos (Florida) and Tyson Cabulagan (Utah)
Description: Staff from two NLPES offices share their experiences and strategies for staff to ensure high quality data analytics, including the roles and responsibilities of methodologists/data analytics staff and their involvement with project teams.
In July, NLPES will finish its data analytics webinar series and host a session with staff from Montana on getting the most out of your interviews. If you haven’t registered yet, click here.
Additionally, NLPES began hosting information sharing calls several years ago for subgroups of its members. The intent of these calls is to facilitate the sharing of knowledge and information among those with similar roles in their respective offices. NLPES subcommittee members hosted information sharing calls in May for the following subgroups:
Calls are scheduled to be held again in November.
NLPES will continue to provide professional development opportunities using the most efficient and effective means possible. Thank you all for your continued support and participation!
Jennifer Sebren is the deputy director for Mississippi’s Joint Legislative Committee on Performance Evaluation and Expenditure Review and serves on the NLPES executive committee.
Ryan Langrill is a principal evaluator for Idaho’s Office of Performance Evaluations and serves on the NLPES executive committee.
Martha Carter, who has served as the legislative auditor for the Nebraska Legislative Audit Office, retired in June.
Christopher Latta is now the executive director at the Pennsylvania Legislative Budget & Finance Committee. He replaces Patricia Berger, who recently retired.
Grant Parks is now the California state auditor after being appointed by Gov. Gavin Newsom to the position at the end of 2022. He started his career as an entry-level auditor in the State Auditor’s office in 1999.
Jennifer Jones, who had served as executive director for the Texas Sunset Advisory Commission, left the agency after nearly 30 years in various roles. She is now a deputy director at the Texas General Land Office.
The Working Paper is published twice a year by the National Legislative Program Evaluation Society, a professional staff association of the National Conference of State Legislatures. NLPES serves the professionals of state legislative agencies engaged in government program evaluation. The purposes of NLPES are to promote the art and science of legislative program evaluation; to enhance professionalism and training in legislative program evaluation; and to promote the exchange of ideas and information about legislative program evaluation.
The Working Paper is produced by the NLPES Communications Subcommittee:
- Jason Juffras (District of Columbia), 2022-23 chair
- Darin Underwood (Utah), member
- Darren McDivitt (Texas), member
Please contact Darren McDivitt with any article suggestions, comments, or questions.