The Working Paper is the official newsletter of the National Legislative Program Evaluation Society. NLPES serves the professionals of state legislative agencies engaged in government program evaluation. The purposes of NLPES are to promote the art and science of legislative program evaluation; to enhance professionalism and training in legislative program evaluation; and to promote the exchange of ideas and information about legislative program evaluation.
Inside the Issue
Research & Office Round Up:
Professional Updates:
News Snippets:
Chair’s Corner
Jennifer Sebren, Mississippi
NLPES Friends,
Much of our work as evaluators revolves around analyzing critical issues through the lens of condition, criteria, cause, and effect—the elements of a finding. I thought it might be fun, and perhaps insightful, to apply this framework to assess the current status of NLPES.
The condition, or “What is?,” might reflect the incredible work our members produce. In 2024 alone, NLPES member offices released hundreds of reports, packed with thoughtful analyses and actionable recommendations—both legislative and administrative.
The cause, or “Why?,” might seem simple—after all, this is our job. But I believe there’s more to it. At our core, we are united by a shared purpose: to make government work better. It’s this commitment that drives us to excel in our roles.
The criteria, or “Says who?,” is rooted in the laws and directives that guide our work. Though the language and mandates may differ across states, the underlying mission is the same: to improve state government through evaluation of agency programs and pressing issues. This shared foundation reminds us of the importance of what we do and the trust placed in us to deliver meaningful results.
Finally, the effect, or “So what?”—my favorite part of any finding. It’s the question that pushes us to dig deeper and focus on the ultimate outcome. And in asking “So what?,” we arrive at a powerful conclusion: our work has impact. Beyond the reports and the numbers, our impact lies in the stories of improvement we help create—inefficiencies eliminated; better policies adopted; more effective programs implemented; better services delivered; lives ultimately made better. Yes, you are a part of that!
As critical as our work is, we sometimes struggle to fully convey its value. This year, at my direction, the Executive Committee will conduct a revised and in-depth Who We Are and What We Do survey of NLPES member offices. The survey has traditionally resulted in a summary of how our offices are organized, the types of work that we do, and our methods of communication. This year, we plan to focus in more on that word impact. In doing so, we will aim to provide a readily available resource that clearly communicates the value of our work to a wide variety of stakeholders.
Our collective impact often extends beyond what we can see. Which brings me to this point—you are not alone as you do your work. I encourage you to take advantage of the resources available through NLPES and other member offices who are eager to share their knowledge and experiences. To help coordinate those efforts, the NLPES Executive Committee is hard at work to provide all members with opportunities for training, for sharing ideas and information, and for being recognized for great work. In addition, the Committee will be working with the Legislative Audit Division in Montana as they plan the NLPES fall training conference to be held on Sept. 30-Oct.2 in Helena, Mont. I hope to see you there!
Jennifer Sebren is the Deputy Director at the Legislative PEER Committee in Mississippi.
Meet NCSL Staff Vice-Chair Lonnie Edgar
Shunti Taylor, Georgia
Our very own Lonnie “Oh, hey ya’ll” Edgar was elected staff vice chair of the NCSL Executive Committee during the 2024 Legislative Summit in Louisville, Ky. Following this one-year term as staff vice chair, he is expected to assume the staff chair position during the 2025 Legislative Summit in Boston, the location of NCSL’s 50th Anniversary celebration.
As staff vice chair of the Executive Committee (NCSL’s governing body), Lonnie will work alongside staff chair John Snyder of Kentucky to represent the interests of legislative staff. “It is a privilege to serve as an ambassador to NCSL and represent the interests and needs of roughly 30,0000 legislative staff nationwide,” said Edgar. One of his goals is to increase legislative staff engagement with NCSL, and he already has several great ideas to promote the work of NCSL and reinforce how it can support legislative staff in the work we do.
Crème de la Crème
He is now among an elite group of program evaluation and audit staff who have served in a leadership role with NCSL. Since its inception in 1975, only four staff from NLPES member offices have served as staff chair: Dale Cattanach (Wisconsin), John Turcotte (Mississippi), Max Arinder (Mississippi), and Gary VanLandingham (Florida). With his election to the staff vice chair role, he is poised to become the fifth NLPES member (and the third staffer from Mississippi PEER!) to serve as staff chair.
A Natural Leader
He began his legislative career as an evaluator with Mississippi’s PEER Committee in 2005. Over the years, he became increasingly responsible for managing teams and, as deputy director, carrying out various administrative and operational duties. “I attribute much of my success and career advancement within my own organization to my experience with NCSL,” he said.
His involvement with and commitment to NCSL and its mission over the years has earned him the respect and admiration of both NCSL staff and legislative staff across the country. Since 2009, he has served as a member of NCSL’s Health Reform Task Force and the Health and Human Services Standing Committee (where he later served as committee staff vice-chair and co-chair). He also served as a fellow in the inaugural Maternal and Child Health Policy Fellows Program. And many of you have likely seen him in action as panel moderator or presenter at numerous NCSL meetings.
A People Person
If you’ve met him, it goes without saying that he has an innate ability to connect with people, making everyone feel valued and heard. “One of my greatest strengths is developing relationships, even when those involved may not approach things from similar perspectives, by fostering meaningful connections through my positive and inclusive energy,” he said. Not only does he excel at inspiring people professionally, his positivity shines through in his work with the Fleet Feet marathon training program and Orangetheory Fitness. Through these programs, he has motivated numerous people to run their first marathon and has led over 500 members through numerous workouts on a weekly basis.
I believe his leadership and enthusiasm for NCSL will inspire legislative staff to be more engaged and actively involved. And with him at the helm, I am confident that NCSL will continue to thrive and position itself for another 50 years of excellence.
Congratulations Lonnie!
Shunti Taylor is the Deputy Director at the Georgia Department of Audits and Accounts.
Spicing Up Our Skills in the Big Easy
Jason Juffras, District of Columbia
As Louisiana Rep. Jason Hughes finished welcoming us to New Orleans for our 2024 professional development seminar (PDS), he exhorted us to “Laissez les bons temps rouler.”
Indeed, we had a good time in the Crescent City, exploring the French Quarter and beyond, enjoying po’boys and seafood gumbo, and sipping Sazerac at the Carousel Bar. But more importantly, the more than 150 attendees delved deeply into evaluation topics, techniques, findings, and recommendations.
The following is a summary of high points of the PDS from this writer’s perspective.
Members tackled tough topics and dug deeply into field work. Members from Louisiana, Utah, and Wisconsin discussed how they navigated a turbulent political environment while evaluating election administration. Evaluators from these offices fanned out through counties and parishes to observe election operations, tested the accuracy of the voter rolls, and examined absentee ballot certificates. These evaluations uncovered some problems, but legislators and election officials acted on many of the proposed solutions, thus enhancing public trust in our democratic systems. Speakers on other panels, from “Evaluating Programs for Vulnerable Populations” to “Capital/Infrastructure Projects,” stressed the importance of site visits and direct observation.
Our work can challenge conventional wisdom and provide a more nuanced understanding of key issues. Alabama evaluators assigned to assess a statewide teacher shortage found there was no such shortage, although rural and impoverished areas faced difficulties in recruiting and retaining teachers. The Alabama team also found that most teachers who leave the profession do so within the first three years.
We can have an impact – even when nothing happens. Former Washington State Legislative Auditor Keenan Konopaski (pictured above), who won our 2024 Outstanding Achievement Award, urged us to transcend partisan divisions by focusing on informing the legislature, remaining fiercely nonpartisan and daring to be boring (but precise and direct). Recalling how an analysis by his office led legislators to drop a plan to shift school support staff to health insurance offered under the Affordable Care Act, Keenan noted that this was a major impact even though nothing changed.
Don’t try to audit by e-mail. Speakers from Wisconsin, Montana, and Colorado on a “Conducting Effective Interviews” panel emphasized the need to develop and capitalize on personal connections. In-person interviews can be better than virtual interviews because you can see non-verbal cues and get a sense of the office environment.
Big data has big potential. NLPES members shared powerful examples of creative data analytics. Kansas auditors used machine learning to assess the extent of fraud in unemployment insurance claims, saving millions of dollars. Louisiana auditors identified thousands of individuals who were not eligible for public assistance by matching claims data to lists of deceased and incarcerated individuals, while also identifying nearly 400 behavioral health providers who were paid $682 million by Medicaid but did not report any wages to the Louisiana Workforce Commission.
Simple findings can be powerful. Louisiana auditors who examined the state’s workforce development system highlighted program deficiencies by pointing out, inter alia, that the state lacked a common workforce development portal and that 31 of 60 website addresses on a one-stop center online lookup tool were incorrect.
Special thanks are due our colleagues in the Louisiana Legislative Auditor’s office, led by Assistant Legislative Auditor Karen LeBlanc. Their hospitality and hard work as conference hosts made things very easy for us in the Big Easy.
Jason Juffras is a senior analyst for the Office of the District of Columbia Auditor and a member of the NLPES Executive Committee.
Electronic NLPES Awards Submission Pilot
Megan McClure, NCSL
NLPES will be piloting an electronic submission method for the 2025 Excellence in Evaluation Awards. Please follow the specific instructions for the pilot listed on the Excellence in Evaluation Award information and criteria page.
All other awards will follow the process that has been used in the past: PDF forms and emailed attachments. You can visit the information and criteria pages for the Outstanding Achievement, Excellence in Research Methods and Certificates of Impact awards for specific instructions for each award.
If you run into any difficulties or have questions, please reach out to NLPES liaison, Megan McClure (303) 856-1355 or at [email protected]
Learn More About Electronic NLPES Awards Submission
(Relatively) New Kid on the Block: Virginia’s Behavioral Health Commission
Lauren Bailey, Idaho
You may know of Virginia’s Joint Legislative Audit & Review Commission (JLARC), but did you know Virginia has a newer legislative commission tasked with evaluation and monitoring activities? Virginia’s General Assembly created the Behavioral Health Commission in 2021 to make recommendations for the improvement of behavioral health services and the behavioral health service system in the commonwealth.
In a recent conversation with Nathalie Molliet-Ribet, executive director of the Behavioral Health Commission, I learned more about the unique role this commission plays in Virginia’s government.
The Behavioral Health Commission had early roots in a joint subcommittee of the Assembly which met a few times each year from 2014 to 2021. The subcommittee lacked dedicated staff and the resources necessary for independent research. Recognizing the need for more sustained and robust attention in this policy area, the Assembly formalized the commission in 2021 and appointed an executive director in January 2022. The commission is composed of 12 state legislators, seven from the House of Delegates and five from the Senate.
Director Molliet-Ribet joined the commission with an impressive background of experience in Virginia government. She spent 13 years at JLARC earlier in her career and was the senior associate director upon her exit. She worked as the deputy secretary of education for Virginia and the vice president of research for the Virginia Economic Development Partnership.
The director said she looked at other Virginia commissions and JLARC’s model as guides for how to structure the commission’s work. Because the commission is topic specific, they seek to develop staff expertise in behavioral health. “It’s like trying to find a unicorn,” the director described the hiring process. “You want someone who has policy research experience, an understanding of Virginia state government and ideally an understanding of behavioral health issues.” The commission currently has a staff of five people with a mix of public policy and behavioral health backgrounds. Staff recently attended the NLPES 2024 Professional Development Seminar in New Orleans.
The Behavioral Health Commission’s work includes policy studies, often aimed at trying to solve specific problems. The commission also has statutory responsibilities to monitor and evaluate behavioral health programs, services, delivery and payment structures, and the implementation of new services and initiatives. A few examples of current work include studies of crisis services and civil commitment, behavioral health services in Virginia jails, and current efforts to improve the behavioral health system.
The commission measures its performance by assessing (1) the number of recommendations generated by commission staff, (2) how many of those recommendations were adopted by the commission, and (3) how many of those adopted recommendations were implemented by legislative or budget action. In 2024, 20 of the 21 staff-developed recommendations were adopted by the commission and 12 of those were implemented by legislative or budget action. Since tracking began in 2023, 63 percent of staff-developed recommendations have been implemented. More information about the Behavioral Health Commission can be found at https://bhc.virginia.gov.
Lauren Bailey is a Senior Evaluator with the Idaho Legislature’s Office of Performance Evaluations.
Report Spotlight: Idaho’s Enterprise Resource Planning System (Luma)
Ryan Langrill, Idaho
Tolstoy wrote in Anna Karenina “all happy families are alike; each unhappy family is unhappy in its own way.” Software implementation is the exact opposite: each unhappy implementation is alike, but each happy one is unique, because successful software implementation caters to the distinct needs and capabilities of an organization. We’ve learned this lesson through reports we’ve done on Idaho’s K-12 Longitudinal Data System, Instructional Management System, Medicaid Claims Processing, and the next K-12 Longitudinal Data System (they didn’t listen the first time). A 2019 report by the Defense Innovation Board, Software is Never Done, includes a good overview of a forty-year history of reports and recommendations about software being shouted into the void at the federal level.
So, we had a pretty good idea what to expect when we were asked to evaluate Luma, Idaho’s new enterprise resource planning system. Idaho is one of the first states to go live with a cloud-based, software as a service ERP that covers accounting, payroll, reporting, human resources, procurement and budgeting. While I won’t opine on the wisdom of adopting a software that does everything rather than multiple, specialized systems, I do hope to share some lessons for your state if you follow in our footsteps.
First lesson: let the project be as flexible as possible. Our Legislature and State Controller’s Office made the right choice to treat this as an infrastructure project with minimal restrictions on when or how the money can be spent. However, the controller’soffice conducted separate procurements for software and implementation, selecting the software vendor first. This meant only implementation vendors experienced with their chosen software could bid on the second procurement. If you're going this route, shortlist multiple viable software options before seeking implementation proposals. You want the best joint software-implementor team, not just whoever can implement your pre-selected software.
Second lesson: organizational change management isn't something you do after selecting software. It needs to drive the procurement. Without understanding how work will change and what users need, you can't effectively evaluate vendor proposals or develop training. In our survey, 76% of respondents reported Luma was less efficient than legacy systems—a predictable result of inadequate change management planning.
Third lesson: contract management starts before contract signing. The controller's office accepted generic, template-based deliverables from their implementation contractor that failed to address Idaho's specific needs. Six of the eight project management deliverables we reviewed were boilerplate documents that repeated information from the contract's scope of work—and Idaho paid $1.65 million for them. The contract lacked specific criteria for rejecting inadequate deliverables, leaving the state unable to hold the contractor accountable. If you're spending tens of millions of dollars on implementation, invest the time up front to define what success looks like in detail. Request examples of key deliverables during procurement, and ensure your contract has teeth when contractors try to pass off generic templates as custom solutions.
Fourth lesson: create training strategies that reflect how adults actually learn in the workplace. We found the same generic training was offered to everyone from occasional timesheet users to fiscal officers running complex federal grants. The training materials weren't updated when processes changed, and users struggled to find help documentation because the search terms didn't match their understanding of tasks. If you're implementing an ERP, recognize that someone submitting a timesheet every two weeks needs different support than someone processing payroll. Consider how frequently users will perform tasks - tasks done daily might need less ongoing support than annual processes that users need to relearn each time. Most importantly, hire training specialists who understand adult learning, not just people who know the software. Because when your employees can't do their jobs, it doesn't matter how powerful your software is.
Perhaps the most important lesson is that while technology has evolved dramatically, the fundamental challenges haven't changed. Success depends more on people than software. As evaluators, we'll likely keep writing these reports until agencies internalize that implementing new technology requires at least as much investment in organizational change management as in the technology itself.
But maybe your state will be different. Let us know how it goes.
For additional resources, check out 18F within the General Services Administration, particularly their de-risking guide and Jennifer Pahlka’s book “Recoding America: Why Government Is Failing in the Digital Age and How We Can Do Better.”
Ryan Langrill is the director of the Idaho Legislature’s Office of Performance Evaluations.
The Synergistic Paradigm of Jargon Deconstruction
Victoria Hall, Maryland and Claude Anthropic, AI
Greetings, fellow stakeholders in programmatic excellence! Today, we invite you to engage with us in a deconstruction of our communicative value propositions. For all intents and purposes, we want to touch base here with the quintessential challenge of our professional ecosystem. Let's conduct a performance audit of our linguistic practices and triangulate how we engage our audience to holistically unpack the transformative potential of jargon elimination.
Imagine, if you will, a logic model that illustrates the theory of change surrounding our communication strategies. Our essential goal is to harvest outcomes from our linguistic landscape, identifying the externalities of our verbal ecosystem. We aim to systematically unpack how jargon creates opportunity costs in organizational effectiveness.
Consider the drafty Overton window of our mainstream communicative practices. Are we really promoting an informed audience and synergistically aligned teams, or crafting an impenetrable wall of verbal complexity? When we use technical terminology, we should ensure we add value, and avoid veering into a buzzsaw of cliché’s that creates a regulatory capture of comprehension.
Jargon has a price: it creates an unfunded mandate of communication by imposing a cognitive load on our audience without the requisite funding of clarity. Where complexity is rewarded over genuine understanding, we create a moral hazard. Our current approach all too often represents a classic case of incrementalism gone awry – small, gradual steps toward complete communication obfuscation.
Root causes for using jargon include a belief, perhaps even subconsciously held, that complex language demonstrates our expertise, or that technical terms imply precision. But when we use phrases like "integrating quality solutions" or "strategically engaging with departments," we're essentially saying not much. It's the linguistic equivalent of a performance measurement matrix with no actual measurements.
Explaining a sunset provision to someone drowning in a sea of technical terminology is just cruel. No, my friends, instead fling that argot overboard and offer a lifeline instead. A kinder approach would be to aim for Pareto efficiency in our communication whereby we cannot improve our message's clarity without potentially diminishing someone's perceived intellectual sophistication. Our proposed counterfactual scenario is simple: What would happen if we communicated clearly?
In conclusion, fellow evaluators, let’s measure our ultimate return on investment not in complex terminologies, but in genuine understanding. Let's speak like humans, or perhaps like humans judiciously but not slavishly assisted by AI. Let's commit to clarity.
Synergistically yours,
Victoria (Tori) Hall, MPA, and Claude Anthropic, AI
Victoria (Tori) Hall is a principal evaluator with Maryland’s Office of Program Evaluation and Government Accountability.
Staff Happenings
Retirement of long-time director of the Idaho Office of Performance Evaluations Rakesh Mohan and the exciting news that NLPES executive committee member, Ryan Langrill was selected as the new director!
Arkansas: Roger Norman retired after 47 years of service with Arkansas Legislative Audit. Kevin White was selected as the new Legislative Auditor.
Maryland Office of Legislative Audit: Previous Legislative Auditor Greg Hook retired after 43 years of service at the OLA, Brian Tanen selected as new Legislative Auditor.
Finally, a warm welcome to Emily Dixon (Louisiana) as a newly elected NLPES Executive Committee member, and a big thanks from the Executive Committee to Darin (DRU) Underwood (Utah) for serving as the 2023-2024 chair.
The Working Paper is published two times a year by the National Legislative Program Evaluation Society, a professional staff association of the National Conference of State Legislatures. NLPES serves the professionals of state legislative agencies engaged in government program evaluation. The purposes of NLPES are to promote the art and science of legislative program evaluation; to enhance professionalism and training in legislative program evaluation; and to promote the exchange of ideas and information about legislative program evaluation.
The Working Paper is produced by the NLPES Communications Subcommittee:
- Ryan Langrill (Idaho), chair
- Darin Underwood (Utah), member
- Emily Dixon (Louisiana), member
Please contact Ryan Langrill with any article suggestions, comments, or questions.