|
Monday Oct. 30 – Registration at the Drury Plaza Hotel
|
|
2-5 p.m.
Palace Ballroom Foyer
|
Registration
|
|
1-4 p.m.
Meem Room
|
NLPES Executive Committee Meeting
Presiding: Darin R. “DRU” Underwood, NLPES chair, Utah
|
|
Tuesday Oct. 31 – All Sessions at the Drury Plaza Hotel
|
|
8 a.m.-4 p.m.
Palace Ballroom Foyer
|
Registration/ Information Desk
|
|
8:30-9:20 a.m.
Palace Ballroom
|
Welcome and Kickoff: Fifty Years of NLPES
Hear from 2023 NLPES chair, DRU Underwood, a welcome from our host state and then regale the past 50 years of NLPES history, purpose and camaraderie as we look forward to the future of the association with unofficial NLPES historian, John Turcotte.
Speakers: Darin R. “DRU” Underwood, NLPES chair, Utah Office of the Legislative Auditor General
Jon Courtney, deputy director for Program Evaluators, New Mexico Legislative Finance Committee
John Turcotte, public administration consultant, Mississippi
|
Plenary
|
|
Concurrent Sessions
|
|
9:30-10:30 a.m.
O’Keefe Room
|
Evaluating School Districts
State legislatures allocate billions of dollars to school districts and charter schools each year to provide public education. States’ significant investments into public education necessitate continual oversight and evaluation. In this session, panelists will discuss the unique findings, challenges and aspects of auditing and evaluating public schools.
Moderator: Clayton Lobaugh, program evaluator, New Mexico Legislative Finance Committee
Speakers: Christopher McClelland, performance auditor, Utah
Scott Swaggerty, director, division of school audits, Arizona Auditor General
Allison Pams, research analyst, Tennessee Comptroller of the Treasury
|
Track
What We Evaluate
|
|
9:30-10:30 a.m.
Palace Ballroom
|
How to Pick a Good Evaluation Topic
Legislative performance audit and program evaluation functions frequently work within a statutory structure than mandates the authority and direction of their examinations; however, these same functions must ensure that any audits are meaningfully developed to ensure quality outcomes. This session will identify and discuss a few different ways in which these functions incorporate risk-based analysis to assess and define what is good evaluation while still being responsive to legislative direction and need.
Moderator: Will Soller, deputy legislative auditor, Montana Legislative Audit Division Speakers: John Harrington, management & program analyst manager, Montana Legislative Audit Division
Gina Brown, audit manager, Louisiana Legislative Auditor
Derek Johnson, Legislative Performance Audit Manager, Colorado Office of the State Auditor
|
Track
How We Evaluate It
|
|
9:30-10:30 a.m.
Lamy Room
|
The Results First Initiative in supporting evidence-based policy decisions: Past successes and future direction
In recent years many state agencies have increased the use of evidence in program funding decisions with the goal of improving outcomes across various sectors of public health. For more than a decade, the Results First Initiative has provided tools for states to help with this including matching collective evidence to existing programs, establishing processes to increase impact, and determining projected return-on-investment from programs being implemented. In this session we will review how Results First has worked with states in the past, how the Results First cost-benefit model works, what challenges exist in this process, and what future opportunities are for using these tools to advance the use of evidence in local funding decisions.
Speakers: Damon Evan Jones, associate research professor, Penn State University Mike Wilson, economist and research consultant
|
Track
Learning from Others
|
|
Concurrent Sessions
|
|
10:45-11:45 a.m.
Palace Ballroom
|
Evaluating Crime and Public Safety
Public safety can be an impactful and challenging area for a program evaluation. This panel will explore examining different parts of crime and public safety from juvenile justice to justice reinvestment, to reentry. We will take participants through the stages of a typical evaluation in these realms while discussing challenges and lessons learned.
Moderator: Jon Courtney, deputy director for Program Evaluators, New Mexico Legislative Finance Committee
Speaker: Ross Ryan, principal performance evaluator, Wisconsin Legislative Audit Bureau Darin R. “DRU” Underwood, deputy auditor general, Utah Office of the Legislative Auditor General
Brittany Utz, senior legislative analyst, Virginia Joint Legislative Audit and Review Commission
|
Track
What We Evaluate
|
|
10:45-11:45 a.m.
O’Keefe Room
|
Managing an Evaluation Team
Managing any team isn’t necessarily easy. There’s the usual job of managing personalities, work styles and deadlines. Then throw in factors like remote or hybrid work, staff turnover, and working with state legislatures, not to mention the highly detailed and specialized work that our teams are doing, and it can get even hairier! Hear from a panel of experienced project managers and team leads on how two agencies of different sizes and organization maturity handle team management, and discuss strategies for keeping your evaluation team engaged, organized and on task.
Moderator: Micaela Fischer, New Mexico Legislative Finance Committee Speakers: Regina Birchum, deputy director, Oklahoma Legislative Office of Fiscal Transparency
Lauren Ames, policy analyst, Texas Sunset Advisory Commission
|
Track
How We Evaluate It
|
|
10:45-11:45 a.m.
Lamy Room
|
NLPES Evaluation Process: How it Works, What We’ve Learned, Why You Should Participate Did you know that NLPES can provide peer reviews for member offices? Whether Yellow or Red Book, or custom review, NLPES can put together a team of program evaluators and performance auditors who really understand what you do and offer constructive feedback and concerted networking amongst members. Hear from the liaison and two members who have experienced both sides of a NLPES peer review.
Moderator: Megan McClure, policy specialist, NCSL
Speakers: Emily Dixon, performance audit manager, Louisiana Legislative Auditor Adam Friedly, audit manager, West Virginia Post Audit Division
|
Track
Learning from Others
|
|
12:00-1:15 p.m.
Palace Ballroom
|
Lunch Plenary
A Voice Crying in the Wilderness: The Past, Present, and Future of Legislative Evaluation Legislative evaluation offices in most states were established in the 1960s - 1980s to 'Speak Truth to Power' by providing independent and professional evaluations and audits to their parent legislatures. The environment in which the offices operate has changed greatly since those days of yore, creating both new opportunities and significant challenges in many states. This session will discuss the implications of these trends for legislative evaluation units and NLPES.
Speaker: Gary Vanlandingham, director, Askew School of Public Administration and Policy, Florida State University
|
Plenary
|
|
Concurrent Sessions
|
|
1:30-2:45 p.m.
Palace Ballroom
|
Artificial Intelligence in Practices and Policy
Hear from two speakers on the frontier of the ethics and application of AI. First with an introduction to AI and tools like ChatGPT, a discussion of the ethics of using AI along with recommendations and strategies for training, proper use and ways to reduce bias and increase trust in AI tools.
Then we’ll discuss and take into consideration the real-world application of AI in policy settings: Several states now require algorithms used by the public sector to be transparent and independently audited for accuracy and fairness. At their best and most transparent, algorithms can help human decision-makers be more objective. But when they are opaque, proprietary, and hidden behind trade secrets — as algorithms from private vendors often are — they should be viewed with skepticism and caution. We’ll explore what’s at stake using examples from criminal justice, housing, and credit.
Speakers: Cris Moore, professor, Santa Fe Institute
Christian Moriarty, professor of law and ethics, St. Petersburg College
|
Track
Learning from Others
|
|
1:30-2:45 p.m.
O’Keefe Room
|
Progress Reports and Other Evaluation Follow-up
Evaluation follow-ups are intended to reflect any progress made to address a report’s findings since the report was released. Three offices share their processes and rationale for formal evaluation follow-ups and other efforts to measure and monitor progress resulting from evaluations.
Moderator: Meri Clare Ringer, analyst, Mississippi PEER Committee
Speakers: Emily Dixon, performance audit manager, Louisiana Legislative Auditor Andy McConnel, Policy Analyst, Texas Sunset Advisory Commission
Jennifer Sebren, deputy director, Mississippi PEER Committee
|
Track
How We Evaluate It
|
|
1:30-2:45 p.m.
Lamy Room
|
Remaining Relevant and Valued as an Office
Our legislatures rely on us to provide objective information as they wrestle with challenging public policy decisions. The legislative public policy arena is becoming more and more fast- paced and polarizing. So more than ever, our respective audit and evaluation offices must provide impactful and timely information, otherwise we risk being devalued.
Moderator: Darin Underwood, deputy auditor general, Utah Office of the Legislative Auditor General
Speakers: Andy Brienzo, principal auditor, Kansas Legislative Division of Post Audit Ryan Langrill, principal evaluator, Idaho Office of Performance Evaluations
Kade Minchey, auditor general, Utah Office of the Legislative Auditor General
|
Track
Learning from Others
|
|
Concurrent Sessions
|
|
3-4:15 p.m.
O’Keefe Room
|
Integrating Datasets for Longitudinal Analysis
At their best, longitudinal datasets can allow evaluators to reveal downstream outcomes from prior programmatic inputs by observing the same individuals, counties, governmental entities, etc., at multiple points in time. However, creating longitudinal data can be difficult, particularly when it requires integrating multiple datasets. This panel will review and give examples of techniques for building longitudinal datasets and the data analysis techniques that become possible when longitudinal data are available.
Speakers: Joshua Karas, quantitative research analyst, Washington JLARC
Edward P. Seyler, Ph.D., director of Economic Advisory Services, Louisiana Legislative Auditor Ryan Tolman, program evaluator, New Mexico Legislative Finance Committee
|
Track
What We Evaluate
|
|
3-4:15 p.m.
Lamy Room
|
How to Drive Post Evaluation Impacts and Working with Budget Staff
Post evaluation, what strategies are there for ensuring recommendations get incorporated into the decision-making process? In this session, we will explore both formal or informal mechanisms that can be put in place to assist decision makers and help guide budgeting decisions.
Moderator: Marcus Morgan, director, Alabama Commission on the Evaluation of Services
Speakers: Mike Jackson, executive director, Oklahoma Legislative Office of Fiscal Transparency
Jodi Munson Rodríguez, deputy legislative auditor, Minnesota Office of the Legislative Auditor
|
Track
Learning from Others
|
|
4:30-6:30 p.m.
Palace Ballroom
|
Impact Award Information Fair and Spoooooky Monster Mash
Don some graveyard garb, grab a frightfully fizzy concoction and magical morsel! Chat with old friends and new phantoms. Then peruse shadowy stations to hear about this year’s NLPES Impact Award winners (see award list in the app for winners).
|
Plenary
|
|
Wednesday Nov. 1 – Drury Plaza Hotel and New Mexico State Capitol
|
|
8 a.m.-4 p.m.
Palace Ballroom Foyer
|
Registration
|
|
8:30-9:20 a.m.
Palace Ballroom
|
NLPES Awards Breakfast (coffee and pastries)
Grab a pastry and cuppa and help us recognize and celebrate the winners of this year’s Outstanding Achievement Award, Excellence in Evaluation Award and Excellence in Research Methods Awards (see awards list on the app for winners).
Presiding: Darin Underwood, deputy auditor general, Utah Office of the Legislative Auditor General
|
Plenary
|
|
Concurrent Sessions
|
|
9:30-10:30 a.m.
Palace Ballroom
|
Evaluating Medicaid
Medicaid is a broad program offering health care to people with low incomes and essential long-term supports to seniors and people with disabilities. It is also one of the largest, if not the largest program in state budgets. This panel will discuss the wide range of evaluations involving Medicaid and give evaluators some lessons for approaching such a broad and complex program.
Moderator: Rachel Mercer Garcia, principal analyst, New Mexico Legislative Finance Committee
Speakers: Ryan Langrill, principal evaluator, Idaho Office of Performance Evaluations Kade Minchey, auditor general, Utah Legislative Auditor General
Kate Shiroff, legislative audit manager, Colorado Office of the State Auditor
|
Track
What We Evaluate
|
|
9:30-10:30 a.m.
O’Keefe Room
|
Holding Ourselves Accountable
NLPES member offices exist to hold government accountable for handling taxpayer funds and executing legislative programs in a responsible and transparent manner. But who holds us accountable? Join this panel as we discuss methods for showing our work and continued follow-up to the public and how offices are ensuring they are adopting and following the best practices they are recommending to their auditees.
Moderator: Mike Jackson, executive director, Oklahoma Legislative Office of Fiscal Transparency
Speakers: Jason Juffras, senior analyst, Office of the District of Columbia Auditor
Darin Underwood, deputy auditor general, Utah Office of the Legislative Auditor General
|
Track
How We Evaluate It
|
|
9:30-10:30 a.m.
Lamy Room
|
Analyzing the Impacts of Stacked Programs
There are numerous income support programs that families can enroll in which provide support and assistance for individuals who make generally under 200% of the federal poverty level. Examining the comprehensive benefits that combined programs provide can be useful when making state policy decisions. In this session we will discuss the take-home value of income support programs for families, program uptake, cliff effects and how these programs and cliffs may impact employment. Along with, the effects of the increase of benefits due to COVID-19 and the subsequent unwinding is reviewed.
Speakers: Sarah Dinces, program evaluator, New Mexico Legislative Finance Committee Allegra Hernandez, program evaluator, New Mexico Legislative Finance Committee
|
Track
What We Evaluate
|
|
Concurrent Sessions
|
|
10:45-11:45 a.m.
Palace Ballroom
|
Evaluations of Housing and Homelessness Supports
State and local governments are currently facing challenges pertaining to the availability of affordable housing and the management of funding and programs to address homelessness. Learn how some states approached evaluating these issues, including how they determined which entities to examine and their process for identifying data sources.
Moderator: Jeanine Brown, senior management analyst, Georgia Department of Audits and Accounts
Speakers: Brian Dean, senior audit supervisor, Utah Office of the Legislative Auditor General Kathleen Gygi, program evaluator, New Mexico Legislative Finance Committee
Stefanie Papps, principal legislative analyst, Virginia Joint Legislative Audit and Review Commission
|
Track
What We Evaluate
|
|
10:45-11:30 a.m.
O’Keefe Room
|
NLPES peer exchange pilot program – a report from the first year
Have you heard about the pilot exchange program NLPES launched this year? Were you interested but have some questions? Want to hear more information about what, how and why the program exists? Didn’t hear about it but the idea of a structured visit to another program evaluation office sounds like an ideal learning opportunity for you or your staff? Come to this informal, part info, part listening session to learn more and let us know how we can make it work for you.
Moderator: Marcus Morgan, director, Alabama Commission on the Evaluation of Services
Speakers: Megan McClure, policy specialist, NCSL
Darin Underwood, deputy auditor general, Utah Office of the Legislative Auditor General
|
Track
Learning from Others
|
|
10:45-11:45 a.m.
Lamy Room
|
Equity Analysis in Evaluations
While equity analyses have been incorporated into program evaluations and performance audits in the past, a new trend is arising of offices being required to perform these often- difficult analyses on a wider range of projects. Issues such as limited or nonexistent data on racial impact arise often and knowing when and how to perform this work can be a conundrum. Hear from two speakers who are currently grappling with the new frontier of required or suggested racial equity impact analysis in their work.
Speakers: Susanna Pratt, research analyst, Washington Joint Legislative Audit and Review Committee
Brittany Utz, senior legislative analyst, Virginia Joint Legislative Audit and Review Commission
|
Track
How We Evaluate It
|
|
Noon-1 p.m.
Palace Ballroom
|
Lunch Plenary
Pre-K: Critical Issues for Planning and Evaluation
In this session participants will learn about and discuss three critical issues for agency staff supporting planning and evaluation of state preschool programs. (1) Short and long-term changes in birth rates affect preschool programs directly and indirectly. (2) Labor market changes have resulted in staff shortages that may persist without policy responses and some policy responses are better for quality and outcomes than others. (3) Evaluators face difficult challenges in producing valid estimates of programs' causal impacts, especially the long-term impacts that generate economic benefits. How can we increase confidence in impact estimates and guide continuous improvement? New methods and measures may help.
Speaker: Steve Barnett, professor/senior co-director, National Institute for Early Education Research, Rutgers University
|
Plenary
|
|
1-1:30 p.m.
|
Walk from Hotel to the New Mexico State Capitol “Roundhouse”
OFF SITE: New Mexico State Capitol:
411 S Capitol St, Santa Fe, NM 87501
Google map of walking route
Ubers will be provided for folks who need them.
|
|
1:30-2:30 p.m.
Room 322
|
Innovative Research Methods – Winners of Recent Methodology Awards
Hear from team members who worked on this year’s NLPES Excellence in Research Methodology Awards. They will talk about the teams who worked on them, what went into the reports and methodologies used.
Reports discussed:
Sufficiency of the Transportation Trust Fund in Meeting the State’s Transportation Needs, Louisiana Legislative Auditor
Options to Make Virginia's Individual Income Tax More Progressive, Virginia Joint Legislative Audit and Review Commission
Speakers: Irina Hampton, senior performance auditor, Louisiana Legislative Auditor
Ellen J. Miller, chief economic development and quantitative analyst, Virginia Joint Legislative Audit and Review Commission
|
Track
How We Evaluate It
|
|
1:30-2:45 p.m.
House Chamber
|
LegisStat, Program Inventories and Other Emerging Programming in the World of Evidence- Based Policymaking
LegisStat, program inventories and other emerging strategies are helping states -- including their legislatures -- use evidence and data to get better results for the people they serve and more return on investment from their spending. These strategies are bringing a new approach to hearings, budget making, and performance monitoring. The panel features two experts on these approaches, including one with a national lens and one who has been working to implement reforms led by New Mexico's Legislative Finance Committee. The panel will focus on practical how-to advice for implementing these reforms, as well as interactive Q&A with the audience.
Moderator: Micaela Fischer, Program Evaluation Manager, New Mexico Legislative Finance Committee
Speakers: Charles Sallee, director, New Mexico Legislative Finance Committee Andrew Feldman, founder, Center for Results-Focused Leadership
|
Track
Learning from Others
|
|
3-4:45 pm
Rotunda
|
Tours of the New Mexico Capitol
The State Capitol, often referred to as the "Roundhouse," because of its circular structure, houses a permanent, public collection of contemporary art by artists of New Mexico. It is a unique example of how art and politics can enhance each other. It is among the most comprehensive collections of contemporary art in the region, featuring works by artists from Taos to Tucumcari and everywhere in between.
|
|
5-7 p.m.
Rotunda
|
Reception & 50 Years of NCSL
Join us in the Roundhouse rotunda for remarks from NCSL Staff Chair, Sabrina Lewellen, followed by refreshments, light hors d'oeuvres and good company.
Introduction: Darin “DRU” Underwood, NLPES chair, Utah
Speaker: Sabrina Lewellen, deputy director and assistant secretary of the Arkansas Senate & NCSL staff chair
|
|
7-7:30 p.m.
|
Walk back to Hotel
|
|
Evening
|
Open evening for dinner on your own
|
|
Thursday Nov. 2 – All Sessions at the Drury Plaza Hotel
|
|
8-9:15 a.m.
Palace Ballroom
|
Program Evaluation - A Keystone of Evidence-Based Policymaking
Dr. Yokum will first highlight recent evidence-based policy developments and examples from local, state, and federal levels. And then second, he’ll forecast and suggest opportunities for where the work should go next, including a particular focus on the potential frontier of work for NLPES members.
Speaker: David Yokum, director, Policy Lab at Brown University
|
Plenary
|
|
Concurrent Sessions
|
|
9:30-10:30 a.m.
Palace Ballroom.
|
Crafting Strong Recommendations
While we likely spend a lot of time writing reports that are the fruits of many hours of work, our job isn’t simply to write reports. Recommendations are the culmination of what may be thousands of hours of planning and fieldwork. For the busiest readers (and those who place great trust in the auditors), recommendations may be the first and sometimes only thing that they need to read. Join this discussion of how you can ensure that the recommendations you are providing are clear, actionable, and measurable.
Moderator: Matt Taylor, deputy director for performance audits, Georgia Department of Audits and Accounts
Speakers: Keith Brown, senior research analyst, West Virginia Performance Evaluation & Research Division
Senaida San Miguel, policy analyst, Texas Sunset Advisory Committee
Julie Vallejo, legislative performance auditor, Office of the Tennessee Comptroller
|
Track
How We Evaluate It
|
|
9:30-10:30 a.m.
O’Keefe Room
|
Evaluating Tax Incentives
Legislatures use tax incentives spur economic development, to subsidize preferred activities or groups, or to promote fairness where the normal tax structure would create unfair outcomes. These tax incentives can have significant fiscal impact but tend to receive much less legislative attention than direct spending. The panelists will discuss their work evaluating tax incentives and share general lessons for evaluating the outcomes of tax policy.
Moderator: Ryan Langrill, principal evaluator, Idaho Office of Performance Evaluations
Speakers: Ellen J. Miller, chief economic development and quantitative analyst, Virginia Joint Legislative Audit and Review Commission
David W. Singer, management and program analyst Legislative Audit Division Charles Wilson, risk assessment manager, Texas State Auditor’s Office
|
Track
What We Evaluate
|
|
9:30-10:30 a.m.
Lamy Room
|
Monitoring and Evaluating Pandemic Spending
After the COVID-19 pandemic emerged, state and local governments received a massive infusion of federal aid to support testing and treatment, aid struggling businesses, and provide education in a virtual or hybrid format. This public health emergency presented NLPES member offices with the challenge of monitoring and evaluating pandemic spending in a timely and thorough manner as the social, economic, and political context changed rapidly. Panelists will discuss how their offices met this challenge and offer broader lessons about how we can do good evaluation work during an unexpected crisis and the large-scale policy, program, and funding changes that ensue.
Moderator: Jason Juffras, senior analyst, Office of the District of Columbia Auditor Speakers: Sohara Monaghan, senior performance auditor, Washinton State Auditor Jennifer Sebren, deputy director, Mississippi PEER Committee
Dean Sewnson, deputy state auditor for performance evaluation, Wisconsin Legislative Audit Bureau
|
Track
How We Evaluate It
|
|
Concurrent Sessions
|
|
10:45-11:45 a.m.
Palace Ballroom.
|
Working in the Field in the Digital Era
In this session, panelists will discuss how their offices work in the field in an era where remote and hybrid environments are common. Amongst other topics, panelists will discuss lessons learned about planning and allocating resources towards work outside of the office, the impact of remote and hybrid work on program evaluation and auditing, and barriers to using the results of field work in reports.
Moderator: Darren McDivitt, policy analyst, Texas Sunset Advisory Commission Speakers: Nehemiah Chinavare, assistant principal performance evaluator, Wisconsin Legislative Audit Bureau
Sadie Smeck, policy analyst, Texas Sunset Advisory Commission
Craig Timmons, staff development manager, Georgia Department of Audits and Accounts
|
Track
How We Evaluate It
|
|
10:45-11:45 a.m.
O’Keefe Room
|
How to Amplify Your Work
One well-known aspect of program evaluation and audit is communicating the results via findings and recommendations to stakeholders and decision makers. In today’s world there is a never-ending list of potential communication channels that could expand the reach of the work beyond legislative stakeholders. In this session we will explore the various strategies and tools that can help in this endeavor to amplify your work.
Moderator: Marcus Morgan, director, Alabama Commission on the Evaluation of Services Speakers: Lisa Kieffer, director, Performance Audits, Georgia Department of Audits and Accounts
Ross Ryan, principal performance evaluator, Wisconsin Legislative Audit Bureau
|
Track
How We Evaluate It
|
|
10:45-11:45 a.m.
Lamy Room
|
Monitoring Procurement as an Evaluation Component
What could be more interesting than watching what the state buys? Staff from New Mexico and Mississippi will review recent evaluations of procurements and procurement processes in their states, share overarching findings, and reflect on lessons learned in performing procurement-focused evaluations.
Speakers: Micaela Fischer, New Mexico Legislative Finance Committee Matthew Holmes, lead analyst, Mississippi PEER Committee
|
Track
How We Evaluate It
|
|
Noon
|
Seminar Concludes
|