2019 NLPES PDS Preliminary Agenda

8:30 a.m.-4 p.m.

NLPES Executive Committee Meeting
Sheraton Park City

Registration/Information Desk
Sheraton Park City

Registration/Information Desk
Sheraton Park City 

Welcome and Plenary Session 1—The Value of Performance Audits: A Legislative Perspective
Sheraton Park City—Prospector 2-3
Legislative evaluators strive to ensure their work is accurate, objective, and relevant in order for it to be useful to legislators. In this session, we will learn from leaders about what they value in the work that we do.

Speakers:
Kade Minchey, Utah Office of the Legislative Auditor General
Representative Brian King, Utah

Concurrent Session 1-General Track—The Complexities of Public Education Audits
Sheraton Park City—Silver Mine A-B
This session is designed to focus on methods of holding public education boards and school systems accountable, and discuss processes used in evaluations and audits for addressing deficiencies in public education.

Moderator:
Jennifer Sebren, Mississippi Joint Legislative Performance Evaluation and Expenditure Review Committee

Panelists:
Laura Carroll, Tennessee Office of the Comptroller of the Treasury
Jaclyn Clute, Tennessee Office of the Comptroller of the Treasury
Edward Seyler, Louisiana Legislative Auditor
Debbie Davis, Utah State Board of Education

Concurrent Session 2-General Track—Predicting the Future: An Introduction to Forecasting Methods and Tools
Sheraton Park City—Prospector 4
Various statistical approaches can be used to identify patterns over a defined period (days, months, years) and can be used to predict future trends. In this session, presenters will introduce the basics of approaching problems where forecasting future performance can be useful, provide some examples and discuss when it is appropriate to use.

Panelists:
Jon Courtney, New Mexico Legislative Finance Committee
Joshua Karas, Washington Joint Legislative Audit and Review Committee

Concurrent Session 3-Basic Skills Track, Part 1—Why Do We Audit? (Understanding Government Accountability and the Types of Evaluations)
Sheraton Park City—Wasatch
In this session, participants will explore the basics of governmental accountability and the purposes of evaluations. Topics will include the different types of evaluations, how evaluations may originate, and the range of expectations that may surround a project.

Instructors: Eric Thomas, Washington Joint Legislative Audit and Review Committee
Patricia Berger, Pennsylvania Legislative Budget and Finance Committee

Concurrent Session 4-Data Took Kit, Part 1—Data Preparation: Cutting the Lead Time for Data Analysis
Sheraton Park City—Executive Board Room 
In this session, participants will explore the basics of governmental accountability and the purposes of evaluations. Topics will include the different types of evaluations, how evaluations may originate and the range of expectations that may surround a project.

This session is limited to 12 participants. Preregistration is required. Email Brenda Erickson to sign up for this session.

Instructor: Tim Bereece, Utah Office of the Legislative Auditor General

Concurrent Session 5-General Track—Insights from Natural Resources and Public Land Audits
Sheraton Park City—Prospector 4 
Most states face challenges in ensuring the availability and quality of natural resources. This panel will discuss the oversight of state water, land, and other natural resource programs.

Moderator:
Will Soller, Montana Legislative Audit Division

Panelists:
Rachel Murata, Washington Joint Legislative Audit and Review Committee
Gina Brown, Louisiana Legislative Auditor
Kevin Brown, West Virginia Performance Evaluation and Research Division

Concurrent Session 6-General Track—Managing Workflow to Ensure Timely Reports
Sheraton Park City—Silver Mine A-B 
The panel will discuss internal procedures that offices have implemented to help manage audit risk and workflow to ensure reports are timely and maintain credibility.

Moderator:
Courtney Phillips, South Carolina Legislative Audit Council

Panelists:
Judy Randle, Minnesota Office of the Legislative Auditor
Meghan Flanders, Kansas Legislative Division of Post Audit
Kevin John, Utah State Board of Education

Concurrent Session 7-Basic Skills Track, Part 2—How Do We Conduct Audits? (Understanding Audit Methodologies and Planning for Success)
Sheraton Park City—Wasatch
In this session, participants will review the attributes of a complete finding, critical areas in planning the audit, the elements of a sound problem statement, and the types of critical thinking required.

Instructors:
Jennifer Sebren, Mississippi Joint Legislative Performance Evaluation and Expenditure Review Committee
Leslie McGuire, Georgia Department of Audits and Accounts
David Arner, Georgia Department of Audits and Accounts

Concurrent Session 8-Data Took Kit, Part 2—Simple Scripting: Convince Excel to Do Your Work
Sheraton Park City—Executive Board Room 
In this session, you will be introduced to simple scripting within Excel using VBA to automate repetitive data preparation tasks that may become hurdles to performing valuable data analytics.

This session is limited to 12 participants. Preregistration is required. Email Brenda Erickson to sign up for this session.

Instructor:
Tim Bereece, Utah Office of the Legislative Auditor General

Lunch and Plenary Session 2—Working with the Media
Sheraton Park City—Prospector 2-3 
Introductions:
Wayne Kidd, Utah Office of the Legislative Auditor General

Speaker: 
Mick Bullock, NCSL

Concurrent Session 9-General Track—Finding and Keeping the Best Employees
Sheraton Park City—Silver Mine A-B 
In an increasingly competitive marketplace, good employees tend to have many job opportunities. This session will explore states' experiences and success stories with recruiting, training and retaining good employees. This will be an interactive discussion with the panelists and audience.

Moderator:
Drew Dickinson, Virginia Joint Legislative Audit and Review Commission

Panelists:
Patricia Berger, Pennsylvania Legislative Budget and Finance Committee
Emily Dixon, Louisiana Legislative Auditor

Concurrent Session 10-General Track—Reviewing Higher Education Operations and Governance
Sheraton Park City—Prospector 4

Higher education governance systems vary widely across states. Generally, states have either one of two models: strong statewide control with authority maintained at the state level or strong institutional control with authority at the local level. Strong statewide systems coordinate and oversee the degrees offered, funding and other functions of the institutions. Whereas, with local level control, each institution decides the degrees to offer and each institution asks the legislature separately for funding. This session will review various higher education governance models and the pros and cons of those models.

Moderator:
Danielle Childress, Virginia Joint Legislative Audit and Review Commission

Panelists:
Alyssa Sorenson, Montana Legislative Audit Division
Katie Cardenas, California State Auditor’s Office
Benn Buys, Utah Office of the Legislative Auditor General

Concurrent Session 11-Data Took Kit, Part 3—Analyzing Big Data Using PowerPoint
Sheraton Park City—Executive Board Room

When Excel’s one million row limit becomes an issue, learn how Microsoft’s PowerPivot utility can expand your capacity to evaluate larger data sets within the comforts of the Excel environment you know.  This session is limited to 12 participants. Preregistration is required.

Instructor:
Michael Allred, Utah Office of the Legislative Auditor General

Concurrent Session 12-General Track—Regulating Regulatory Audits
Sheraton Park City—Silver Mine A-B
Occasionally, we encounter barriers when auditing regulatory entities. This session focuses on strategies to provide accountability in industries such as payday lending and bail bonds, who may be receiving insufficient oversight from their regulators.

Moderator:
Karen Leblanc, Louisiana Legislative Auditor

Panelists:
Ryan Langrill, Idaho Office of Performance Evaluations
Alan Leonard, Texas Sunset Advisory Commission
Michael Midkiff, West Virginia Performance Evaluation and Research Division

Concurrent Session 13-General Track—Staying Relevant and Advancing the Field of Program Evaluation
Sheraton Park City—Prospector 4
In this session, panelists and the audience will discuss approaches their offices are using to stay relevant in meeting legislatures’ expectations, and in advancing and promoting the use of evaluation services.

Moderator:
Caitlin Badger, Minnesota Office of the Legislative Auditor

Panelists:
Eric Thomas, Washington Joint Legislative Audit and Review Committee
Paul Navarro, California State Auditor’s Office
Darin Underwood, Utah Office of the Legislative Auditor General

Concurrent Session 14-Data Took Kit, Part 4—Visualization: Surveying Risk Areas through Data Visualization
Sheraton Park City—Executive Board Room 
Don’t wait for the report, use visual analytics tools such as Excel and Power BI to quickly see the patterns, anomalies, and other incongruencies that lead to vital program evaluation questions.

This session is limited to 12 participants. Preregistration is required. Email Brenda Erickson to sign up for this session.
Instructor: Tyson Cabulagan, Utah Office of the Legislative Auditor General

Plenary Session 3-Impact Award Information Fair
Sheraton Park City—Prospector Patio 
Representatives from offices who received the 2019 impact awards will discuss their reports.

Plenary Session 4—Identifying and Addressing Ethical Dilemmas
Sheraton Park City—Prospector 2-3
Dr. Rick Green will discuss common ethical dilemmas that arise in evaluations and offer a framework for how to resolve them.
Introduction:
Sarah Flanigan, Utah Office of the Legislative Auditor General
Speaker:
Dr. Rick Green, University of Utah

Concurrent Session 15-General Track—Making Qualitative Data Count: Essential Qualitative Research Skills and Tools
Sheraton Park City—Silver Mine A-B

The panel will discuss various qualitative research skills and tools, such as focus groups, reviewing documents, and observations used to identify effective public programs. This session covers interpreting, analyzing and including different perspectives in evaluations.

Moderator:
Nick Gavin, Washington Joint Legislative Audit and Review Committee

Panelists:
Amanda Bartlett, Idaho Office of Performance Evaluations
Stefanie Winzeler, Colorado Office of the State Auditor
Van Christensen, Utah Department of Workforce Services

Concurrent Session 16-General Track—Leadership Skills for All Levels of the Organization
Sheraton Park City—Prospector 4 
We all have different roles in the audit team. In the session, the panelists and audience will discuss important leadership skills that help to build better teams, and help participants understand how teams can be more productive.

Moderator:
Kiernan McGorty, North Carolina Program Evaluation Division

Panelists:
Melissa Simpson, Arkansas Division of Legislative Audit
Emily Johnson, Texas Sunset Advisory Commission
Joy Hill, Wyoming Legislative Service Office

Concurrent Session 17-Basic Skills Track, Part 3—How Do We Structure and Compile Evidence? (Understanding the Process of Gathering and Filing Evidence)
Sheraton Park City—Wasatch 
In this session, participants will learn tips for collecting data, analyzing data, combining evidence and documenting results.

Instructors:
Jon Courtney, New Mexico Legislative Finance Committee
Wayne Kidd, Utah Office of the Legislative Auditor General

Concurrent Session 18-Data Took Kit, Part 1—Data Preparation—Cutting the Lead Time for Data Analysis
Sheraton Park City—Executive Board Room

This session is a hands-on interactive session reviewing tips and tools such as Excel to help prepare your data for analysis. These tools will improve the functionality of your spreadsheet, keeping it clean and auditable, and leaving you more time for analysis. This session is limited to 12 participants. Preregistration is required.

Instructor: Tim Bereece, Utah Office of the Legislative Auditor General

Concurrent Session 19-General Track—Data Visualization: A Picture’s Worth a 1,000 Words
Sheraton Park City—Silver Mine A-B

Panelists will share examples and suggestions on how to effectively use graphics, maps, and other visuals to present findings and enhance reports.

Moderator:
Edward Seyler, Louisiana Legislative Auditor

Panelists:
Liz Thomas, Washington Joint Legislative Audit and Review Committee
Katherine Theisen, Minnesota Office of the Legislative Auditor
Tyson Cabulagan, Utah Office of the Legislative Auditor General

Concurrent Session 20-General Track—Collaborative Problem Solving
Sheraton Park City—Prospector 4 
Legislative staff can be part of situations where two sides are locked into a conflict, and they need help to resolve it. This hands-on workshop is designed to teach staff to apply the principles of negotiation to help explore creative ways of delivering solutions to conflicts in a way that all sides can leave feeling satisfied.

Introduction:
Linda Triplett, Mississippi Joint Legislative Performance Evaluation and Expenditure Review Committee

Speaker:
Curt Stedron, NCSL

Concurrent Session 21-Basic Skills Track, Part 4—How Do We Report Audit Findings? (Understanding How to Clearly Communicate the Results)
Sheraton Park City—Wasatch
In this session, participants will discuss how to develop the written report and presentations to clearly and concisely communicate audit findings.

Instructor:
Kiernan McGorty, North Carolina Program Evaluation Division
Kristen Rottinghaus, Kansas Legislative Division of Post Audit

Concurrent Session 22-Data Took Kit, Part 2—Simple Scripting: Convince Excel to Do Your Work
Sheraton Park City—Executive Board Room
In this session, you will be introduced to simple scripting within Excel using VBA to automate repetitive data preparation tasks that may become hurdles to performing valuable data analytics.

This session is limited to 12 participants. Preregistration is required. Email Brenda Erickson to sign up for this session.
Instructor:
Tim Bereece, Utah Office of the Legislative Auditor General

NLPES Awards Lunch
Sheraton Park City—Prospector 2-3 
NLPES awards will be distributed to 2019 recipients, and additional information will be presented about next year’s PDS.

Presiding:
Jon Courtney, New Mexico Legislative Finance Committee, and NLSSA Chair

Concurrent Session 23-General Track—Challenges of Auditing Non-governmental Entities
Sheraton Park City—Silver Mine A-B 
How does our process change when our auditee is outside of government? Panelists will present lessons learned from audits involving non-profit organizations, foundations, and private organizations that either receive state appropriations or are affiliated with state agencies.

Moderator:
Zack Freeman, Washington Joint Legislative Audit and Review Committee

Panelists:
Mike Paoni, Illinois Office of the Auditor General
Dean Swenson, Wisconsin Legislative Audit Bureau
Jenny Page, Colorado Office of the State Auditor

Concurrent Session 24-General Track—Strategic Planning: Developing a Long-term Plan for Your Office
Sheraton Park City—Prospector 4 
A strategic plan can help set priorities, focus resources and help the organization work toward common goals. This session will discuss how to go about organizing a strategic plan and the benefits offices have realized in having a strategic plan.

Moderator:
Kade Minchey, Utah Office of the Legislative Auditor General

Panelists:
Keenan Konopaski, Washington Joint Legislative Audit and Review Committee
Christopher Harless, Colorado Office of the State Auditor

Concurrent Session 25-Data Took Kit, Part 4: Visualization—Surveying Risk Areas through Data Visualization
Sheraton Park City—Executive Board Room

Don’t wait for the report, use visual analytics tools such as Excel and Power BI to quickly see the patterns, anomalies, and other incongruencies that lead to vital program evaluation questions.  This session is limited to 12 participants. Preregistration is required.

Instructor:
Tyson Cabulagan, Utah Office of the Legislative Auditor General

Plenary Session 5—Debate Thinking: Responding on Your Feet
Sheraton Park City—Prospector 2-3
In the heat of a disagreement, argument or debate, it can be difficult to plot a persuasive strategy that effectively articulates your point of view while refuting the position of the other party. Debate Thinking will explore a model of thought that will sharpen your ability to think quickly and to develop compelling offensive and defensive arguments in real time.

Introduction:
Linda Triplett, Mississippi Joint Legislative Performance Evaluation and Expenditure Review Committee

Speaker:
Curt Stedron, Program Principal, NCSL

Registration/Information Desk    
Sheraton Park City

Plenary Session 6—Engaging in Empowering Evaluations
Sheraton Park City—Prospector 2-3
This session will discuss the tools to demonstrate effectiveness in accomplishing your audit objectives, to inspire change, and eliminate practices that don’t work.
Introduction:
Nicole Luscher, Utah Office of the Legislative Auditor General
Speaker:
Dr. Eva Witesman, Brigham Young University

Concurrent Session 26-General Track—Driving Carefully through Transportation Audits
Sheraton Park City—Silver Mine A-B 
Many states are confronted with transportation infrastructure issues. In this session, we will discuss approaches used to evaluate transportation programs for efficiency and effectiveness.

Moderator:
Jeremy Verhasselt, Montana Legislative Audit Division

Panelists:
Brian Hoffmeister, New Mexico Legislative Finance Committee
Noah Natzke, Wisconsin Legislative Audit Bureau
James Sutcliff, Montana Legislative Audit Division

Concurrent Session 27-General Track—The History of NLPES
Sheraton Park City—Prospector 4 
In the session, participants will learn about how NLPES got started and how it has evolved over the years.

Introduction:
Linda Triplett, Mississippi Joint Legislative Performance Evaluation and Expenditure Review Committee

Speaker:

John Turcotte, North Carolina Program Evaluation Division

Concurrent Session 28-Data Took Kit, Part 3: Analytics—Analyzing Big Data Using PowerPivot
Sheraton Park City—Executive Board Room

When Excel’s one million row limit becomes an issue, learn how Microsoft’s PowerPivot utility can expand your capacity to evaluate larger data sets within the comforts of the Excel environment you know.  This session is limited to 12 participants. Preregistration is required.

Instructor:
Michael Allred, Utah Office of the Legislative Auditor General

Concurrent Session 29-General Track—The Integrated Data System: Opportunities and Challenges
Sheraton Park City—Silver Mine A-B

Integrating data across agencies and data systems is an increasingly popular approach that can better inform performance management, program evaluation, cost-benefit analysis and policymaking. However, data systems designed to integrate data from multiple sources typically carry high risk and sharing of data across agencies also brings ethical concerns around data usage and privacy. In this session presenters will discuss the opportunities and challenges of using integrated data. Presenters also will discuss best practices for integrated data systems, common implementation challenges, and offer insight from real world examples.

Moderator:
Shunti Taylor, Georgia Department of Audits and Accounts

Panelists:
Sarah Dinces, New Mexico Legislative Finance Committee
Drew Mingl, Utah Department of Technology Services

Concurrent Session 30-General Track—Encountering Lobbyists and Special Interest Groups
Sheraton Park City—Prospector 4
Panelists and audience members will discuss approaches they have used to efficiently obtain input from lobbyists and special interest groups. They will also discuss tips about dealing with lobbyists, and lessons learned.

Moderator:
Emily Johnson, Texas Sunset Advisory Commission

Panelists:
Patricia Berger, Pennsylvania Legislative Budget and Finance Committee
Ryan Langrill, Idaho Office of Performance Evaluations
Darren McDivitt, Texas Sunset Advisory Commission

Lunch and Plenary Session 7—Realities of Legislative Leadership
Sheraton Park City—Prospector 2-3
We want our work to be used by legislators, but we must understand their information needs and perspectives. The former speaker of the House of Representatives in Utah will discuss the realities of legislative leadership and ways that we can best serve their needs.
Introduction:
Kade Minchey, Utah Office of the Legislative Auditor General 

Speaker:
Greg Hughes, former speaker of the Utah House of Representatives

Plenary Session 8—The Urgency of Women's Leadership
Sheraton Park City—Prospector 2-3
Elevating women in business and in politics is not just the right thing to do, it's the smart thing to do. This session will help the participants understand the important role that men play as allies and advocates of women, the economic benefits of gender diverse leadership in organizations, and the emerging demand for different perspectives in making critical decisions. As an added bonus, participants may walk away with a better understanding of relationships with the opposite gender.
Introduction:
Darin Underwood, Utah Office of the Legislative Auditor General
Speaker:
Patricia Jones, CEO, Women's Leadership Institute