Professional Development Resources: National Legislative Program Evaluation Society

3/3/2020


The information below indicates the most appropriate target audience for each of the training items, the type of media used for the training item (e.g., PowerPoint), and its appropriateness for Yellow Book training.  Thanks to the states that have contributed to the training resources collection.

Planning and Scoping

Developing Evaluation Plans—This PowerPoint is about developing evaluation work plans.  A key part of that process is laying your ideas out on paper.  This presentation describes how to develop a plan to incorporate your research questions/issues, methods, staff assignments, deadlines and communication strategies into one document that will help your team stay on task and on time.

  • Power Point—(36 slides)
  • Designed as a 60 minute class with instructor notes and group exercises.

 

Planning and Scoping Evaluations—The audit and evaluation process is broken up into three phases—planning/scoping, fieldwork and writing.  This 60-minute webinar  addressed planning and scoping evaluations.  It covered an overview of Texas's scoping tool and staff training; Michigan's referent group meeting; and general best practices and pitfalls for planning and scoping.

  • Webinar (60 minutes)
  • Can be used for individual training or in a classroom setting.

Fieldwork

2020's Remarkable Research Methods—In 2020, NLPES recognized the New Mexico Legislative Finance Committee and  the Washington Joint Legislative Audit and Review Committee for their outstanding use of evaluation research methods.  During this webinar, you will learn more about each office's award-winning methodology and the lessons learned. New Mexico staff will describe its evaluation of childcare assistance effectiveness. Washington staff will describe how they evaluated the opportunity scholarship and opportunity expansion programs

  • Webinar (60 minutes; to be held Nov. 10, 2020; registration is not yet open)
  • Can be used for individual training or in a classroom setting.
     

Critical Thinking—For the most part, the work of legislative program evaluators involves solving problems.  To identify and solve problems, staff must be able to think critically. This PowerPoint describes the "critical thinking" concept and provides classroom exercises for applying the concept.

  • Power Point—(51 slides)
  • Designed as a 50 minute class with instructor notes and group exercises.

Confidential Data Access Issues for Auditors and EvaluatorsAccess to confidential data is often a key factor in the ability to answer legislators' concerns when performing an audit or evaluation of an agency or program. In this webinar, staff from three offices will share their experiences, issues and solutions when it comes to getting access to sensitive or confidential data necessary to do their work. Specifically, we will focus on what types of training and communications are in place to ensure our own staff know their rights and responsibilities regarding confidential data. We will discuss mechanisms our offices have in place to communicate with the auditee on getting confidential data. We also plan to discuss innovative ways or best practices to help overcome agencies' fears or reluctance to provide confidential data to audit and evaluation staff.

Finding Savings—Identifying opportunities for cost savings and cost avoidance is particularly important in times of budget shortfalls. This PowerPoint offers concrete suggestions on where to find savings or additional revenues.

  • Narrated Power Point (10 slides)
  • Can be used as a 50 minute class; does not include group exercises. 

Innovative Evaluation in New Mexico—The New Mexico Legislative Finance Committee is continually working to improve research methodologies to ensure staff are providing policy makers with the best and most up-to-date research available. The office is working with the Abdul Latif Jameel Poverty Action Lab (J-PAL) at the Massachusetts Institute of Technology on a randomized control trial on the effectiveness of early college high school curricula in New Mexico. Learn about New Mexico's innovative research methodology and J-PAL's State and Local Initiative.

  • Webinar (50 minutes)
  • Can be used for individual training or in a classroom setting.

Interviews—Interviewing is a critical part audit and evaluation work. Conducting good interviews takes time and practice. This PowerPoint provides information on basic interviewing skills.

  • Power Point (20 slides)
  • Designed as a 50 minute class, with instructor notes and group exercises.

Pivot Tables Are Easy—This webinar illustrates that a pivot table report is a surprisingly easy, interactive way to quickly explore large amounts of data in many different ways. A pivot table report can be used to analyze numerical data in detail and to answer unanticipated questions about your data.

  • Webinar (60 minutes)
  • Can be used for individual training or in a classroom setting.

The Power of Pivot Tables:  Number Crunching Made EasyLearn to use pivot tables to do frequency counts, cross-tabulate data, calculate subtotals and averages by grouping categories, plus much more.

  • Webinar (60 minutes)
  • Can be used for individual training or in a classroom setting.

Quantitative Methods—Hard numbers can provide compelling evidence. This PowerPoint provides an overview of three quantitative methods: central tendency, regression and survival analysis.

  • Power Point (27 slides)
  • Designed as a 50 minute class with instructor notes and discussion slides at the end.

Research Methods That Rock—In 2019, NLPES recognized Kansas, Louisiana and Virginia for their outstanding use of evaluation research methods.  During this webinar, you will learn more about from each office's award-winning methodology and the lessons learned. Kansas staff will describe how they evaluated the state's Medicaid program. Louisiana staff will describe how they measured local government bond issuance costs. Virginia staff will discuss its analyses of spending on inmate healthcare.

  • Webinar (60 minutes; to be held Mar. 26, 2020; registration is open)
  • Can be used for individual training or in a classroom setting.

Reviewing State Agency Contracts—The increase of government contracting has increased legislative interest in the accountability of contracted projects. This training describes how to conduct consistent, well-documented performance reviews of state contracts.

  • Power Point (36 slides)
  • Designed as a 50 minute class with instructor notes and group exercises.

Samples—Properly executed samples allow us to collect less information but reach reliable conclusions. This PowerPoint provides an overview of types of samples, explains when each is appropriate, and discusses how to determine sample size.

  • Power Point (14 slides)
  • Can be used as a 50 minute class if local examples are added for discussion purposes.

Survey Development—Getting good survey data starts with survey development.  This PowerPoint discusses determining your survey goals and objectives, sample selection, methodology, survey layout and question and response sets.

  • Power Point (29 slides)
  • Designed as a 50 minute class with instructor notes and group exercises. Adding local examples would be helpful.

Survey Refresher—These slides present ways to present surveys and when to conduct them.

Two Outstanding Research MethodsEach year, the National Legislative Program Evaluation Society recognizes up to three offices for reports that demonstrate outstanding use of evaluation research methods.  In 2016, the Louisiana Legislative Auditor and the Colorado Office of the State Auditor won Excellence in Research Methods awards.  During this webinar, Louisiana and Colorado staff will explain their award-winning methodologies.

Louisiana staff will describe its method to measure how lottery ticket sales respond to changes in the prize payout percentage and how this method can be applied to estimate how a change in policy will affect outcomes in a variety of settings.

Colorado staff will describe how GIS-based methodologies were applied to the sales tax system and how this method can be used to identify, analyze and communicate patterns and relationships that have a spatial dimension.

  • Webinar (60 minutes)
  • Can be used for individual training or in a classroom setting.

Utah's Award-Winning Methods to Test Regulation of Payday Lending—The Utah Office of the Legislative Auditor General won the 2017 NLPES Excellence in Research Methods Award for its report testing the regulation of payday lending. Staff from the Utah Office of the Legislative Auditor General will explain the audit issue that they were asked to address, the award-winning methodology that they developed to address the issue, and how the methodology might be applied to other contexts.

  • Webinar (60 minutes)
  • Can be used for individual training or in a classroom setting.

Writing and Publication

Narrated PowerPoints—Produce short, narrated PowerPoint presentations to quickly summarize or pique interest in your reports.  This presentation provides tips on how to make them.

Publishing Reports Online—Moving to online reports involves cultural, process and technological changes. Staff from the Washington Joint Legislative Audit and Review Committee will share lessons learned and discuss the processes involved with publishing reports online.

  • Webinar (60 minutes, scheduled for May 2, 2019; registration is open)
  • Can be used for individual training or in a classroom setting.

Rapid Response Assistance—More evaluations offices are being asked to provide information quickly to meet legislative research needs. This narrated PowerPoint describes how to quickly organize and execute rapid response assistance requested by the legislature.

  • Narrated Power Point (5 minutes)
  • Can be used as the basis for a class if the instructor adds local examples of requests and discusses local office policies and procedures for rapid response requests.

Report Writing—Every report’s usefulness hinges on its ability to communicate effectively. This PowerPoint discusses keys to crisp writing, including tips at the sentence, paragraph, and overall structure levels, as well as how to provide a good final polish.

  • Power Point (48 slides)
  • Can be used as the basis for a class but does not include instructor notes or exercises.

Report Writing—This webinar by Stan Stenersen explores how to convey complex information to busy readers. It gives writers an understanding of how busy readers learn as well as a set of tools for writing documents that can be understood and acted upon. The webinar includes several opportunities for participation using handouts.

  • Webinar (90 minutes)
  • Can be used for individual training or in a classroom setting.

Using Data Visualization to Improve Messaging—In order to draw attention to work that incorporates issues and convinces others to be proactive, a message must be clearly and accurately articulated.  The webinar will provide a brief synopsis on how people see and understand information from data displays. It will provide fundamentals on how to present good data visualizations and discuss the importance of using the appropriate colors and charts to engage the audience.

  • Webinar (60 minutes)
  • Can be used for individual training or in a classroom setting.

Management

Strategic Planning for Legislative Oversight Offices—Oversight offices, which look at other agencies’ strategic planning, can benefit from the process as well. Good strategic plans can help your office identify your goals and objectives, allocate resources accordingly, and develop and use appropriate performance measures.

  • Narrated Power Point (8.5 minutes)
  • Can be used to develop or enhance a strategic plan for your office.

Legislative Code of Conduct—Legislative staff must make difficult judgments sometimes to guide their service to their state legislatures. This PowerPoint provides basic concepts to help staff develop a sense of responsibility for their actions.

  • Power Point (21 slides)
  • Can be used for a 50 minute class but does not include local examples of ethical dilemmas and how they should be addressed.

Ethics Training for Program Evaluators and Performance Auditors—Evaluators and auditors generally are the legislative staff responsible for assessing and judging state agencies and programs.  Because of the important role they play, we want to believe that evaluators and auditors are ethical.  Ethical behavior, however, is not always black and white.

  • Webinar (archived version)
  • Can be used for individual training or in a classroom setting.

Wikipedia for Your Office—Provide factual information about your office history and services (thereby also preventing others from creating an entry about your office.)

Twitter Your Office Products—Use Twitter to provide short, quick notification of report releases and other timely information.

Managing Millennials—Integrating new employees is an important part of maintaining an effective staff. This podcast offers insights into the working style of millennials, as well as other generations of employees, to help managers understand employee needs and preferences.

  • Podcast (10 minutes)
  • Can be used as the basis for a class or discussion for supervisors about working with millennials.

Mentoring—Mentors can help new staff understand your work environment and expectations. This narrated PowerPoint addresses three questions: what is mentoring, what are the different types of mentors, and how can mentors be successful?

  • Narrated Power Point (8 minutes)
  • Can be used as a class for managers or a discussion about how to assimilate new employees.

Succession Planning—Most legislative program evaluation and audit offices face a future in which senior staff will be retiring. This narrated PowerPoint outlines basic concepts for the transfer of knowledge and responsibilities from senior to newer staff.

  • Narrated Power Point (8 minutes)
  • Can be used for a class for managers about preparing for the retirement of baby boomers.
     

Evidence-based Policies and Programs

Drugs and MoneyThe data analysis hurdles and solutions involved in measuring participant outcomes and cost savings generated by expansion drug courts—This webinar will explain the rationale and use of the methodological techniques featured in OPPAGA Report 14-02, which won the 2015 NLPES Excellence in Research Methods Award.  The data revolution provides many new opportunities for program evaluators to empirically measure and analyze the real world outcomes of the programs we study, but it also brings a new set of challenges.  One of the major challenges is how to properly compare the outcomes of groups participating in a program to non-participants.  Groups differ in characteristics that could affect outcomes and these measurement challenges are exacerbated when the outcomes being measured are affected by exposure time.  For example, criminal recidivism rates can differ drastically, when measured over the period of six months versus eighteen months.  This E-Learning webinar provides an example of how to deal with these types of data challenges using a technique called matched pair survival analysis, which can be performed in many commonly used statistical software packages.

  • Webinar (45 minutes) 
  • Can be used for individual training or in a classroom setting.

Teen Pregnancy Prevention Program: A Case Study in Evidence-based Policymaking—As one of the few government programs that both uses evidence as a criteria for funding decisions and continues to rigorously evaluate its efforts and results, the Teen Pregnancy Prevention Program can offer important lessons for state policymakers.  This interactive webinar provides an opportunity for state legislators and legislative staff interested in evidence-based policymaking to learn more about TPP and hear insights that could help inform state-level efforts.