Chair's Corner
A Work in Progress | Kiernan McGorty (North Carolina)
Wow, a lot has changed for me since my last chair letter in January 2021! Many of you already know that on Feb. 15, 2021, North Carolina legislative leaders decided to eliminate the Program Evaluation Division.
I grew up, professionally, at PED, having worked there since its inception in 2007. During my 13 years at PED, I transitioned from serving as a team member, to team lead, to manager, and finally to the director role last September. Additionally, during that time I had three children! Although the latter may seem separate, something the pandemic and working from home has allowed us is a glimpse into each other's personal lives and how much we all are juggling. For me, PED's elimination resulted in a work identity crisis that made me question my world view and professional choices. It's been a messy process, and I'm still in it. Turns out we grieve in character, and I can best be characterized as someone whose feelings have feelings. 😊
Recently, I started seeing a life coach to help me sort out what's important to me going forward. One of my homework assignments is to keep a gratitude journal and to view it not as another task I need to do but rather as something I enjoy (turns out attitude is everything). It's such a simple idea and I've heard about gratitude journals before (from Oprah), but knowing my coach is going to ask me if I've kept up my entries has made me actually do it. And it works! It helps me see my life through the lens of what I have rather than dwelling on what I'm missing. Usually it's filled with simple things like discovering a new song or laughing at a witty comment.
For today's entry, I'm going to focus on some of the bigger things this experience has made me grateful for.
- I'm grateful for my 13 years with PED where every day was a new challenge. I learned so many new skills and discovered my strengths and interests. I am amazed at what we accomplished as a team!
- I'm grateful for colleagues who became my work family, seeing me through good and bad days in my professional and personal life. And for all the laughs we had along the way.
- I'm grateful for my NLPES and NSCL network. Besides always being a great resource, you provided me with a caring, professional community across the country. The number of you that reached out to offer your sympathy and support in the last few months warmed my heart at a time when I really needed it.
- I'm grateful for my fellow NLPES Executive Committee members and Brenda. This is a team that takes care of all of its players and has made my job as chair easy. I'm so proud of the good work we've done this year creating professional development opportunities (shameless plug: remember to register for the fall Professional Development Seminar) and building the online community (remember to sign up for the NLPES LinkedIn Group).
- I'm grateful for my inner circle. My friends and family who have listened to all of my long-winded work stories through the years. And my husband, whose partnership has made each of my career accomplishments possible and whose reassurance is a great source of comfort as I fall and get back up again.
- I’m grateful for this opportunity to reflect and reassess. I look forward to the new challenges that await and emerging even stronger, professionally and personally.
Kiernan McGorty is the 2020–2010 NLPES Executive Committee Chair. She can be reached by email or at (919) 301-1393.
Research and Methodology
Washington JLARC to Add Racial Equity Analysis to Audits and Other Studies | Rebecca Connolly and Eric Whitaker (Washington)
During the 2021 session, the Washington State Legislature directed the Joint Legislative Audit and Review Committee (JLARC) to incorporate a racial equity analysis into each of our audits and reports where necessary and appropriate (ESSB 5405).
The bill’s prime sponsor, a JLARC member, noted that JLARC audits have not typically addressed racial disproportionalities or disparate impacts. Our study assignments come from the Legislature and racial equity is often outside the scope of the audit. ESSB 5405 makes racial equity analysis a part of every study when JLARC staff determine it is necessary and appropriate as related to the study mandate.
Based on our early research, this legislative mandate appears to be unique in at least three ways:
- It is broader than those in several other states and is not limited to a specific policy area such as health care or criminal justice.
- The analysis must be retrospective rather than a forecast of what might happen in the future.
- Since the mandate is a broad directive, there will be an initial planning process to determine whether a racial equity analysis is or is not appropriate for each specific audit we undertake.
We will address this mandate like we address our studies—carefully, deliberatively, and objectively. We plan to involve stakeholders, including seven Commissions that focus on minority affairs and a newly-created Office of Equity, and to review other agencies’ practices and existing literature on this topic. We will identify criteria for when and how to include an analysis in a study as well as methods for performing this type of analysis.
If you have experiences or ideas to share about how to incorporate racial equity analysis in a study, please reach out to Rebecca Connolly or Eric Whitaker. If your office would be interested in forming a group to discuss how to incorporate equity analyses, please let us know.
Report Spotlight
Editor’s note: If your office would like to highlight its award-winning report in the Working Paper, please let Eric Thomas know.
The University of Utah Fixed a Broken Laboratory Safety System Detailed by the Utah Legislative Auditor General’s Office | Tim Bereece (Utah)
Like other research institutions across the country, the University of Utah’s laboratories contain biological, chemical, radioactive, and physical hazards that researchers and students interact with daily. Tragedies at other institutions where lab personnel died or lost arms and fingers serve as reminders of adverse events that result when safety protocols were disregarded. Broad concerns about the University of Utah’s occupational safety practices were the basis for the Utah Legislature prioritizing this audit; however, risk assessment activities quickly focused audit efforts on university labs based on similar but less-serious lab injuries. The audit was clearly impactful as its findings brought awareness to a broken risk management system in the University’s labs. The swift and comprehensive response by the University Vice President for Research and his team, which included unprecedented lab shutdowns, attest to the severity of the audit’s findings and its impact.
The University’s Risk Management System for Lab Safety Is Broken
The “broken” system of laboratory safety practices, deficiency tracking, reporting, and enforcement involved multiple hazards and levels of the institution. Safety deficiencies observed during the audit were corroborated by University of Utah documentation, specifically.
- Prevalence of major chemical hazard deficiencies.
- Failure to provide requested hepatitis B vaccinations.
- Not requiring health risk questionnaires for animal handlers.
Focusing on major chemical hazard deficiencies, these were defined by the university as “any condition which is immediately dangerous to life or health or in violation of code requirements, regulatory requirements, or accrediting agency mandates.” The extent that deficiencies went uncorrected was unknown due to inadequate use of information systems designed to track this data, so a limited data set was constructed that tracked three major chemical deficiencies. In 2017, at least one of the tracked deficiencies was cited for 44 percent of audited research groups. Those deficiencies were frequently repeated as 49 percent of audits in 2016 and 2018 for deficient lab groups indicated that these deficiencies were or would be repeated. These same deficiencies were present at major incidents at other institutions that included the death of one researcher.
The extent of deficient activities at all levels of the institution led to the conclusion that a broken system of risk management for laboratory safety existed at the University of Utah. At the ground level, research groups were not conducting safety self-assessments that serve as an effective awareness tool. Subsequently, audits by the university’s safety department were not systematically tracked, leading to inaccurate conclusions and underreported deficiencies. While university administrators were not provided comprehensive information about safety deficiencies, some issues like limited use of prescribed personal protective equipment (PPE) were repeatedly presented. In these cases, university administrators failed to exercise their authority to ensure deficiencies were resolved. This failure to exercise authority ultimately led to our recommendation that the university president direct administrators to prioritize and enforce the goal of eliminating repeat safety deficiencies from lab safety audits and inspections.
The University’s Swift and Comprehensive Response Included Unprecedented Lab Closures
In her response to the audit, the President of the University of Utah specified a couple changes that set the stage for significant actions during the year after the audit. Specifically, the Occupational Health and Safety Department was consolidated with the institution’s Radiation Safety Office and placed under the Vice President for Research. In addition, the leader of the combined unit was given specific authority to take reasonable action to ensure compliance, including shutting down lab operations when justified.
This authority was exercised on July 19, 2019, which was two months after the audit was released and followed the evacuation of the Chemistry Department following a pyrophoric chemical spill. Administrators took the unprecedented action of suspending all research activities in the Chemistry Department as comprehensive inspections were conducted of all laboratories. As corrective actions were implemented for all deficiencies, individual labs were cleared to resume their operations.
This new authority was also supplemented by information systems and strategic partnerships to identify areas needing safety enhancements. By the end of the 2019 calendar year, the University of Utah had fully implemented a laboratory management system designed to track safety audit results and corrective actions. In addition, the university was solidifying partnerships among internal departments and external experts to develop an improved occupational medicine program to identify the hazards and risks for individuals in the university’s labs.
The combined improvements of increased authority, information systems enhancements, and strategic partnerships constitute a comprehensive and timely response by the university. As the University President stated in her response to the audit “…all of us—administrators, faculty and staff—are committed to making the University a safe place for everyone.” Eighteen months after the audit was released, the Executive Director of Environmental Health and Safety acknowledged the critical role the audit played, stating: “The state audit was the catalyst for the improvements that we are making, and for that I am personally grateful. I wanted you to know that your audit resulted in real changes and improvements that are making people’s lives safer.”
Tim Bereece is a senior audit supervisor with the Utah Legislative Auditor General’s Office and was the audit team leader.
Office Spotlight
New Mexico Legislative Finance Committee Program Evaluation Unit | Jon Courtney (New Mexico)
Housed in the nation’s highest and arguably oldest capital city of Santa Fe, the Legislative Finance Committee’s (LFC) Program Evaluation Unit originated as a good idea. The idea came to life in the form of a pilot program at the Office of the State Auditor written into the state budget in 1989. At that time, the office existed under a Joint Powers Agreement between the State Auditor and LFC to support the work of an LFC audit subcommittee. The office proved to be highly impactful almost immediately finding potential issues with procurement, along with managerial and procedural deficiencies at a state agency. The office proved to be so valuable to the LFC, it was officially transferred to LFC in 1991, and the LFC as a whole took on the oversight functions of the audit committee—integrating the evaluation work of the office into the Committee’s budget- and decision-making process.
Since 1991, the Program Evaluation Unit’s ongoing mission is to help New Mexicans get the most from their tax dollars by reviewing the costs, efficiency and effectiveness of state services. LFC program evaluations cover the breadth of state government responsibilities, from public schools to information technology. The unit is led by one of two LFC deputy directors (me), has a program evaluation manager, and has now grown to 11 evaluators strong. Program evaluators typically have graduate degrees in public administration, public policy, education, law, social science, business administration, social work, or public health.
Over the last 30 years the Program Evaluation Unit has examined just about every part of state government and many local governments. Over the last four years, LFC completed 50 reports identifying over $1 billion in potential cost savings. LFC evaluations have also had uncovered criminal behavior, identified fraud and abuse, and perhaps most importantly, helped spur major investments in evidence-based programming. Just this last year during COVID we have completed evaluations showing:
- Overpayment of unemployment insurance benefits of $250 million including potentially fraudulent payments.
- Evidence of nursing home staff failing to follow guidance for use of protective equipment potentially contributing to preventable COVID deaths.
- Impacts of prekindergarten participation on high school graduation resulting in an estimated $6 to $1 return on investment.
The Program Evaluation Unit has also been working to improve methods and advance the field. For example, for the last decade LFC has worked with the Pew-MacArthur Results First Initiative, a project of The Pew Charitable Trusts and the John D. and Catherine T. MacArthur Foundation, to implement a cost-benefit analysis tool that provides policymakers with new information estimating the long-term costs and benefits of investments in public programs. Similarly, LFC staff have worked with MIT J-PAL’s State and Local Initiative to complete our first randomized control trial and are working on submitting a paper for review in a peer reviewed journal.
Major investments in what works (and moving money away from ineffective programs) is in part possible due to the unique structure of the LFC. New Mexico is one of a handful of states in which the legislative agency tasked with preparing a comprehensive budget recommendation is also tasked with oversight work via program evaluators (or auditors). LFC staff use multiple tools including evaluations, program inventories, agency report cards, and cost benefit analysis to inform their budget recommendation through a framework we call Legislating for Results. I think of the framework as a continuous quality improvement (CQI) cycle for state government where the legislature uses the best research available to fund evidence-based approaches to improve outcomes. This framework supports our non-partisan staff and Committee members in accomplishing the LFC overall mission of providing objective analyses, recommendations, and oversight of state agencies.
News & Snippets
A Retirement Is Coming
James Barber completed his bachelor’s and master’s degrees in business administration from the University of Southern Mississippi in 1977. A few months later—in April 1978, he started working for the Mississippi Joint Legislative Committee on Performance Evaluation and Expenditure Review (PEER).
Now, some 43 years later, he has decided, on Sept. 30, it will be time to experience his next great adventure—retirement!
According to former (second) PEER Executive Director John Turcotte, James “has the ability to handle highly sensitive and complex responsibilities,” and as a result, “rose through the ranks to become the fourth PEER director since the agency’s establishment in 1973.”
Max Arinder, the third PEER executive director, concurs. “His tenure represents an amazing span of time in a work environment that, on a given day, can be challenging or routine, frustrating or rewarding, disappointing or stimulating, castigating or affirming...and all the while James personally has remained calmly focused on the tasks at hand and the significance of the job to be done.”
Longtime PEER General Counsel Ted Booth also agrees, “I have never worked with a person who could do as many things extremely well as James. He will be missed.”
Turcotte remembers two occasions that illustrate James’s exceptional professional competence.
“In 1978 after just a few months as a new evaluator, using rough working papers and disjointed drafts by two audit managers,” Turcotte recalls, “James drafted—in perfect longhand—an 878-page report on the financial management of Mississippi’s 16 community colleges.”
And what would you do if, during an audit, you discovered more than $300,000 in undeposited checks kept in a cardboard box in an unlocked filing cabinet? Well, according to Turcotte, you should follow the process used by James when he dealt with this situation in 1983. Create “a perfect working paper” by “methodically listing each check on a manual spreadsheet, running a calculator tape, and having the responsible official sign and date the schedule and tape.” It’s hard for anyone to argue with documentation as solid as that!
“Throughout his career, James has shown versatility and sensitivity, being an anchor when you needed one and wind in your sails when that was best,” Arinder notes. “The last six years of his career are a fitting cap to his dedicated service to the State of Mississippi and the nation at large. Though it has been a highly charged political time, his steady leadership has allowed the PEER Committee to effectively carry out its statutory role backed by a sound, nonpartisan staff. James has faced this turbulent time, not by embracing the prevailing winds or by sheltering from them in a safe place, but by steering the course, providing the Committee with clear, reliable staff support. In doing so, he has helped to affirm the important role of the legislature in government oversight. The people of Mississippi owe him a debt of gratitude, as do all who value transparency in government. I take great pride in claiming him as both a colleague and a friend.”
“If you look up “loyalty” in the dictionary, there should be a picture of James,” says Linda Triplett, retired director of PEER’s Performance Accountability Office.
“The longest serving PEER staffer to date, James never wavered in his support of PEER, the legislative institution, his home state of Mississippi, or NCSL,” she continues. “Throughout his career, he exhibited tremendous personal loyalty to his staff, members of the legislature and their staff. He helped PEER staff to build on their strengths and address their weaknesses, and he provided legislators with the timely and accurate information and analysis needed to carry out their oversight responsibilities. James embodies the spirit of the true public servant—one who is not in it for himself, but for the privilege of serving others.”
An active member, James demonstrates his dedication to NLPES in many ways. He served as the NLPES newsletter editor for nine years and twice as NLPES chair—in 2000 and 2009. He also supports his staff’s involvement—authorizing their travel to meetings, permitting them to participate on peer review teams and encouraging them to serve in leadership roles.
We’re sure James will head into retirement with the same enthusiasm and dedication he has always shown. We just hope he works as hard at relaxing as he did during his long career.
Thank you for the decades you’ve spent with PEER, NLPES and NCSL. Thank you for being a mentor, friend and one of the hardest workers we know. You truly are appreciated!
Now, get ready to enjoy the retired life! But don’t forget to stop by and see us sometime!
More Staff Comings and Goings
Many staff changes have occurred during the past few months. Here’s a snapshot of what’s happened.
- New evaluation office in Alabama. Marcus Morgan is the director of the new Alabama Commission on the Evaluation of Services.
- Colorado Office of the State Auditor. Dianne Ray did not seek reappointment. Kerri Hunter was appointed as the new State Auditor. Her appointment became effective July 1, 2021.
- Florida OPPAGA. Phil Twogood retired on June 30, 2021. The new OPPAGA coordinator is Patricia (PK) Jameson.
- Louisiana Legislative Auditor’s Office. Daryl Purpera left March 2, 2021 (to become a fulltime pastor). Michael J. “Mike” Waguespack was elected by the legislature as the new Legislative Auditor on April 19, 2021.
- New Jersey Office of the State Auditor. Stephen Eels retired Fall 2020. David Kaschak was appointed by the legislature on Feb. 23, 2021.
NLPES Election Results
The votes for the 2021 NLPES election have been tallied, and the results are here!
- 192 NLPES members from 27 states voted.
- The four candidates elected to the 2021-2022 NLPES Executive Committee are Will Soller (Mont.), Jeanine Brown (Ga.), Drew Dickinson (Va.) and Jason Juffras (D.C.).
The membership of the 2021-2022 NLPES Executive Committee will be as follows.
- Kristen Rottinghaus, Kan. (who moves up to chair automatically)
- Kiernan McGorty, N.C. (who becomes immediate past chair)
- Jeannine Brown, Georgia.
- Drew Dickinson, Virginia.
- Emily Johnson, Texas.
- Jason Juffras, D.C.
- Mary Jo Koschay, Michigan.
- Paul Navarro, California.
- Jennifer Sebren, Mississippi.
- Will Soller, Montana
- Eric Thomas, Washington
- Darin Underwood, Utah
The term for the 2021-2022 Executive Committee will begin this fall, in September or October. The new vice chair and secretary will be selected at the Executive Committee’s first meeting.
NLPES Goes Social
NLPES is social! As you may have seen already, the Executive Committee established a LinkedIn group exclusively for NLPES members. We are excited to offer another way for offices to connect, exchange ideas, and share information relevant to our field. Maybe you are a listserv member but find those emails have a way of disappearing or you’re like me and struggle with an unwieldy inbox (2,100 messages and counting…)? Or maybe you’re just young and hip and tire of us old folks relying on email as the primary means of communication? 😉 Well, then we hope you’ll join the group! Announce job openings, share media stories about your office’s awesome work (because of course the work we do is awesome!), and just generally stay connected when doing so seems increasingly tough. For more information about the group and how to join, please visit here.
NLPES PDS Is Back
After having to cancel its main event in 2020 due to the pandemic, the NLPES Executive Committee is happy to bring back the NLPES Professional Development Seminar (PDS)! The PDS will be held in two months, on Sept. 28-30, 2021. For the safety of staff, and to make participation easier and less expensive than the usual in-person conference, the 2021 PDS will be convened virtually.
The conference offers 12 hours of learning on a variety of topics. That’s 12 hours of continuing professional education credits for Yellow Book agencies!
Among the highlights:
- Jon Schwabish, a national expert and Senior Fellow at the Urban Institute, will provide training on each of the three days of the conference on how to improve your data visualizations.
- Rohit Naimpally, senior research and policy manager at the North American office of the Abdul Latif Jameel Poverty Action Lab (J-PAL) at MIT, will talk about his organization’s work with states to strengthen the evidence on the effectiveness of state education, health, and other programs.
- Nick Kittle, a nationally acclaimed speaker, will explain the nature of innovation in government and how to sustain it.
- Scott Paine and Schwanda Barnette from the Florida League of Cities, will teach participants how to have meaningful conversations about diversity, equity, and inclusion.
The conference also will include sessions from your peers in Oregon, Kansas, New Mexico, and Washington, D.C. on using data analytics to fight fraud, producing clearer reports with simple writing, and strengthening your agency’s public profile.
But wait, there’s more! All of this learning is only $150 per person for all three days and 12 hours, which is less than half of the typical registration fee for an in-person PDS.
Join your colleagues (virtually) at this one-time only event. Register now!
Websites, Professional Development and Other Resources
Now available: Who We are and What We Do: A National Survey of State Legislative Program Evaluation/Performance Audit Programs—What’s the most common internal performance measure for evaluation shops? How many offices tweet? What percentage of staff has fewer than 10 years of experience? How can you contact a sister office in another state? Find out in Who We are and What We Do, formerly Ensuring the Public Trust,which provides descriptive information about the legislative offices that conduct performance audits and program evaluations.
Report Library—The NLPES Audit Report Library is a topical listing of recent reports that may be of interest in your work. The library pilot went live in the summer, and based on the positive response we received to it, we are working to expand the library from just those reports released by the agencies represented on the NLPES Executive Committee to include reports from all of member agencies. To facilitate the expansion as well as the ongoing posting of reports, the Executive Committee has reached out to the member offices’ directors to appoint Report Library Liaisons to ensure these efforts. The library will include a topical listing of all reports released by member agencies beginning in 2018. Because of some data limitations, it is not a database and the reports will need to be accessed through the individual agency’s website, which is listed with the report name in the library. In addition, the Executive Committee is reviewing the suggestions for improvements to the library received after the library went live. Ideally, we hope to work toward a more user-friendly, high-tech version of the library in the future but in the meantime, as the library continues to grow and you use it more, if you have suggestions for improvements, please send those to Emily Johnson. The goal is to make the library useful to our members and, therefore, it will continue to evolve based on the needs of our members.
NLPES listserv—The NLPES listserv is an email discussion group for NLPES members. Authorized listserv users can query other states about evaluation work, receive announcements about performance evaluation reports and job opportunities from other states, and are notified when the latest edition of this newsletter is available! To join the listserv, send an email to Brenda Erickson, NCSL liaison to NLPES, with the subject “SUBSCRIBE to NLPES Listserv.” Include your name, job title, legislative audit agency/organization name, mailing address (including city, state, zip code), phone number and email address. A “Welcome” message will be sent to you once you are successfully added to the listserv. See the listserv link on the NLPES website for additional information on how to post messages and “netiquette” niceties, like not hitting “Reply All” when answering a query posted to the listserv.
Website staff photos—Take a look at the NLPES website homepage for the new photo ribbon featuring some of your colleagues expressing why NLPES is important to them