 |
Home | Contact Us | Press Room | Site Overview | Help | Login | Register |
 |
 |
| About NCSL | State & Federal Issues | Legislatures | Legislative Staff | Meetings | Bookstore | Legislators & Staff Only |
| NCSL Home > Legislative Staff: Staff Sections & Networks > NLPES > | Add to MyNCSL |
NLPES NewsNational Legislative Program Evaluation Society Fall 1999, No. 74
Inside the NewsChair's Corner
Chair's Corner
One of the most significant accomplishments is the continued tradition of first rate, high-value Fall training conferences. This year's conference in Charleston, West Virginia was no exception. David Walker, the United States Comptroller General gave a thought-provoking keynote address urging each of us to think of ourselves as accountability professionals. The three thematic tracks seemed to provide something for everyone. And, the hospitality shown by Aaron Alred and his staff was top notch. They will be a hard act to follow. The use of technology has continued to influence and improve our ability to share information and exchange ideas. Electronic "robots" automatically update the Legislative Program Evaluations Database with new offering from member organizations. The new and improved NLPES web site is a great source of information and the most frequent destination among my bookmark favorites. Also, the listserve is being used more frequently to share information, despite our vast geographic dispersion. The newsletter has improved with each issue. The electronic version, announced via the listserve, is like getting doorstep delivery. All of this and more are available to us through NLPES, an organization that we are members of by virtue of the jobs we hold. It sounds like a pretty good deal, and it is! But alas, as I peek around the corner to the future, I see that there is more to be done. The executive committee will be defining a mission and performance measures for NLPES. There are questions about the completeness of the LPE database to be addressed. Interest has been expressed in providing different networking opportunities for very small shops. Processes to ensure that future training conferences and annual meeting sessions to meet member expectations need to be reexamined. These and other matters will be attended to by your executive committee during the coming year. On behalf of the membership, I want to thank Gary VanLandingham for his leadership of this organization during the past year. He set a good example for those that follow. I also wish to thank Kirk Jonas, Joe Fiala and Mary Noble. Their active involvement and leadership in NLPES have benefited all of us. As a final note, the executive committee serves you. Please let us hear your likes and suggestions for improvement. Your input will help shape the future of NLPES. Also, as with anything in life, the more you put in, the more you will get out. I have certainly found this to be true regarding NLPES. So, get involved. You won't be sorry. Muses from the Outgoing ChairIt has been a true honor to serve as NLPES chair over the past year. One of the greatest privileges has been to work with a superb group of colleagues - the NLPES Executive Committee. This group works very hard to plan and carry out NLPES' services to our membership, and I want to thank them for a job very well done. I am sure that NLPES will continue to excel in the very capable hands of its new leadership team -- Craig Kinton of Texas (chair), James Barber of Mississippi (vice-chair), and Gerald Hoppmann of Louisiana (secretary). They assure me that although they all come from the South, they will occasionally say "you guys" instead of "Y'all" in order to be inclusive. Both NLPES and the field of legislative program evaluation are facing a series of challenges resulting from the increasing pace of change that is impacting states and legislatures. New technologies, term limits, performance budgeting, tax and revenue limitations, and legislative demands for more information delivered faster are causing us to stretch and adapt in ways we could not even imagine a few years ago. NLPES can and must be a primary resource for legislative program evaluators in addressing these challenges. Simply put, we need each other's best ideas and success stories to successfully do the job of serving the keystone of democracy - our state legislatures. Just as NCSL describes itself as "The forum for America's ideas," NLPES needs to be the forum for exchanging information on what works (and what doesn't) in legislative program evaluation. This will require NLPES to reach far beyond the 10% or so of its membership who can come to the annual training conferences and reach all staff who work as legislative program evaluators. We need everyone's help in doing this. I want to suggest four ways that this can be done. First , read and share this newsletter with everyone working in a legislative program evaluation unit! Make copies and encourage people to read it via the NLPES web site. Submit articles on great things that are going on in your state and lessons learned along the way. The newsletter is only as good as we, the collective membership, make it. Think of what you would like other states to know about your shop, and write an article about it. Suggest that your colleagues in other states do the same.Second , use the NLPES web site! The web site is a primary tool for exchanging information on research, training ideas and opportunities. Although the number of weekly visits (versus "hits") to the site have increased from an average of three per week to over 300 weekly over the past year, only a minority of NLPES members (and states) regularly use the web site. Check out the research and training resource pages of the site if you haven't already. Bookmark the site, and show it to your colleagues and demand (or at least suggest) that they visit it regularly. Discuss the web site and its content at staff meetings to let everyone know about it, and include it in your office's orientation materials for new staff.Third , use the NLPES listserve! Most of us spend hours and days on finding out what other states are doing in a policy area. The listserve allows us to send out an electronic message to about 40 states and 500 colleagues, which is a lot faster than having to mail or call each state. The more people sign up for the listserve, the more useful it will be as a communication tool. Don't be afraid to use the listserve to ask a question; that's why it exists.Finally , give us your ideas! Send them in via e-mail to the executive committee or Bob Boerner, our hardworking NCSL staff contact, by phone call to your state's designated NLPES contact, or to the webmaster of the NLPES site. There is tremendous talent and energy among our member offices, and the NLPES Executive Committee certainly has no monopoly on ways to better share ideas and information. We work better when we work together, and making sure that NLPES succeeds as a networking and training tool is a job we all need to share.Office ProfileMissouri Oversight DivisionAbout Missouri: Missouri is a Midwestern state with the majority of its population residing in the two major cities, Kansas City and St. Louis. The cities are on opposite sides of the state, with the state capitol, Jefferson City, lying between them. Our legislature meets from the first of January to the middle of May every year. There are 163 representatives and 34 senators in the General Assembly. Missouri is referred to as "the Show Me State," which conveys the natural skepticism of its citizens. Naturally, that's why Missourians make the best auditors/evaluators! The southern part of the state, namely Branson, is growing rapidly due to the arrival of country music theaters, entertainment facilities and the Osmond family. The Lake of the Ozarks is another popular tourist spot. Missouri is also known for its production of mules, cows and corn.Office History: The Oversight Division was formed in 1984 by a statutory change that transferred the Office of Fiscal Affairs to the Oversight Division of the Committee on Legislative Research, along with adding responsibilities for program auditing. The Committee on Legislative Research, a 20-member permanent joint committee of the Missouri General Assembly, is the governing body of the Oversight Division. The Oversight Division has three main duties: the preparation of fiscal notes, the performance of program evaluations and the preparation of special reports. All assignments are made through the Committee on Legislative Research. The division's fiscal note work consumes about six intense months, while the evaluations and special reports are completed during the remaining six month interim.Staff: Oversight is staffed with 17 full-time, dedicated employees. They include 13 analyst/evaluators, three support staff and a director. Director Jeanne Jarrett is a certified public accountant and a certified government financial manager. Eight other staff members are CPA's. Oversight's staff also includes one certified fraud examiner, one certified internal auditor and three master's in public administration as well as several certified government financial managers. Most staff have experience in state agency financial management or auditing.Dream Assignment: The Department of Economic Development's overseas offices.Difficult Assignment: A program audit of the Missouri State Auditor's Office, which resulted in a lawsuit lasting two years. We lost and did not complete the "audit." We now refer to our work as program evaluations instead of "audits" due to the language in the Supreme Court decision.Favorite Agency Response to an Audit Question: "You just can't turn this ship on a dime" (when questioned about the timing of program implementation).Most Questionable Agency Statement: "We're so glad you're here; we view this as an opportunity to show the legislature what a great program we have."Embarrassing Moment: When two evaluators accompanied an employee of the state's milk board on a routine inspection of facilities. They were side-swiped in the milk board's state car (luckily no one was seriously injured) and had to hitch a ride home in a wrecker, state car in tow.Recent Reports: Federal Drug Seizure Fund, Department of Economic Development Tax Credit Programs, Caring Communities, Highway Patrol Motor Vehicle Inspection Program, Report on State Bonded Indebtedness, Unfunded Federal Mandate Report.A View From the StatesThe Increasing Need for Audit Shop Flexibility
Flexibility and versatility are key words that we all hear during discussions of how best to meet the needs of the legislature. As government becomes more technical and less comprehensible, policymakers look toward performance auditors and program evaluators to demystify state-run programs. However, these same policymakers also expect "traditional" performance audit and program evaluation entities to "turn on a dime" and provide other types of services as well. To remain a credible and informative source for the legislature, performance audit and program evaluation entities must accept the challenge and provide different services that may be new or uncomfortable. In addition, these entities must diversify their work without sacrificing the timeliness and credibility of more traditional products. Louisiana has experienced a strong demand for flexibility during the past couple of years. As a result of various legislative requests, the Louisiana Office of the Legislative Auditor has had to quickly prepare staff to respond to the various needs of the legislature on several occasions. For example, our office has been asked countless times to produce products on a variety of legislative requests, as well as performance-based budgeting, education issues, and some local issues. We also have been asked to develop staff expertise in areas not traditionally associated with performance audit or program evaluation. Our office was recently charged with a new legislative function beginning with the 1999 Regular Session. House Concurrent Resolution 121 of 1998 gave the auditor the responsibility of completing fiscal notes for bills that affect the expenditures and revenues of political subdivisions and state boards and commissions. Traditionally, the Legislative Fiscal Office in Louisiana has been responsible for completing all fiscal notes. However, because of the increasing number of bills in recent years, our office now shares the fiscal workload. To prepare for our new legislative function, we created a separate fiscal note unit which oversaw the installation of the necessary computer equipment. Our office also conducted both formal and on-the-job training for staff. By the time the 1999 legislative session began, we had created a fiscal note manual, trained approximately 27 staff, and equipped a fully functional fiscal note division. By the close of the session, our office had completed and transmitted almost 1,000 versions of fiscal notes for over 300 bills. Many legislators gave us positive feedback on the quality of our fiscal notes. While preparing fiscal notes, our staff also worked on ongoing performance audits to meet our statutorily mandated five-year plan. Responding in this evolving environment has served as a testimony to our staff and their ever-increasing need for flexibility and versatility within a performance audit or program evaluation entity. Program Evaluator InternationalLegislative Oversight South African StyleClick here to view a picture of Cape Town, South Africa. I am sure the phrase sounds better in French, but I think Napoleon once said "an army travels on its stomach." I am frequently reminded of that saying because most stories that program evaluators and auditors bring back from road trips revolve around food and restaurants. If that were the case with this story, I would attempt to wax eloquently page after page about the wonderful tastes of kudu, springbok, ostrich and the fabulous local wines. Unfortunately, this is the NLPES News, not an amateurish knock-off of Bon Appetit magazine. Consequently, the editor has sternly warned me to keep my comments to work-related information only. This past May, I had the opportunity to spend two weeks in the Republic of South Africa (yes, we saw then-President Mandela. OK, it was from about 100 yards) talking with legislators and staff about the concepts and mechanics of legislative oversight. Based on that research, we prepared a report to the National Council of Provinces (NCOP), the upper house of parliament, with recommendations on how it should implement its constitutional oversight responsibilities. Our project was commissioned by the chairperson of the NCOP, which contracted with a nongovernmental organization to assemble our three-person team of experts to do the work. The two other team members were a legislator from Arkansas who serves on the budget committee and a constitutional law professor from the University of Cape Town. Why does one of the houses of parliament have to ask how to do its job? The NCOP is a relatively new body, just created by the post-Apartheid constitution adopted in 1997. While the government in South Africa went through several forms during Apartheid, it generally functioned with a single house of parliament. Not only is the NCOP new, it is meant to give the provinces (most of which are also new) a voice in national government. Members of the NCOP are selected by each provincial legislature, much like members of our U. S. Senate were once selected by state legislatures prior to direct elections. Our task was especially daunting for several reasons. As I indicated, because the NCOP itself is relatively new, it is still developing a concept of how it wants to operate as an institution. Further, because the constitution was the result of numerous compromises between the old ruling white party, the African National Congress (ANC), other black parties, labor unions, and others, it has more than a few ambiguities. The constitution is so new that those who are charged with implementing it are still resolving these ambiguities and have had precious little time to piece together resolutions. Also, the NCOP is the weak sister in parliament since its actions on most legislation can be overridden by a two-thirds vote in the main chamber, the national assembly. Finally, the largest challenge is how to implement oversight in a Westminster parliamentary system where one party so dominates the others. While Great Britain has a long established tradition of "value for money" auditing under its auditor general, oversight in the parliament resides with the opposition party, rather than the legislative institution, as is the case in the U. S. The experience in Great Britain has been positive because of the frequency with which the majority shifts from one party to another. In South Africa, however, the ANC party is completely dominant (though the strikes occurring as I write this in late August may lead to some power shifting) and came within a few hundredths of a percentage point of gaining a two-thirds majority in the June elections, which would have allowed the ANC to make unilateral changes to the constitution. So, how were a state legislator, a bureaucrat and a law professor received by parliament members hardened by years of incredibly brutal and bloody struggle, facing the unbelievably difficult task of nation-building with unemployment over 40% and a majority of constituents living without running water in their homes? Surprisingly well. Christina, the resident law professor, had been relatively non-political during the 1990s and was one of a handful of legal experts chosen by the various parties during the constitutional convention to provide technical expertise in the drafting of the constitution in 1996. She seemed to be on close speaking terms with everyone in government and within the parties. As a result, we had great access, with warm receptions and immediate credibility wherever we went. However, even though Christina had a hand in much of the document, more than once while we three were struggling with some section describing the power and responsibility of the NCOP, she would exclaim, "Oh damn, what were we thinking then?!" Anybody for Madison and Hamilton? Who did we consult with? Lots of people. Speaker Patrick "Terror" Lakota, the presiding officer of the NCOP, outlined his vision for us. (The nickname "Terror" may have had something to do with the reason he spent a number of years in prison on Robben Island with Nelson Mandela. In fact, many of the ANC members with whom we spoke were graduates of what is jocularly referred to as Robben Island University.) We also spoke with the chairs of the budget and public account committees in the national assembly, various members of the NCOP, the speakers of several provincial legislatures, the chairperson of the ANC, various opposition party members, and parliamentary staff, as well as staff of the auditor general's office. Where did we go? Cape Town, of course, which is where parliament sits. We also traveled 1,100 miles to Pretoria, which is where the executive branch is located. (The Supreme Court sits in yet a third city!) Aside from Cape Town, the capitol of the Western Cape Province, we also visited two other provincial capitols: Johannesburg, the country's industrial and commercial hub and the capitol of Gauteng Province, and Nelspruit, the capitol of Mpumalanga Province. What is South Africa like? I have been asking myself that for the past several months since I returned. I found the country and culture incredibly deceptive. For the first few days following my arrival, I foolishly nurtured a pleasant and simple understanding of this country. English is the official language, and Cape Town looks sort of like Vancouver, lush and green with palm trees and beautiful views of the ocean and mountains. However, after several more days of reading about the rise and fall of Apartheid, traveling and talking with so many different people-black, white and colored-I began to discern how incredibly complex society and politics are in South Africa. Our nation-building happened two centuries ago, and we know of those long-ago struggles, the torn and complex loyalties, the sorting out of the winners and losers, the need to compromise and coexist from the vantage point of time and our history books. South Africans are embroiled right now in this painful process, and they do not know how it will turn out. Would I go again? In a heartbeat. The country is incredibly beautiful, with widely varied topography. The people are nice. There are wild animals in the game parks. The dollar is exceptionally strong against the rand and will likely stay so while gold prices remain depressed. [Answers to the culinary quiz. A kudu is an elk-like animal. A springbok is a bit smaller than an impala. The wine growing region around Cape Town is many times larger than Sonoma Valley in California and produces both reds and whites.] Liaison Review
The American Society for Public Administration (ASPA), established in 1939, is an association comprised of more than 10,000 practitioners, scholars, teachers, and students. ASPA has more than 110 local chapters representing 46 states, the District of Columbia, and the Virgin Islands. ASPA works to serve the public service community by striving to make government better, providing networking opportunities, and improving the image of public service. Mary Hamilton, ASPA executive director, is excited about the chance to exchange ideas and information with NLPES. She expressed interest in a formal exchange of publications between ASPA and NLPES. In addition, she suggested that NLPES members could potentially share their experiences and expertise with ASPA members through presentations and panels at future meetings. For example, the planned topics for the next ASPA National Conference may be of interest to NLPES members. ASPA's annual meeting will be held in San Diego, California, in April 2000. The conference plans to focus on six major topic areas that, not-so-ironically, seem to coincide with many recent legislative evaluation and oversight requests: (1) technology; (2) organizational and inter-organizational issues (e.g., strategic planning, budgeting, re-engineering, performance measurement and evaluation, intergovernmental relations, and public-private initiatives); (3) environmental and infrastructure (e.g., natural resource management, public works, transportation, and emergency and crisis management); (4) education, health and human services; (5) economic development; and (6) constituency issues (including empowerment, diversity, demographics, ethics, and advocacy issues). For more information, check out the ASPA web site. ASPA serves as an umbrella organization for practitioners in the general field of public administration, as well as a myriad of related subfields and disciplines. Recognizing the diverse interests of its membership, ASPA seeks to cooperate with organizations like NLPES to enhance learning and information sharing opportunities. When asked how ASPA and NLPES could cooperate on future ventures, Ms. Hamilton pointed to the Center for Accountability and Performance (CAP) as an area where the interests of the two organizations seem to overlap. ASPA established CAP in November 1996 based on the three-year effort of its Task Force on Government Accomplishment and Accountability. The task force recommended the CAP be created as an ASPA standing committee or "center" to address the recent emphasis at all levels of government on performance-based, results-driven management. Since that time, CAP has been disseminating information concerning the value of performance measurement and identifying national and international best practices in public sector performance management. CAP also provides members with opportunities to acquire the requisite knowledge and technical skills by developing resource materials, presentations, workshops, and publishing workbooks, case studies and news articles. For example, a new CAP workbook, Performance Measurement - Concepts and Techniques, was published recently by ASPA. Practitioners trying to get up to speed on managing for results should find this workbook helpful and easy to use. In addition to the workbook, ASPA's Center for Accountability and Performance is currently engaged in a multi-year program to develop more than 50 case studies from all levels of government on performance management in government. At present, 37 case studies from local, state and federal government are available from CAP. For example, an article written by NLPES member Kirk Jonas of Virginia's Joint Legislative Audit and Review Commission, entitled The Development of Performance Measures for Virginia Financial Management and Oversight, has been included among CAP's case studies. This article is one example of the interdisciplinary nature of and cross-cutting connections between the interests of ASPA and NLPES. ASPA communicates its work through two main publications: a bimonthly scholarly journal, Public Administration Review (PAR), and a newsletter, the PA Times. In addition, the ASPA web site has become a major tool for communication between ASPA and its members. PAR serves as an information source for practitioners, scholars and academics, teachers, and others interested in the public sector. Written by experts in the field, PAR articles, columns and book reviews cover many aspects of public management at all levels of government, including TQM, strategic planning, ethics, information technology, and organizational culture, among others. The PA Times serves as ASPA's monthly membership newsletter. Along with regular sections such as chapter news, upcoming events, and recruitment, the PA Times features timely articles on topics that impact the field of public administration. For example, recent issues contained the following titles: Best in the Business: Improving Customer Service in Government; Leading With Less: Resource Realities and Critical Choices in Public Administration; and Reforming Medicaid Through Managed Care. The ASPA web site contains a great deal of information about the national organization as well as local chapters and sections. In addition, the web site helps keep members in contact with one another through a page where members can post requests for information from other members. All CAP case studies are available for $6 each (includes P&H). To order, call ASPA at (202) 393-7878. Summaries of case studies may be found on the ASPA web site. Both the PA Times and PAR can also be found on the site. The PA Times online features selected articles and announcements from recent issues of PA Times, ASPA's monthly membership newspaper. Like the PA Times, PAR has an online presence where one can search for previously published PAR articles and book reviews, review past issues in the PAR "archives," or even submit an article. ASPA may be reached by phone at (202) 393-7878, by fax at (202) 638-4952, or by e-mail. News From the NetThe Power of a Weekly NewsletterWould you like to increase the visibility of your office's publications while providing a valuable service to other policy analysts, legislative staff, and legislators? Are you interested in providing others with electronic access to publications that are relevant to your home state? Do you want other policymakers and evaluators to alert you to cutting-edge, policy-related publications? If the answer to any of these questions is "Yes," then the Florida Monitor Weekly (http://www.oppaga.state.fl.us/weekly/) may provide you with an excellent model for launching a weekly electronic newsletter. The Florida Monitor Weekly is a free source for brief announcements of research reports, policy papers, web site launches, and other resources of interest for policy research and program evaluation. The Weekly features publications of Florida's Office of Program Policy Analysis and Government Accountability (OPPAGA) along with the reports of other Florida legislative offices, state and federal government agencies, think tanks, and media organizations. For examples of past Weekly editions, click on the "Archive" link on the Weekly home page. Since it was launched in October 1998, the subscriber list for the Florida Monitor Weekly has grown to 462 subscribers (not including OPPAGA analysts). On average, 50 new subscribers join the list each month. Subscribers include legislators, legislative staff, policy analysts, government employees, college/university faculty, students, and journalists. Although most subscribers are from Florida, the list has attracted subscribers from 16 states and six countries. We even have a subscriber from the land of sheep and kiwi -- New Zealand! This rapid expansion in readership can be attributed to the Florida Monitor Weekly's many benefits, including Disseminating timely, relevant, and useful information. The publications described in the Weekly provide in-depth consideration of current policy issues. Legislative staff have found Weekly content to be highly useful.Keeping analysts up to date with current policy issues. Given today's workloads and time constraints, it is not always easy to stay on top of new policy issues. The Weekly provides "one-stop shopping" for concise information from a variety of sources.Publicizing organizational reports. The Weekly provides OPPAGA with an inexpensive and easy way to notify others about its newest publications. Although OPPAGA posts all new reports to its web site, the Weekly allows users to access new publications without having to continually check the web site.The development of the Florida Monitor Weekly is a relatively simple process that isn't time or labor intensive. Resources for each issue are gathered by searching relevant web sites, joining listserves, and subscribing to other organizations' electronic newsletters. Subscriber submissions are also encouraged. Once resources are deemed appropriate for the purpose of the Weekly, entries are developed, formatted, and proofread. When an edition is completed, it is distributed to subscribers via an e-mail message. Less than six hours of staff time per week is needed to compile resources and format the Weekly for distribution and archiving. There are a number of companies that can manage your subscriber list, often for only a minimal annual fee. OPPAGA uses Microsoft Network (MSN) LinkExchange ListBot (http://www.listbot.com/cgi-bin/gold) and has been very pleased with the service. MSN LinkExchange provides confidential spam-free services, trouble-free subscribing via a web site, and an archive that contains every issue of the Florida Monitor Weekly. The company also gathers and maintains demographic information about subscribers, providing a snapshot of the people who are using the service. With a limited investment of both time and money, your office can launch its own version of the Florida Monitor Weekly and start reaching out to the masses in its home state and beyond. For more information about creating an electronic newsletter, contact Kara Collins-Gomez (collins-gomez.kara@mail.oppaga.state.fl.us), or call (850) 487-4257. To subscribe to the Florida Monitor Weekly, visit the subscription web site (http://www.oppaga.state.fl.us/weekly/).
July NLPES Executive Committee Minutes SummaryFull minutes available on-line (with hyperlinks) July 24, 1999 Web Site NCSL has redesigned its web site. NLPES will need to revise its web site to conform to the changes made by NCSL. NLPES will probably need to move its web site from Florida OPPAGA back to NCSL in the near future. The NCSL has developed a compact disc that can be used as an orientation tool for new legislators and legislative staff.Committee Reports Awards. 1999 award recipients were announced: Washington State JLARC received the Excellence in Legislative Program Evaluation Award; Judith E. Frye received the Outstanding Contributions to Legislative Program Evaluation Award; Florida OPPAGA received Special Recognition for Accountability Reporting; and 11 states received Certificates of Recognition of Impact.Liaisons. NLPES members have been appointed to serve as liaisons to all of the professional organizations with which NLPES associates, except the Association of Government Accountants (AGA). Names of members who may be interested in serving in this capacity were requested.Nominations and Elections. There was a sufficient number of candidates to fill the six vacancies on the NLPES Executive Committee without holding an election.The Executive committee agreed that executive branch staff could participate in NLPES activities, but actual membership should be limited to those individuals employed by a legislature or an entity associated with a legislature, as stated in the NLPES bylaws. Training. Kansas City, Missouri will be the site for the 2001 fall training conference, with the conference being co-hosted by Kansas and Missouri.The training committee continues to work on a handbook for host states to use in planning a fall training conference. In addition, the training committee is updating its training directory. ********** July 28, 1999 Election of Officers The executive committee elected James Barber (MI) and Gerald Hoppmann (LA) as vice-chair and secretary, respectively. Appointment of Committees The chair appointed the following committees for 1999-00.
New Business Gary VanLandingham will take over as staff chair for the Assembly on State Issues (ASI) Fiscal, Oversight, and Intergovernmental Affairs Committee. The executive committee voted to authorize NLPES to participate with this committee to develop presentations for training sessions during the spring and winter ASI meetings as well as to develop a joint session during the next NCSL annual meeting. He also discussed the benefits of conducting a survey of the 50 states to determine what states are doing in the area of legislative evaluation and oversight. A proposed NLPES research study with the Texas State Auditor's Office (SAO) and the LBJ School of Public Affairs was discussed. LBJ would like to work with SAO to conduct a research project that deals with the differences and similarities between performance auditing and program evaluation. For example, the study could focus on how program evaluations and performance audits have helped make policy related to various social issues or to identify best practices for social programs. NCSL Annual Meeting NLPES photos Click here to view a picture of NLPES Executive Committee members. Click here to view a picture of Gary VanLandingham addressing NLPES luncheon attendees. Click here to view a picture of NLPES devotees at staff section reception. Training: Program Evaluation 101NCSL Skills Development SeminarJim Pellegrini (MT), Max Arinder (MS) Each year the National Conference of State Legislatures holds a seminar for new legislative staff. It is called the Skills Development Seminar. NLPES is one of four staff sections involved in the seminar for new staff or staff with new responsibilities. NLPES sponsors the Program Evaluation Track at this seminar and recruits and uses a couple of "senior" legislative staff to serve as trainers. That would be us! Looking out at the faces of the new staff who attended the last seminar, August 25-28 in Boulder, Colorado, we can say that we indeed are senior legislative staff. But with our longevity comes knowledge and experience. It comes in handy during the day and a half of discussion. It also comes in handy on knowing when to come in at night - so you are ready for the next day. The training seminar itself is used to enhance the skills of the participants. The participants use a combination of lecture, hands-on case studies, questions, and discussions to develop an understanding of program evaluation. Our general approach is to describe program evaluation as a problem-solving process. In teaching the seminar, we realize that no two states may be alike in their approach to evaluation. However, there is a common ground for all evaluations. We note that evaluation is to be systematic. Evaluation involves a continuum of activities from careful, accurate description of a program or process to determining the worth or merit of some program or activity. What makes it hard to generalize about evaluation is that there are no rigid rules. Evaluation is often creative design, and there is no ideal standard. We use the term "disciplined inquiry" to help explain this unique aspect of program evaluation. In its basic concepts, program evaluation is determining "what is" and comparing it to "what should be." The evaluation process gets us from where we are now to where we want to be. The seminar materials revolve around actual legislative requests for program evaluations. We look at how evaluations are brought about. We look at and stress the background research that is necessary. We work through the survey and planning stages and briefly design an evaluation. Seminar case studies illustrate where evaluations begin, how we plan our work, and how we must think about our objectives in each stage of performing an evaluation. Each year we emphasize something different. This time around we provided a little more emphasis in the area of evidence - explaining the difference in the levels of assurance we get from interviews vs. documentation vs. analysis. We stressed the importance of doing analysis and using hard facts and documentation. This provides for a better problem-solving tool - not having to rely entirely or for the most part on interviews. Questions from seminar participants are essential. The new staff members are trying to get an understanding of what they are or will be doing. We relate our experiences and also the experiences of other states. The use of case studies helps us focus the seminar work and provides the participants with concrete examples. We stress communication as an important part of the seminar process. Those attending the course work in teams. They go through the case studies as team exercises. They discuss different issues and different approaches to the issues presented. There are several problem solving exercises were they work in teams to come to a logical solution to a problem. Later in the seminar we discuss as a group what to do with all the information that we've gathered. We always talk in terms of our basic problem-solving paradigm. We stress the importance of what makes a good conclusion and finding. We talk about keeping the purpose of our study in mind and relating back to the underlying objectives. It is important that we determine the real problem, and we stress that. Bringing together individuals from different states and backgrounds is essential to the success of the seminar. These people interact, ask questions of each other, and provide practical information. The contacts they make are invaluable in future evaluation work. The "elders" who stand at the front of the seminar room are the facilitators for the group. We provide our experiences and knowledge. The participants can use that to ask questions and gain insight into what they are doing. What makes this seminar very beneficial is that we are not talking theoretically - we are talking from practical examples with practical problems and practical solutions. Once the participants understand that program evaluation is an art as well as a science, they understand the important role they play in program evaluation. That's because the most important part of program evaluation is the person doing it. These new staff members are bringing to the table a toolbox full of different skills. We try to relate how they can use that toolbox in doing a program evaluation. The easiest way is to relate it directly to problem solving, because that's what we are - problem solvers. And it never gets old. Maybe that's why we're elders! The Elders - Jim Pellegrini and Max Arinder NLPES Award Winners1999 NCSL Annual Meeting NLPES Awards LuncheonEach legislative program evaluation or audit office has as its primary goal the creation of a quality product in a timely manner. Each of us spends many hours "in the trenches" in pursuit of that lofty goal. Periodically, we all need a 'pat on the back'-a reassuring 'job well done.' NLPES created its award program as a tangible means to provide that positive affirmation of worth that we all deserve at one time or another. Since 1982, NLPES has recognized the work of at least 19 of its member states by presenting them with excellence in research methods or legislative program evaluation awards. Other states have received certificates of recognition for the impact their work has had within their respective states. Today, we will continue the tradition of recognizing the work of our member states. Thanks to the individuals who gave of their time and effort on selection subcommittees to select the 1999 award recipients. These included the following individuals: Mary Noble (CA), Don Bezruki (WI), Gerald Hoppmann (LA), Lois Sayrs (AZ), Mary Stutzman (FL), Rob Krell (WA), Heather Moritz (CO), Gary VanLandingham (FL), Craig Kinton (TX), Max Arinder (MS), and Roger Brooks (MN). Certificates of Recognition of Impact Probably the greatest affirmation of our work comes when legislators or agency personnel utilize information and recommendations contained in our reports to improve government. In an effort to highlight such efforts, NLPES created the Certificate of Recognition of Impact. Certificates were awarded to 11offices and highlight the reports that resulted in improvements within their respective states. The following were the winners.
Arizona Office of the Auditor General William Thompson, Director, Performance Audit Division This report addresses ways to maximize revenues generated from state trust lands administered by the State Land Department, particularly those lands located in urban areas. The report recommended that the Department adopt a more proactive approach. ********** Performance Audit: Division of Safety, Florida Department of Labor and Employment SecurityFlorida Office of Program Policy Analysis and Government Accountability John Turcotte, Executive Director The Florida State Consultation Services Unit was not complying with state law, was not effectively using state assets, and was not significantly impacting private sector safety. Among other things, the report led to reorganization within the Department that included the elimination of 34 full-time staff positions. ********** Program Evaluation: Drug Rebate ProgramGeorgia Department of Audits Paul Bernard, Director, Performance Audit Operations Division The Georgia Drug Rebate program's billing system produced invoices that included large and obvious errors, and that its accounting systems produced records that were untrustworthy. The Department of Audits estimates that average quarterly collections have risen over $5.6 million (41%) following implementation of its recommendations. ********** Management Audit: Tuition and Fee WaiversIllinois Office of the Auditor General William Holland, Auditor General This report received extensive newspaper coverage, and legislation to implement audit recommendations is pending. Full implementation of the audit's recommendations would result in savings of more than $8 million at just one university. This report found that in 1995-96, nearly 41,000 tuition waivers worth $117 million were granted. ********** Report: Department of Mental Retardation Including "Are You Sure About This Guy?" Massachusetts House Post Audit and Oversight Bureau James Tansey, Director Provision of care by the Department of Mental Retardation was found to be substantially impaired. This report received significant media attention. Recommendations for program improvement have already been implemented. ********** Performance Audit: Charter Schools Office and Michigan Resource Center for Charter Schools, Central Michigan UniversityMichigan Office of the Auditor General Thomas McTavish, Auditor General Numerous problems were found in the oversight of charter schools by Central Michigan University. The audit received substantial public recognition, including being cited in a report on ABC's Nightline. Numerous operational and programmatic changes have been made as a result of the audit. ********** Child Protective Services Minnesota's counties have widely varying practices regarding child protective services, and there is confusion about requirements related to the reporting of suspected maltreatment. Many child care professionals lack confidence in the system's ability to intervene effectively in children's lives. This report received substantial media coverage, and the legislature has passed laws during the last two years to enact many of the report's key recommendations. ********** Management Review: Department of Natural ResourcesSouth Carolina Legislative Audit Council George Schroeder, Director Although Department of Natural Resources funding was being used in accordance with the purposes expressed in law, this review showed that financial management needed improvement. The audit found that some department-wide problems might be avoided by establishing an internal audit department. The report contained 55 recommendations to improve operations at the Department. ********** A Review of Oversight for the State's Embedded Systems Year 2000 Repair Efforts Many Texas entities were found to not have contingency plans in the event their computer embedded systems failed, funding was a problem, and no single entity was responsible for coordinating and reporting on statewide repair efforts. The project significantly increased awareness, and led many agencies to implement processes to help ensure Year 2000 preparedness. ********** An Evaluation of the Petroleum Environmental Cleanup Fund Recommendations were made to improve the efficiency and effectiveness of the Wisconsin Petroleum Environmental Cleanup Fund program and to increase monitoring. This report received wide media attention, and a bill to implement many of its recommendations has been voted out of committee and is currently pending action in the Senate. ********** Crime Victim Services The structure of the Wyoming crime victim services, which was split between two entities, was found to be ineffective. This review recommended that the legislature consider consolidating the state level administration of victim services. In response, a bill requiring consolidation passed in 1998 with an immediate effective date.
The Florida Government Accountability Report The Awards Selection Committee was especially impressed with the uniqueness and innovation of one submission and believed that it should be recognized in its own right with a special award. A Special Recognition Award for Accountability Reporting was bestowed on the Florida Office of Program Policy Analysis and Government Accountability and its Florida Government Accountability Report. OPPAGA created the Florida Government Accountability Report, or FGAR, in December 1997. FGAR is a massive web-based encyclopedia that contains profiles that cover each of Florida's 400 major state programs. FGAR is designed to present real time information to legislators who are often unwilling or unable to wait until a traditional program evaluation is conducted before making policy decisions. FGAR has received many positive comments from legislators and their staff members since its creation. Frequently, legislative staff copy FGAR material directly into bill analyses. New legislators and legislative staff directors have used FGAR profiles to quickly become familiar with the policy areas within their jurisdiction. Since its premier in December 1997, FGAR's tracking system has logged more than 110,000 visits by legislators, their staffs and the public.
Washington State Joint Legislative Audit and Review Committee The criteria for the Excellence in Legislative Program Evaluation award states that the award will be presented to the office producing a printed report or innovative alternative product which utilized exemplary means to provide information to the legislature. The mission of Washington JLARC is to ensure that its work is responsive and useful to the legislature's priorities and issues. To accomplish this mission, JLARC has developed tools to communicate audit results to public policymakers. Realizing that legislators have limited time to read an entire audit report, JLARC develops a one-page report summary for each study. The goal of each summary is to convey the most important information concerning a report in a concise and easy-to-read format. In addition to report summaries, JLARC staff make visual presentations to legislative committees through the use of state-of-the-art presentation technology. Audio recordings of some JLARC presentations can be accessed through an Internet location. In addition, JLARC regularly produces newsletters and summary reports that update legislators on current developments and impacts of JLARC audits and studies. These efforts allow JLARC to summarize concisely the direction, content and results of a wide range of studies, as well as highlight fiscal impacts of some of the agency's major efforts. JLARC's Impacts Report for 1998 states that the agency's 1998 performance audits, program evaluations, sunset reviews and other studies identified possible biennial cost savings of $26 million. NLPES commends Washington State JLARC for its years of service to the Washington State legislature and its contributions to legislative program evaluation.
Judith E. Frye (WI) Like all successful endeavors, NLPES has had a rich heritage of trailblazers, those individuals who have set the pace, and who have tirelessly given of themselves to establish high standards for our profession and organization. A special recognition award is presented to Judith E. Frye for her contributions and service to legislative program evaluation and to NLPES. Judy holds a Bachelor of Arts in Political Science from Kent State University and a Master of Arts in Political Science from Michigan State University. Judy began her career in legislative program evaluation in 1979 as an employee of the Wisconsin Legislative Audit Bureau. During most of her nineteen years with the bureau, she was the senior manager of the Program Evaluation Division, which was responsible for performing evaluations and audits of all major and many smaller agencies of state government as well as various local agencies. During her tenure with the bureau, Judy directed projects in the areas of human services, education, transportation, revenue, natural resources, local government, and legislative management. At the time of her resignation from the bureau in 1998, Judy was the Assistant State Auditor for Program Evaluation. In addition to her service at the Wisconsin Legislative Audit Bureau, Judy has been an integral part of NLPES and NCSL. From 1984 through 1986, she served as NLPES chair and represented NLPES on NCSL's Legislative Staff Coordinating Committee. She also served in leadership positions on NCSL's Fiscal Affairs and Oversight Committee from 1991 through 1993. Judy shared her knowledge of legislative program evaluation by serving as a faculty member of NCSL's Skills Development Training Seminar for more than four years. In 1990, she was a member of the inaugural class of NCSL's Legislative Staff Management Institute held at the University of Minnesota. Judy shared her legislative program evaluation skills internationally by participating in NCSL's 1993 Partnership of Parliaments project with Germany. In November 1998, Judy accepted a position with the Wisconsin Department of Health and Family Services. It is with deep appreciation that this special recognition award is bestowed on Judith E. Frye for her many, many contributions to legislative program evaluation and NLPES. Ask Eva, Lua & TorQuestion: Are workers who will be dealing with the auditors told to be honest or are they cautioned about not divulging family secrets? Eva, Lua & Tor Comment: No Prodding from Above to "Get it Right." None of our guest columnists has experienced any direct guidance or coercion from above to maintain the "party line" or avoid certain issues. On the contrary, the usual instructions to staff were to be honest and to work cooperatively with audit staff. In one shop, staff members are simply directed to keep copies of documents provided to auditors as references to identify possible conflicting information that might appear later in the report. However, some subtlety of approach is required because the staff members you interview range from those well seasoned by numerous state or federal audits to those who just found out they were being evaluated after reading about it in the newspaper. Keep in mind that agencies are different, and the meanings of cooperation and honesty can vary among them. Lua cautions, "Please don't assume that the executive branch is a monolith and that everyone acts the same way in the same type of situation, because they don't."Personal Protection, Internal Motivations and Judicious Truth. Although there may be no direct motivation to "spin the truth" or keep items confidential, there are justifiable bars to full, honest disclosure. In one executive agency, employees wondered whether their "honesty" would result in real changes or retribution. Some questioned whether audit follow-up would take place to ensure that recommendations would be implemented, while old hats weighed past "strong recommendations" that had faded into meaninglessness. In this atmosphere of disillusionment, some employees did not think the risk to their jobs for their "honesty" was worth a future of little change or improvement.In every program there is much information flowing from staff to top management describing programmatic and management shortcomings. In fact, as Lua points out, "One of the advantages [to doing evaluations within the executive branch] is people actually volunteer a lot of information. They see you as working with them and not against them. I am sorry to say that, no matter how many times an external evaluator says he or she is there to help, this message is not always believed." Eva's concern with being too candid with auditors is a perception issue. If a critical report, beneficial for internal use, suddenly becomes an audit issue, " . . . management might feel that I am disloyal, destructive, or not a team player. In many cases, management could easily determine that I am the source of negative information, or they could narrow it down to the usual suspects." It's not being dishonest to answer indirect questions with similarly indirect responses. If you, as an evaluator, are on a "fishing expedition" and do not know exactly what you want, don't expect the average agency staff person to map everything out for you. Some may choose to point you in the right direction. Others may not. Eva explains, "I can easily be evasive with the auditors unless they ask very precise questions, leaving no way to wiggle out. It is easy to answer a question honestly but not provide important details. If 'X' is asked when 'Y' would be more appropriate, am I going to answer the 'Y' question? No, I answer what is asked of me." Three Kinds of Honesty. Versions of the truth abound, depending on the position of the staffer being interviewed. Tor submits the following three categories of employee honesty:(1) Employees bitter over perceived administrative abuses. These employees provide unsolicited "family secrets" that may be more sour grapes and rumors, than vital audit findings. An auditor might perceive these revelations to be the ultimate in honesty but should beware and look for evidence to support the revelations. (2) Employees as loyalists. These staff feel that past administrative decisions, which are being questioned by the audit, were right. Honesty from these employees might involve proving their point that past decisions were correct. (3) Employees providing the simple truth. The majority of staff members will answer audit questions with simple facts, answering the questions but not necessarily giving more than what was requested. Most employees do not want to air the dirty family linen before strangers, feeling a need to be forthright but a bit protective at the same time. Turnabout is Fair Play. Eva asks evaluation shop directors and their team leaders to put themselves in the place of an outside review and, " . . . you suspect one of your staff is volunteering a little bit too much 'inside' information. What would you say or do to that employee? Or do you encourage being 100% candid with outsiders?"Lua suggests exploring the various shades of honesty with your own shop and how you might explain the evaluation process to an outside reviewer who's asking increasingly uncomfortable questions about the timeliness of your reporting process.
Developing Standards for Student EvaluationsJoint Committee on Standards for Educational Evaluation The Joint Committee on Standards for Educational Evaluation is an international organization with 16 sponsoring organizations, including NLPES, from the United States and Canada. Its primary role is to promote sound evaluations in educational settings through development and use of standards for educational evaluation. To date, the Joint Committee has developed two sets of standards: The Program Evaluation Standards, published by Sage Publications, Ltd., and The Personnel Evaluation Standards, published by Corwin Press, Inc. It is currently developing a third set that focuses on student evaluations. The Joint Committee is a member of the American National Standards Institute (ANSI). As such, standards developed and approved by the Joint Committee become American National Standards. The Joint Committee is the only organization that develops ANSI-approved standards for evaluation in education. The Joint Committee met for its annual meeting September 30-October 2, 1999. The major objective of the meeting was to review and revise preliminary drafts of the Student Evaluation Standards to complete a "first draft" of these standards. As a result of the Committee's three days of review and critique work, 28 standards were prepared in first draft form. These include seven standards on evaluation propriety, seven standards on evaluation utility, three standards on feasibility of evaluations, and 11 standards on evaluation accuracy. Preliminary draft information is available at the Joint Committee web site. The first draft form of the standards will be available on that site in November 1999. These draft standards will now be compiled, edited, and submitted to both a National Review Panel and an International Review Panel, each comprised of approximately 60 persons. Each panelist will be asked to critique the first draft by applying a number of criteria and to offer suggestions for improvement. Prior to the Joint Committee's September 2000 annual meeting, project staff will collect and summarize the results of the reviews. Based on this information, the Joint Committee, in a working session, will develop a second draft, a field-test version (semifinal draft) of the standards, and approve plans for field tests and public hearings. As a sponsoring organization of Joint Committee, NLPES is required to nominate individuals to participate in the panel reviews of the draft student evaluation standards. NLPES volunteers will have the opportunity to provide specific feedback in areas that include: the need for the standards; responsiveness to concerns in the field; scope of the standards; validity of their advice; practicality, political viability, legality, clarity, and depth of treatment; and appropriateness of language. Those interested in participating in the panel reviews should contact David Summers, NLPES' representative, at (800) 531-2477. Additional information about the field tests will soon be available at the Committee's web site. In other work at the conference, the current program evaluation standards (The Program Evaluation Standards) were affirmed in their present form for an additional three years. Following the 1999 annual meeting, ANSI was notified of the Joint Committee's intention to revise and seek ANSI approval of The Personnel Evaluation Standards. As part of its programmatic work to ensure that both sets of standards remain current, a task force of the Joint Committee will solicit input from all sixteen sponsoring organizations regarding use, viability, and any needed changes. News Makers
From the EditorIt was a wonderful surprise to be given an award for my work on this newsletter at the NCSL Annual Meeting this past summer. Click here to view a picture of Mark Bucherl accepting an award for his work on NLPES News. It also belongs to you contributors who take time out of hectic schedules to write articles that make this newsletter great. Allow me to pass the thanks on from the NLPES Executive Committee (need I say keep it up?). That said, as I write this Saturday evening, I feel like I'm digging out somewhat from the avalanche that has covered me recently from paid work, home work, and work work. You know how it goes. Thanks to former Indiana staffer Kristin Breen, who wrung any needless verbosity and knotty phrasing from several articles via our Indiana/Virginia 'net communications (I'm so glad you finally got your own computer there). Here, Dick Sheets put on a final clear coat of precision and standardization (which I've probably messed up with my nighttime tampering). If I didn't see you this summer in Indy-it was fun, wasn't it. Maybe I'll see you at the 2000 NCSL Annual Meeting Chicago or in Texas, for the NLPES meeting. Cheers. |
|||||||||||||||||||||||||||||||||||
© 2008 National Conference of State Legislatures, All Rights Reserved
Denver Office: Tel: 303-364-7700 | Fax: 303-364-7800 | 7700 East First Place | Denver, CO 80230 | Map
Washington Office: Tel: 202-624-5400 | Fax: 202-737-1069 | 444 North Capitol Street, N.W., Suite 515 | Washington, D.C. 20001
