• Mission & Governance
• Services Overview
• How to Get Involved
• Standing Committees
• NCSL Foundation

• Issue Areas: A-Z
• State-Federal Relations
• NCSL Standing Committees
• News from D.C. and the States

• About State Legislatures
• Elections, Campaigns & Redistricting
• NCSL's StateConnect Directory
• Web Sites

• Online Guide to NCSL Services
• Staff Sections and Networks
• Meetings & Professional Development
• Staff Directories

• Meetings Calendar
• Online Registration
• Meetings Home Page
• Legislative Summit

• Bookstore
• State Legislatures Magazine
• NCSL's StateConnect Directory

NLPES Site Map

NLPES News

September 1998, No. 71

Inside the News

Chair's Corner
Outgoing Chair
Office Profile: West Virginia Office of the Legislative Auditor
A View from the States

         Lessons from the Academy: Performance Auditing of
                    Universities
         Sunset of the South Carolina State Reorganization
                    Commission

July NLPES Executive Committee Meetings
News From the Net

          The Evaluator's Internet Toolbox
          The NLPES Listserve
          WIPS Data Base
          Web Site of the Issue: US Newspaper Archives
                     on the Web  

News Makers
NLPES Fall Training Conference: "Keeping Government on Track"
Small Shop Talk: Avoiding Employee Turnover
From the Editor
Calendar

Chair's Corner

Gary VanLandingham (FL)

I am honored to serve as NLPES chair for 1998-99. I want to thank Mary Noble for her outstanding leadership of NLPES over the past year and current and past Executive Committee members for their long hours of service. I also want to thank everyone who has participated in NLPES activities over the past year.

Over the next year, I hope to increase NLPES' ability to help the folks who work in the trenches doing legislative program evaluations. Our new Web site is a major initiative designed to make life easier by bringing together a wide variety of research resources. From this site you can search data bases of reports issued by NLPES member offices; search LEGISNET, NCSL's national data base of legislative research reports; link to good research Web sites; and send e-mails to people signed up for the NLPES listserve to ask for information and advice. Check it out! It will continue to be developed over the next year, so please give your ideas and content.

I also hope to develop a long-range schedule for future NLPES fall conferences. To make the conferences as accessible as possible to members, we plan to rotate the conference by region. West Virginia has generously offered to host the conference in 1999, so put a trip to Charleston on your planner for next year. We are now soliciting states to host future conferences - we need a Midwestern state to host in 2000, a southern state to host in 2001, and a western state to host in 2002. Conferences beyond 2002 will also rotate by region. If you are interested in hosting a future conference, please contact me or Joel Alter, chair of the Training Committee.

Over the next year, the Executive Committee will also work to update our by-laws and consider modifications to the criteria for the NLPES Excellence Award. Please contact James Barber, chair of the awards and by-laws committees, if you have ideas on these issues.

Finally, I want to encourage everyone to attend the NLPES fall conference in Sacramento. The California State Auditor Bureau of State Audits has planned an outstanding meeting; all that they need is you! Let's give them a record turnout.

P.S. -- Check out the new Web site!

Back to the top

Outgoing Chair

Mary Noble (CA)

By the time you read this, NLPES will have a new chair, but I did not want to miss an opportunity to talk to you one last time. First, I want to thank all the members of the Executive Committee for a stellar job of keeping the organization on track and moving forward. This process was somewhat complicated by the loss of three committee members during the year, causing us to ask some members to join or chair committees that they originally had not planned to. However, being the great program evaluators that they are, all were able to change directions very quickly. I am leaving the new Executive Committee in the very capable hands of Gary VanLandingham. Although we accomplished much this year, there are new challenges ahead. One that I am sure the new chair will agree with is to have more of you involved in NLPES.

I hope to see you at the fall training in Sacramento. We have some exciting sessions planned that will further our skills in several areas, including supervision. And of course I have ordered perfect California weather!

As many of you know, this has also been a challenging year for me personally, and I wanted to thank you for your good wishes and assistance throughout the year. It meant a lot and really helped me through some tough times.

Back to the top

Office Profile

West Virginia Office of the Legislative Auditor

The legislature created the Performance Evaluation and Research Division of the Legislative Auditor's Office in 1994. The performance evaluation division implements the West Virginia Sunset law and conducts special studies as requested or mandated by legislation.

The director of the Performance Evaluation and Research Division reports to the legislative auditor who reports to the Senate president and House speaker.

The division has 17 staff members: 10 research analysts, three research managers, three support staff, and a director. The educational background of the staff includes the following: economics, political science, business, liberal arts and accounting. Recent evaluation reports include child protective services, child support enforcement division, welfare reform, workers' compensation, foster care and adoption, state park system, and various licensing boards.

Recent Report Highlights

Board of Risk and Insurance Management. The Board of Risk and Insurance Management (BRIM) was basing its premiums completely on loss history instead of a combination of loss history and risk exposure. Data were available to measure risk exposure, but were not being used. Consequently, premiums were potentially insufficient to cover actual losses incurred. Furthermore, BRIM is required by law to appraise the value of state property.

Child Protective Services (CPS). A review of 663 CPS cases found that in 46 percent of the cases, CPS had no record of having face-to-face interviews with alleged victims of child abuse. Furthermore, only 29 percent of the cases had interviews within 14 days as required by law. Evidence showed that the cases without recorded interviews were never investigated based on memoranda revealing a statewide policy that prioritized the investigation of cases. A follow-up one year later showed that a statewide emphasis resulted in improvements, but the improvements were inconsistent for each county.

Child Support Enforcement. A sample of 340 child support cases revealed that one-third of the agency's cases with support orders were enforced incorrectly because of inaccurate information. The incorrect enforcement actions generally involved collecting the wrong amount from absent parents, or distributing the wrong amount to caretakers. Sample estimates showed that the agency wrongfully collected an estimated $1.7 million from 3,788 absent parents. In individual cases the amount incorrectly collected was thousands of dollars. An estimated 1,428 caretakers received the wrong amount of child support and were required to pay back the money, in some instances several thousand dollars. Also, the method of sending payments by paper checks was found to be inefficient compared with electronic funds transfer (EFT). EFT could reduce disbursement costs by as much as 80 percent a year.

Foster Care and Adoption. Six percent of court orders granting custody of children to the state were not signed by the judge. Due to this lack of signature and improper wording of the orders, the department of Health and Human Resources was at risk with possible legal entanglements. The loss of federal Title IV-E funding was estimated to be $5.1 million annually. There were 258 children legally free and available for adoption. However, the lack of adoption workers to solicit proper homes required the children to remain under the supervision of a child protective services worker. Basic information such as the status of care, where the child is located, and age, race and years in foster care could not be generated because of the lack of an information system to provide management with reports on the status of children. This results in children staying in foster care longer than may be necessary thus reducing their chances for a normal life.

Currently, the West Virginia Performance Evaluation and Research Division is working on the following projects: Adult Protective Services; Family Law Master's; Public Defenders Services; Purchasing; Soil Conservation Committee; and Workers' Compensation.

New Web Site

The West Virginia Performance Evaluation and Research Division is pleased to announce our new presence on the World Wide Web. The site includes a listing of our current projects, staff e-mail addresses, and a listing of all performance evaluations our office has conducted since its inception. We are in the process of placing our reports and their abstracts on the site. We welcome any suggestions for improving the site or any feedback on problems encountered during your visit.

Recently, the head of one agency saw two of our analysts approaching his office. He opened the second story window and shouted to a WV state trooper in the parking lot, "Hey, shoot the ugly one in the knee, those bunch of communists." The trooper replied, "Which one?"

One agency head said to one of our analysts after the worst electrical storm in recent history, "I had prayed that the storm had struck your office and wiped all of you out."

Having to ride anywhere with our director. Speed knows no limits while he drives. Once the legislative auditor saw one of our auditors at an out-of-town interim meeting. He asked the auditor why he looked so pale. The auditor honestly stated that he had been holding on to the "Jesus strap" in the director's car for 180 miles and had yet to recover from the experience. On another occasion the director passed two legislators twice in a 20-mile stretch on Interstate 79. The legislators only caught up to him, after he passed them the first time, because the director stopped at one of the interstate rest stops.

Back to the top

A View from the States

Lessons from the Academy
Performance Auditing of Universities

Lois Sayrs and Shan Hays (AZ)

This article is based on six audits of Arizona's three state universities, and discusses how auditors and evaluators can avoid some of the problems inherent in university work. Our work has included The Undergraduate Student Experience (focusing on graduation rates), University Administration, University Space Utilization, University Research Parks, University Auxiliary Enterprises, and University Enrollment Management.

There are enormous difficulties in auditing universities. However, performance auditing brings a new perspective, not influenced by institutional folklore and driven toward pragmatic solutions. Auditing challenges the status quo by exerting the pressure of independent oversight. Through our audits, we have helped to bridge the culture gap between what universities do and what the public perceives they should do. Although our early audits met with resistance, our latest efforts have been supported by university officials, and in fact, our office has been invited to participate in a Board of Regents committee to review privatization of university functions.

Legislative performance audits are tied directly to the public's concerns, many of which are obvious but easily overlooked or dismissed by universities in pursuit of a complex organizational mission. For example, why can't students graduate in four years? Are there too many administrators? Are new buildings needed to meet growing enrollment? Is enrollment going to continue to grow? Should certain university functions be privatized? These questions usually do not emanate from within the institution's walls. Nevertheless, they reflect the serious concerns of the public and the Legislature. Besides the taxpaying public, other population segments are often marginalized in university culture despite their close ties to the universities. Staff, untenured faculty, students, and parents of students generally have little voice in university decisions. Auditors can provide a voice to those marginalized groups. Auditors can also independently assess the merits of complaints made to legislators, so legislators can respond appropriately to their constituents without giving too much weight to the loudest voices.

When universities identify problems, institutional needs and biases drive their research and solutions. Thus, when we asked about poor graduation rates in The Undergraduate Student Experience, the universities' most frequent explanation was that students are poorly prepared for college work, an explanation based on anecdotal personal experience with individual students. Similarly, in the Space Utilization audit, the universities stated that even if overall space would be adequate if fully utilized, much of the existing space was inadequate for current high technology instruction.

Because performance auditing strives for practical recommendations -- GAO standards indicate that there is no finding without a recommendation -- we were prompted to step back from the institutional wisdom and probe further in an effort to identify more feasible short and intermediate range solutions than those proposed by the universities. For example, we found that even students with high aptitude test scores graduated at a lower rate than comparable students at other universities, leading us to conclude that all students could use help in graduating, and we suggested simple solutions such as offering better advising. When we evaluated university classroom space, we found that existing high technology classrooms were among the most underutilized spaces on campus.

Universities are usually the only source of information about universities. Regents and legislative staff have very little opportunity to gather information independently because of the time required, costs and complexity of the issues. As a result, strong oversight of the universities is very difficult. For example, Arizona's regents rely solely on universities to report, accurately and without bias, data on important issues such as graduation rates, recruitment of minority students, and faculty tenure decisions. In Administration, we generated separate data to identify how many of the universities' nearly 17,000 employees performed administrative and support work, as opposed to directly carrying out the three missions of instruction, research and public service. Our work directly challenged the results of earlier university-influenced studies, which had identified a very low proportion of administrative staff. In Enrollment, we debunked a widely published projection about the number of new students who would enroll in the system over the next 20 years.

Our independence was a powerful mechanism in challenging the status quo. The overriding result of Administration, and to a lesser extent the other audits, was an informed dialogue among the governor, legislature, Board of Regents and universities, leading to a much deeper understanding of university budgets and operations. The knowledge gained from our assessment gave them sufficient leverage to question universities' budgets and projections.

Universities have historically been successful at isolating themselves from the government mainstream. They set many of their own rules, without chafing under the restrictions that govern other state agencies' personnel and procurement practices. Building walls through support by various private sources, including sacrosanct university foundations (and their authoritative confidentiality requirements), allows the universities much freedom to spend money as they see fit. In Arizona, universities received only routine financial audits for many years, until specific legislation in 1993 directed the auditor general to begin conducting performance audits.

Isolated from state government, universities have also long been isolated from the mainstream of society. Wide gaps exist between what universities do and what the public perceives they do (or should do). Benchmarks are set by educational establishments or national associations without reference to the world outside academia. Influential members of the university work force arrive from other educational institutions, so no natural avenue exists for entry of noneducational ideas and changes.

Through performance auditing, we applied standards of practice, norms and values of mainstream American society to the institution. Simple, prevalent ideas that improve efficiencies in the nonacademic workplace are not necessarily standard in the universities. For example, in the Auxiliaries audit, we found that some of the universities did not use commonplace bulk purchasing arrangements but allowed individual departments to select and purchase copy machines one at a time. Similarly, a private sector working committee, charged by the Board of Regents with examining the need for new campus buildings, found a number of recommendations based on common sense and successful practice in nonacademic environments. Although in Space, audit standards of evidence required us to be on firmer footing before making these common sense recommendations, we brought in a similar outside perspective through our interviews and focus groups with stakeholder groups.

For its part, the public does not generally understand the tripartite mission of the universities. Even the teaching mission is often interpreted differently by various sectors of the public. Some view the universities as sophisticated career preparation mechanisms and criticize the universities for not offering enough classes in business Spanish, while others see an intrinsic value to education and are critical if universities make decisions to eliminate less obviously useful classes like ancient Persian. The public's perspective on the universities' research mission is often negative, influenced by press reports and political speeches on frivolous research. And the universities' third mission-public service-is not widely recognized or well defined. As representatives of the public, performance auditors can serve as a conduit between the universities and the public; we can help to find common ground and explain the universities' mission in simple terms.

Our experience has taught us several hard lessons about how we can evaluate universities in the future and make a positive impact. In particular, we have learned important lessons on audit scope, on methods and data, and on reducing conflict and maximizing impact.

Overall scope. Consider studying universities one at a time. For university audits and evaluations, scope is everything. The complexity of the institution can be overwhelming. Added to the loose confederation of departments that makes up the academic structure is a massive framework of administrative, operational and support units. When multiple institutions are involved, the difficulty is multiplied. Further, multi-university audits may not have the advantages one might expect; each university can point to its unique character-a long-established land grant university, a smaller residential campus, an urban commuter school-to explain away unfavorable comparisons with the other universities and to avoid adopting any improved practices found at another institution.

Scoping topics. Even within a single university, the scope must be carefully chosen. Our Administration audit's potential impact was lessened by its broad scope, which allowed us to make only the most general recommendations. In subsequent audits, we kept the work to a manageable scale by first choosing limited topics and then ruthlessly cutting subissues. In Student Experience and Auxiliaries, we were able to manage broader topics for several reasons: (1) a good body of literature on the topics; (2) general agreement on definitions of key terms; and (3) some existing data documenting conditions, collected by universities, allowed us to focus on causes and solutions.

Evaluability assessment. Although auditors may not have the option of rejecting unevaluable questions assigned to them by law or legislative request, the evaluability assessment will still give them critical information about the project. Whether or not the overall audit scope is negotiable, auditors can use information from the assessment to determine staff and resource requirements, to get an early start on data requests with a predicted long turnaround time, and to identify the blind alleys and wild goose chases that could soak up the calendar with little payoff. For example, athletics is a major auxiliary enterprise at all three universities, but due to the link with unauditable foundation money and other factors, we felt an audit of that area would be unproductive.

Methods and data. Anticipate data problems. Surprisingly, universities are no better than any other state agency in their ability to provide data. In fact, the universities have multiple systems, each seemingly developed for a particular function and incapable of adapting to ad hoc information requests without extensive programming. In Administration, it took six months to receive the data we requested, despite extensive planning and negotiations with key staff; as a result, by the time we received usable data, there was little time left for the analysis and qualitative follow-up that normally begins early in an audit. Evaluators and auditors should consider minimizing data requests, using them only when there is no reasonable alternative way to get the information.

Define terms early. Coming to an early agreement on terms and goals is generally essential, but with university work it can be difficult or impossible to accomplish. For example, we never reached agreement on a definition of administration with the three universities, but to offset the disagreement, used multiple measures of administrative activity. In Auxiliaries, we found each university included a different group of services in its auxiliary category, so we included only those that could be generalized to all three universities.

Select defensible methods. Since methods are always an easy target, the design, methods and data used should be given careful scrutiny. Universities have their own experts who will examine your methods closely with a view to discrediting any unwelcome conclusions. It is not a bad idea to test methods by running them past key university officials. Although a complete consensus on methods may be impossible to achieve, universities may supply useful information and some acceptable ideas regarding methods, and at a minimum, the team is forewarned of university arguments and can prepare for future battles.

Communicate. It is essential to identify an appropriate university official to work with throughout the audit and to provide frequent opportunities to discuss and resolve issues (although that person cannot become a gatekeeper without negative effects on the audit). Maintaining open communication throughout the audit prevents some problems and provides an avenue for resolving others. Auditors should insist on frequent meetings with key university officials to discuss progress. In these meetings, the team can also discuss upcoming decisions and sometimes allow the officials to help in making audit choices. In Space Utilization and Research Parks audits, we found it beneficial to involve university officials in our selection of peer institutions to be surveyed, not only by asking for their suggestions (as we would in any audit) but by obtaining their agreement on the final list. In addition to preventing end-of-audit arguments about the comparisons we make, these discussions resulted in an improved level of understanding and trust between the team and the university.

Plan for attacks. Evaluators are aware that there may be criticism of their work, but they usually turn the political management of the evaluation results back over to the client. However, auditors and independent evaluators should consider developing plans to respond to university criticisms of their topic, work and methods. By thinking ahead and countering potential criticisms as part of the audit program, university "spin" can be minimized.

Using lessons learned in the earlier audits, we conducted our later audits within the budget and target dates set at the outset, and produced reports with a real impact. Future work at the universities will present new opportunities and new challenges.

In Research Parks, practice taught us not to be thrown off course by an early obstacle. The problem was the complex network of legal relationships between the universities, the park corporations, park management firms, and park tenants. Universities argued that the parks were not part of the universities and therefore not subject to audit. While the team pursued other research, upper management negotiated terms of a confidentiality agreement. To ensure that the delay did not result in cost overruns on the audit, we reduced the size of the audit team and ran the audit over a long calendar period. We took that lesson to heart in Auxiliaries and simply stayed out of foundation-funded auxiliary enterprises.

In Space Utilization, continuous monitoring of audit progress and rescoping as needed enabled us to complete work on the core, essential issues despite the inherent difficulties of auditing three entities and the lack of comprehensive data. Finally, when we selected the topics for our last two audits (Auxiliaries and Enrollment Management), we focused on auditability. We conducted preliminary research on each of about two dozen potential topics, using a formal decision making process where we assigned weights to attributes chosen as criteria, and then we scored each topic on each attribute. As a result, we selected two topics which drew upon our strengths, enabling us to avoid some of the difficulties that plagued our earlier university work.

As we continued to evaluate the universities, we became more adept at managing and even avoiding some problems. The investment in our first six audits paid off not only in the direct results of that work but in improving our ability to do more useful audit work in the future.

Back to the top

A View from the States

Sunset of the South Carolina State Reorganization Commission

With the new fiscal year, begun July 1, 1998, the South Carolina State Reorganization Commission no longer exists. With its unique makeup of legislative and executive members, the commission served as an instrument of accountability for more than four decades. Originally created to promote economy and efficiency in state government, its role had become more prominent as it led reviews of agencies, holding them accountable for correcting problems identified by the Legislative Audit Council (LAC) in performance audits. With the adoption of the sunset review process and other statutory oversight responsibilities in the 1970s and 1980s, the commission filled a key legislative oversight role with the state auditor and LAC.

One of the most appealing aspects of the commission was the fact that it was not controlled by forces outside the General Assembly that represented financial or ideological interests. As the name "Reorganization" suggests, the commission initially developed plans for reorganizing state government. The results of projects from the late 1940s to the early 1970s created the nucleus around which the South Carolina state government is currently organized. The paradigm shifts in the 1980s and 1990s changed the commission's emphasis on outcome measurement strategies.

In 1988, the commission's role in legislative oversight was strengthened by legislation which created a post-audit review process based on timetables and reporting requirements. Before the General Assembly enacted this process, agencies were on their honor to correct problems identified in performance audits released by the LAC. Few did. However, between January 1989 and June 30, 1998, post-audit compliance reviews were completed on 29 audits documenting agencies' work on solving problems, real savings potential to the state and improved efficiency and accountability. The dollar-saving opportunities resulting from the reviews undertaken by the commission were important, but more important was the fact that agencies and political appointees were put on notice that their actions were subject to valid, reasonable and independent assessments.

The idea to abolish the commission originated in the Republican-led House of Representatives whose efforts began almost as soon as they assumed control in January 1995. In each of the last four years, the commission's operating budget was reduced and key appointments delayed. In the 1998 legislative session, the House eliminated commission funding in its version of the budget bill; the Senate subsequently replaced the funding in its version. In the final version, the House included language protecting the employment status of commission employees, reassigning them to House and Senate positions. Senate conferees, concerned that the governor could exercise his line-item veto to abolish funding altogether, consented to the House proposal. Senate conferees, including Senate Finance Committee Chairman and President Pro Tempore John Drummond, who also served as commission chairman, believed that this compromise offered the best opportunity to ensure continued use of commission staff.

Most commission staff have been reassigned to budget or other committees in the House and Senate, thereby creating renewed opportunities for incorporating effective oversight into the deliberations of the annual budgetary process. Senator Drummond has indicated that he has no intention of allowing the post-audit review process to lapse altogether. While it cannot continue with the same degree of staff and other resources that had once been dedicated to such efforts, a modified version of the post-audit review process will exist in the Senate Finance Committee. Three-member subcommittees will follow-up on recommendations to ensure that agencies continue to take corrective actions. Since July 1, Senator Drummond has appointed two subcommittees in response to two recently released LAC audits.

This post-audit review process, hailed by one former governor as the "good government" act, put South Carolina at the forefront of the legislative audit process. Though diminished in stature, former commission staff will continue their efforts toward fostering better government practices.

Back to the top

July NLPES Executive Committee Meetings

Excerpts from meetings held at the Las Vegas NCSL Annual Meeting (full minutes are available on the NLPES web site or from NCSL)

Executive Committee Meeting
July 20, 1998

Membership and Nominations. Jane Thesing (SC) emphasized that nomination statements needed to be submitted on a more timely basis. Jane also presented the finished NLPES brochure that she and James Barber (MS) had prepared. Mary Noble announced that Art Heikkula (AZ) had resigned from the Executive Committee as a result of a job change. He will be replaced by Joe Fiala (KY), the next leading vote getter in the May election.

Communications and Newsletter. Gary VanLandingham (FL) demonstrated the new NLPES Web site which will carry enhanced content and extensive lists of research-related sites. The Executive Committee voted that the new site replace the existing NLPES site link on NCSL's Web site as soon as possible. Craig Kinton (TX) announced that Mark Bucherl (IN) is unable to continue to serve on the Executive Committee, but will continue to edit the NLPES News. Craig also announced the following: Work is continuing on placing more of NLPES's Works in Progress (WIPs) and completed works in the NCSL data base. The LEGISNET data base should be checked to determine that submitted reports are being uploaded. Work will continue with NCSL and Bob Boerner in producing NLPES reports in searchable formats.

Gary said that no password is necessary to use the NLPES Web site nor the NCSL Web site. The Executive Committee determined that the NLPES listserve and other NLPES data and information materials continue to be publicly accessible through the Web site.

Training. Joel Alter (MN) reported that a training survey has been submitted. Mary discussed training topics for the fall training meeting in Sacramento. The theme for the Sacramento meeting will be "Keeping Government on Track."

Awards. Gary announced the recipients of awards. Virginia and California received the "Excellence Award." Arizona received the methodology award. Twelve states received impact awards. Gary also reported on an initiative to develop an award recognizing individual contributions to the field of legislative program evaluation.

Liaisons. Joel reported that organizational liaisons had been sent a statement that outlined the duties of liaisons. There remain three organizations for which NLPES needs to recruit new liaisons: the American Society for Public Administration, the General Accounting Office, and the Joint Committee on Standards for Education Evaluation.

American Evaluation Association Journal. Kirk Jonas (VA) reported that the AEA "New Directions" journal project was on schedule and is due to be published in the spring of 1999.

Fall Training. Antonio Jones (WV) said that West Virginia would be hosting the 1999 fall training conference in Charleston and will work with James and Mary regarding their recent conference hosting experience.

**********

NLPES Annual Luncheon
July 22, 1998

Mary Noble announced the 1998-99 members of the Executive Committee:

Gary VanLandingham (FL)
Craig Kinton (TX)
Gary Brown (MI)
Tony Jones (WV)
Mary Noble (CA)
Kirk Jonas (VA)
Jane Thesing (SC)
Gerry Hoppmann (LA)
Heather Moritz (CO)
James Barber (MS)
Joel Alter (MN)
Joe Fiala (KY)

Gary announced Virginia and California as recipients of the "Excellence in Program Evaluation" awards. Accepting on behalf of the Virginia Joint Legislative Audit and Review Commission were Senator Kevin G. Miller, Delegate Harry J. Parrish and staff director Philip Leone. For a copy of the Virginia award statement, click here.

Mary accepted the "Excellence in Program Evaluation" award for the California Bureau of State Audits. For a copy of the California award statement, click here.

Mary Noble accepts award from Gary VanLandingham

The award for "Excellence in Research Methodology" was presented to Bill Thompson, Lois Sayrs and Elizabeth Holtzopple on behalf of the Arizona Office of the Auditor General. For a copy of the Arizona award statement, click here.

Impact awards were announced for 12 states. Senator Peggy Reeves and Larry Gupton accepted the Colorado impact award. Other awards were accepted by staff.

Staff award recipients (l. to r.) Marion Higa (HI) Impact, Janice Mueller (WI) Impact, John Turcotte (FL) Impact, Mary Noble (CA) Excellence, Larry Gupton (CO) Impact, Phil Leone (VA) Excellence, Jane Thesing (SC) Impact, Lois Sayrs (AZ) Methodology, Joel Alter and Roger Brooks (MN) Impact.

Mary encouraged staff to review the new NCSL CD-ROM. She invited members to provide suggestions on revising the CD-ROM sections pertaining to NLPES. Gary discussed the revised NLPES Web site, LEGISNET, and other policy and training Web sites.

On behalf of the Executive Committee, Jane, presented Mary with a special service award. Mary thanked the members of the Executive Committee and NLPES for their support during the past year.

**********

Executive Committee Meeting
July 23, 1998

Election of Officers

Gary VanLandingham (FL) was elected chair. Craig Kinton (TX) and James Barber (MS) were elected as vice-chair and secretary, respectively.

Appointment of Committees

The chair appointed the following subcommittees for 1998-1999.

The Executive Committee voted to establish a new NLPES award to recognize outstanding contributions to legislative program evaluation.

The chair discussed the following Executive Committee goals for 1998-1999:

• Improving NLPES's outreach to the states' evaluation and audit staffs, primarily through a more effective and active Web site.
• Selecting future sites for the annual fall conference training.
• Revising NLPES's bylaws.
• Reviewing criteria used to select recipients of NLPES's annual awards.
• Expanding training and professional development opportunities provided by NLPES to the states' evaluation and audit staffs.

The Executive Committee addressed the following new business items:

Craig stated to the committee that comments should be given to Mark Bucherl as to how he could continue to improve the NLPES Newsletter to make it more useful to members.

The Executive Committee voted to accept the offer of Florida OPPAGA to maintain NLPES's Web site and to transfer responsibility from NCSL to OPPAGA immediately.

The chair asked Executive Committee members to consider ways in which the NLPES listserve could be used to more effectively serve members.

The chair announced that NLPES state assignments would be divided among Executive Committee members in the near future.

The Executive Committee's spring meeting will be held in conjunction with the Assembly on State Issues meeting in Jacksonville, Fla, April 9-11, 1999.

Max Arinder (MS), newly elected member of the NCSL Executive Committee, reminded members that the NLPES Executive Committee chair and vice-chair serve on the Legislative Staff Coordinating Committee, which gives the members a voice in NCSL affairs. He encouraged the NLPES Executive Committee to continue reaching out to and coordinating with other NCSL staff sections, and he offered his assistance.

Back to the top

News From the Net

The Evaluator's Internet Toolbox

Steven Birnholz (FL)

Everyone knows evaluators use the Internet. The question is what do we use it for? Many of us surf the Web in search of articles, data, airline prices, weather forecasts, and other information crucial to our reports. Now, buried within the Internet's 300 million pages are several sites that provide conceptual, statistical, and financial tools to aid the evaluation process.

The first stops for any evaluator surfing the Web should, of course, be the NCSL web site and the NLPES site. The NCSL site will provide information on recently released studies by NCSL, while the NLPES site will (soon) let you search for studies released by other states' evaluation units, works in progress by other states, and links to great research sites. Next, you should visit the American Evaluation Association's home page. Here, you'll find recognized program evaluation standards as well as links to other useful evaluation sites. Program evaluation guidelines can also be found on the United States General Accounting Office's web site. Not only does the GAO publish the "Yellow Book" standards on-line, it also provides numerous guidelines regarding topic-specific audit methodologies and links to copies of all GAO studies.

Once we have established our evaluation frameworks, it is often necessary to conduct survey research and statistical analysis to test our hypotheses. Many Internet sites now provide tools to enhance our statistical endeavors. The American Association for Public Opinion Research, for example, maintains a list of best practices for survey and public opinion research. Two other web sites, http://trochim.human.cornell.edu/index.html and http://www.deakin.edu.au/~agoodman/sci101, can guide evaluators through the jungles of survey design, administration and analysis. For those closet statisticians out there, both StatSoft and David Lane of Rice University have published on-line statistics textbooks. Useful calculators of statistical significance and sample size can also be found on the Web.

Financial and economic analyses also play big roles in the evaluation process. The Internet now houses guides for both activities. Evaluators can find informative Web sources about financial accounting and cost accounting. Additionally, government-specific financial information resides on FinanceNet and on the Financial Reporting in Government site. If you need to brush up on your economics, Drexel University provides an on-line textbook on its web site.

Re-engineering government is currently a hot legislative topic. Two great web sites provide tools to help evaluators implement cutting-edge reform efforts. The Alliance for Redesigning Government has established the on-line Reinventing Government (REGO) Database (click on the "Concepts and Resources" link). This database includes information on a wide range of topics regarding innovation in government, including quality management, performance measurement and alternative service mechanisms. Another site worthy of investigation is the National Performance Review Home Page. Here, you will find many re-engineering best practices that could be used during your evaluations.

With an increasing number of policymaking and research organizations publishing their goods on the Internet, the Web is quickly becoming a valuable source for evaluation tools. Only a few of the available instruments have been described above. The Internet Scout Project manages a listserve that regularly e-mails the Scout Report, a list of potential Internet research sources, to subscribers. Checkout this time-saving service, and let us know if you find any other sites that could be helpful to evaluators. Remember: The Internet can be an evaluator's toolbox but only if we work together to fill it up.

Back to the top

News From the Net

The NLPES Listserve

Sandra Larson (AZ) sought other states' "how-to's" regarding report review and processing. She went to the NLPES listserve for help, "We had three or four responses that were very prompt and helpful. We called the respondents for more detailed information, which seemed the best next step. The listserve got us an idea of who would help.

OK, that's just one more testimonial about the power of the NLPES listserve. If you haven't signed up yet, you're missing out on the opportunity to expand your horizons electronically. One more comment for you veteran listservers who've learned the hard way: The NLPES members and associates on the NLPES listserve consistently generate "all-meat" messages, with no byproducts (i.e., no spam).

**********

Using the NLPES Listserve

(1) Address an e-mail message to listserv@ncsl.org.

(2) Type in the message area: Subscribe NLPES-L Firstname Lastname.

(3) Wait for confirmation of your subscription-usually within 24 hours.

(4) Send a message to all NLPES subscribers by e-mailing to: NLPES-L@ncsl.org.

Back to the top

News From the Net

WIPS Update

The Works in Progress (WIPS) records in this Web on-line directory are updated regularly.  If you have any questions or need assistance, contact Bob Boerner at (303) 364-7700, fax (303) 863-8003.

Back to the top

News From the Net

Web Site of the Issue

US Newspaper Archives on the Web

Need to verify some of the local color, check a few facts or perhaps find out who in the local rag was "in the know" regarding some past controversy you are investigating now? Here's a nice little site to peruse. Not exactly among the apex of researchers' Web delights, nonetheless, I've found this little site very informative in portraying public response to certain initiatives other states happened to try before Indiana. Have a look! Editor

Back to the top

News Makers

Back to the top

NLPES Fall Training Conference

"Keeping Government on Track"
September 30 through October 3, 1998
Sacramento, California

Make tracks to Sacramento and attend this year's NLPES training conference. Gain valuable knowledge from keynote speakers, as well as colleagues from other states, who will discuss challenges and accomplishments at concurrent sessions in each of the following three tracks:

The Challenges of Leading and Managing
Creative Auditing Techniques and New Technologies
Emerging Issues and Hot Topics

Dynamic speakers and representatives from several states will explore topics such as project management; team building, collaboration and communication; recruiting, hiring and retaining staff; development and use of surveys; usefulness of focus groups in gathering information; methods for evaluating agency staffing levels; electronic commerce; privatization; analyzing education expenditures; health care reform; and much more.

While at the conference you can also enjoy the rich and varied settings that the Sacramento area has to offer, including Old Sacramento, which provides a glimpse of California's Gold Rush era with its shops, restaurants, and fabulous railroad and history museums, and beautiful foothills enveloping authentic gold mines and award-winning wineries. Sacramento has something for everyone.

Back to the top

Small Shop Talk:

Avoiding Employee Turnover

Nancy VanMaren (ID)
Barbara Rogers (WY)

Employee turnover is a challenge for any office. However, small shops are particularly vulnerable, with any departure reducing staff by a sizeable percentage. The work of the small shop can grind to a virtual standstill during the recruiting, hiring and training of new people.

How then to retain quality staff? In our experience, there is no one magic solution. Each person's job satisfaction is based on a complex set of expectations that may not entirely dovetail with the demands of the office as a whole. Retention requires a delicate balance between personal needs and professional demands. With that in mind, we have assembled some of the ideas we have found helpful in our offices:

Pay attention to staff training. Travel to national conferences is a training perk that every staff person should be able to enjoy occasionally. Other opportunities are closer to home: specialization and cross-mentoring between staff members can increase job satisfaction and performance. In one office, a staff member well-versed in the workpaper process coaches each new staff member during the first year to hone these skills. A staffer with legal expertise mentors staff on legal research so others can develop these skills as well.

Consider cross-state, short-term projects or shared training with other legislative staff units within or outside your state to increase the office knowledge base and stimulate staff with new ideas. Our two offices are exploring a short-term exchange, involving travel to each other's states, observation of work methods, and the exchange of written conclusions about what was learned.

Help staff to identify professional skills they want to develop, and arrange the time for them to do it. In small shops, everyone does a little of everything and focuses more in the areas in which he or she has special skills. Heavy workloads often mean that little time is set aside to develop expertise in a skill for which someone else already has responsibility. In one of our offices, a staff member is learning to write and administer contracts with outside individuals and organizations, normally the responsibility of the director. Similar steps can improve the overall office skill set as well as job satisfaction.

Develop alternatives to promotions. In relatively flat organizations, title changes may not be possible, although titles that accurately reflect increased ability can be as important as pay increases. Aside from this, varying staff responsibilities helps to prevent burnout over time. For example, allowing the "data person" to do the qualitative analysis while a staff member stronger in issue framing develops a project's data base(s) helps both to develop new skills and avoids the monotony of repetition. Inviting staff members to accompany managers to high-level meetings will likely increase the feeling of being "in" on things and may not significantly detract from other office efforts.

Create an appealing office culture. Small shops have an increased opportunity to develop camaraderie. It may sound simplistic, but developing an environment that favors smart, nice and adaptable people provides an essential work foundation that is personally satisfying and professionally rewarding. Occasional casual dress days and office social events increase camaraderie and productivity. One of our offices just began annual participation in a citywide tree planting campaign; we're symbolically replenishing the mass of paper our office consumes.

Anticipate turnover as best you can, and factor in down time for medium and long-term project plans. One of our offices recently determined a "productivity factor," the percent of time any one evaluator spends on a project after accounting for vacation, sick leave, administrative responsibilities and things that "come up." This factor is being applied to estimated time requirements for project completion. Accounting roughly for turnover as well, we anticipate it will help even-out workloads over time.

Don't sweat the small stuff. This may run counter to your nature and training, but there's only so much work you can do; "steering bread crumbs" should be the first thing to go. Remember that the larger goal is to produce quality products in a work environment that doesn't burn out good people.

Finally, remember that despite the challenge, some turnover is natural and can reinvigorate the office. The loss of a particular staff member can be painful, but new blood, a different mix of talents, and even a new sense of humor can ease the turnover's downsides. Work to avoid turnover, but recognize that change may also have its upsides.

Sometimes (as in this recent survey), employees think differently than their supervisors. Supervisors thought employees were mostly motivated by money; employees said they wanted appreciation more than anything else. Here is a list, in order, of what employees wanted most, and a list of what supervisors thought they wanted:

(Glenn Tobe & Associates)

Back to the top

From the Editor

Mark Bucherl (IN)

A few comments on this edition of the News. A special thanks to Arizona contributors Lois and Shan for an excellent article on how to audit universities-those with university audit experience will nod their heads in agreement with the challenges involved, and those considering the task are aptly forewarned. The Web edition appears on the new NLPES new Web site. New NLPES president Gary VanLandingham and OPPAGA have developed quite an admirable Web presence, and I am sure will maintain our site with equal aplomb. Quality control thanks to several pairs of eyes in my office: Kevin Kent, Daria Stafford and, of course, Kristin Breen, co-editor. Thank you, too, Janet Randolph, NCSL editor.

On a personal note, I am no longer a part of the Executive Committee, but will continue to edit the newsletter for the indefinite future. The persons I've worked with on NLPES are outstanding, the articles you have submitted "meaty," and our evaluation field is, more than ever, necessary and useful. That's enough to maintain my interest...

Back to the top

Calendar

Back to the top