• Mission & Governance
• Services Overview
• How to Get Involved
• Standing Committees
• NCSL Foundation

• Issue Areas: A-Z
• State-Federal Relations
• NCSL Standing Committees
• News from D.C. and the States

• About State Legislatures
• Elections, Campaigns & Redistricting
• NCSL's StateConnect Directory
• Web Sites

• Online Guide to NCSL Services
• Staff Sections and Networks
• Meetings & Professional Development
• Staff Directories

• Meetings Calendar
• Online Registration
• Meetings Home Page
• Legislative Summit

• Bookstore
• State Legislatures Magazine
• NCSL's StateConnect Directory

NLPES News
National Legislative Program Evaluation Society

Chairs's Corner
The Times, They Are A-Changin'
Observations From a Performance Auditing Seoul-Mate
A Field Report From Our Men (and Woman) in Hong Kong
From the Trenches: Confessions of a Recovering Academic
A Stranger In a Strange Land
Using the NLPES List Serve
Calendar
A Cross-Reference Guide to Auditing and Evaluation Standards
Office Happenings

Chair's Corner
James Barber (MS)

As I write this column, the Mississippi legislative session, like those in many of your states, is in full swing. Our staff has recently completed several major reports that have captured the attention of legislators and legislative committees and drawn the ire of agency managers. We have been asked to assist with bill drafting and committee hearings on those topics. It is during these busy times that I find myself reflecting on the importance of legislative program evaluation and performance auditing. Many important issues are debated and resolved based on information contained in our reports. Although our work is challenging and tedious, it is gratifying to know that we can (at least occasionally) have an impact on public policy decisions made by our legislatures. I commend each of you for the jobs that you do day-to-day on behalf of your legislatures. Keep up the good work!

The spring and summer will be especially busy seasons for NLPES. The Executive Committee will conduct its annual election during March and April, with six positions to fill. I encourage you to consider submitting your name as a candidate for this election. During April and May, the Awards Committee will be busy soliciting and evaluating entries for the Excellence in Evaluation, Excellence in Research Methods, and Certificates of Recognition of Impact awards presented annually by NLPES. Again, I encourage your office to consider making a submission for one of these awards.

The Executive Committee will also be in the process over the next several months of developing NLPES-sponsored sessions for the NCSL Annual Meeting to be held in San Antonio, Texas, on August 11-16. It is our goal to present sessions of interest to both legislative staff and legislators. Should you have any ideas for such sessions, please contact Heather Moritz (heather.moritz@state.co.us), chair of the NLPES Annual Meeting committee. During the spring and summer, the NLPES Training Committee will continue to assist the staffs of the Kansas Legislative Division of Post Audit and Missouri Joint Committee on Legislative Research, Oversight Division, as they plan the NLPES fall training conference to be held on September 5-8 in Kansas City, Missouri. Please make plans now to attend this worthwhile training conference.

On a final note, I began this column by describing how busy we all are and the importance of the contributions made by legislative program evaluators and auditors. It is important to remember that you are not alone as you do your work. Other NLPES colleagues are literally only a "click of a mouse" away through a posting to the NLPES listserve or a visit to the NLPES website. I encourage you to take advantage of the resources available through other NLPES offices and members. Best wishes for a productive spring and summer!
 

The Times, They Are A-Changin'

Many years ago, a prophet wrote these words that now hold a particular insight into the state of legislative program evaluation. For the legislative world is changing rapidly, and many of the precepts that have undergirded our profession are rusting out beneath us. We must either adapt to this changing world or die.

Most legislative program evaluation units were established in the 1960s and 1970s. As aptly described by legislative scholars such as Alan Rosenthal and Karl Kurtz, this was a time when legislatures increased their staffs in order to become co-equal to the executive branch and gain the ability to independently analyze policy issues and oversee policy implementation. Program evaluation was a hot new government reform that swept through most states and the federal government. There was an expectation that social science researchers would apply rigorous scientific methods to discover the "truth" about which programs worked and which didn't. It was also expected that policy makers would eagerly use this information to fine tune programs and make government more efficient and effective. These were heady times when our profession was seen as the cutting edge of enlightened policy making.

Most of the norms of our profession were laid down at this time. As reflected in the GAO Yellow Book and other sets of evaluation standards, we are to be independent and hard-nosed in our thinking, scientific in our data collection and methods of analysis, and focused on documentation in order to prove the truth of our findings. We are to take the time required to do high quality studies and are to have elaborate review processes to make sure that our reports meet the highest professional standards.

Unfortunately, the world has changed since this golden age. More tragically, our field generally has not recognized or adapted to these changes. Three factors particularly stand out.

First, performance evaluation is no longer the hot point of government reform, and hasn't been for a long time. In recent years, new reform waves have hit -- zero-based budgeting, management by objectives, strategic planning, continuous process improvement, total quality management, re-engineering, performance measurement, privatization, and e-government. Although, as highly trained good government experts, we could and should have played a leadership role in such reform efforts, our offices instead generally ignored these initiatives and focused instead on our narrow missions.

Second, we now have a lot of competition. Back in the good old days, the only entities that issued program evaluations and policy analyses were our own high-minded legislative evaluation units, self-serving executive branch agencies, and generally indecipherable academics. Nowadays, there are literally thousands of think tanks, foundations, and interest groups that churn out policy studies, issue papers, and evaluation reports. More importantly, these groups are usually highly media savvy and specialize in grabbing the political spotlight. They use modern media techniques to catch legislators' attention that we either consider to be off limits or beyond our capabilities. We do better work, but legislators in most states hear a lot more about studies conducted by these groups than our own reports.

Third, we have often become internally focused. Due largely to the strong emphasis we place on being "independent" and the introverted nature of our work, we generally do little to market ourselves or our studies. Further, our reports are often internally focused - designed to meet peer review requirements or long-standing office conventions - rather than externally focused - designed to capture the interest of and educate busy readers.

The result of these factors is that our impact is often marginal, a fraction of the initial vision that created our profession. It is a safe bet that legislators and other policy makers in most states have little knowledge or understanding of who we are and what we do.

In case you think this is a massive overstatement of angst, consider several questions that demand honest answers.

• How many legislators in your state would know who you are (as an office or as an employee) if you called them on the phone? How many could greet you by name if you saw them in the hall? Does your office try to establish strong relationships with legislative leadership, or does it view such efforts as heretical threats to its independence?
• Is your office a "player" in policy and government reform initiatives? How often do legislative leaders call your office to ask for help and advice on policy issues? If your state is facing a budget shortfall, is your office asked for advice on where to cut?
• How attuned to leadership issues is your office's research agenda? How often does your office brief legislators and testify before committees about reports that are issued -- always, sometimes, or hardly ever?
• Are you confident that your reports cover the issues that are most important to your legislature? Are you confident that your reports are always timely by the legislative calendar - meaning when members need them? Do you ever find yourself having to refuse to brief members or do presentations because a report hasn't yet cleared internal quality assurance procedures?
• Do you have a way to truly gauge the impact of your studies in terms of cost savings achieved or recommendations implemented? Could you give a "return on investment" statement if asked?
• How often does the press cover reports released by your office -- always, sometimes, or hardly ever? How many political and state government reporters know that your office exists and what it does? How much work has your office done to cultivate such press contacts and awareness, or is this viewed as heretical?

To address this situation, we need to remember why we are here. In virtually all states, evaluation offices were established with the vision of actively helping their legislatures make the tough policy and budget choices. We were attracted to this challenge and became legislative evaluators because we wanted to make a difference. We still can. However, we need to get out of our comfort zone and realize that having an impact requires that our legislative bosses know who we are, care about what we are doing because our work helps them achieve their goals, and know that we can demonstrate our worth. Independence that borders on irrelevance isn't a state of grace; it is instead the highway to oblivion.

Collectively, we are very good at what we do. But we aren't the ones who need to know this.

To move forward, we must challenge some of our long-standing paradigms and adapt to the real world. The times are a-changin, and we must either swim or sink like a stone.

To juxtapose the wisdom of three divergent figures:

"Who am I? Why am I here? - Admiral Stockdale

"If not us, who? If not now, when?" - Ronald Reagan.

"The fight goes on, the cause endures, and the dream shall never die." - Ted Kennedy

Observations From a Performance Auditing Seoul-Mate

Note: Jung-Soo Han graduated from the Korea Military Academy in 1980 and subsequently was a platoon leader, company commander, and regiment officer in the Korean Army. In 1987, he joined Korea's Bureau of Audit and Inspection (BIA). He earned a master's degree in public administration from Sungkyunkwan University in 1991 and a master's degree in public policy from the University of Southern California in 1995. He presently has a Korean government fellowship to study performance auditing in the U.S., and he is doing this while working in the Arizona Office of the Auditor General.

Korea's Board of Audit and Inspection (BAI)

The audit system of Korea has a 1,300-year history. It originated in the Sajongbu, which was established as one of the central agencies of the Shilla Kingdom in 659 AD. This agency later became the Sahonbu in the Koryo and Chosun Dynasties. The main function of these ancient organizations was uncovering wrongdoings of central and local government officials.

The Board of Audit was founded when the Republic of Korea's government was established in 1948. The Board was merged with the Commission of Inspection in the revised constitution of 1962, forming the current Board of Audit and Inspection (BAI). To guarantee the independence and objectivity of the Board, the Constitution clearly states its mandates, functions, and organization.

The BAI has a seven-member Council, appointed by the President, which makes decisions on audit policies, major audit topics, audit recommendations, reports, and the Board's budget and regulations. A Secretariat, composed of seven audit bureaus, conducts BAI's audit and inspection duties.

The BAI has three major responsibilities: (1) confirming the State's final accounts of revenues and expenditures; (2) auditing central government agencies, provincial governments and government-invested organizations; and (3) inspecting the duties of government agencies and their employees. In addition, BAI investigates claims that government has infringed on the rights or interests of others, and it provides authoritative opinions concerning legislation and interpretations of accounting laws.

Altogether, BAI audits nearly 68,000 entities that have more than one million employees. More than half of the entities are subject to mandatory audits. BAI has about 870 staff, and it struggles every year to perform required audits with limited personnel and budget.

The BAI Act mandates that "BAI... shall inspect the duties of the administrative agencies and public officials in order to improve and promote the operation of public administration." The law does not specifically mandate performance auditing, but this function has generally become recognized as a legitimate one for BAI.

In 1999, the BAI unveiled new standards for financial and performance audits, consistent with auditing standards put forward by the INTOSAI (International Organization of Supreme Audit Institutions, of which GAO is an active member). The standards are not yet enforced by law, so BAI relies on its traditional regulations and manuals to conduct performance audits.

Requests and ideas for performance audits may come from the President, congressmen, and interested groups and individuals. However, the Council has the sole authority to select performance audits.

The Council may deliver six types of resolutions based on the audit findings: Judgment on Liability for Reparation; Request for Disciplinary Action; Request for Correction; Request for Improvement; Institution of an Accusation with the Public Prosecutor; and Recommendations. The lightest resolution, the Recommendation, is made only when other types of resolution seem inappropriate and the audited agencies are deemed to be able to handle the matter on their own initiative. By law, audited entities must implement BAI's resolutions. However, they may request re-examination if they believe the resolution is unjust or illegal.

BAI's unusually strong power to force compliance with its resolutions was partly a response to citizens' demand to eradicate public fraud and irregularities-which are not rare in developing countries. BAI's power also reflects the fact that the legislature does not have enough resources to ensure that audited agencies fulfill prescribed corrective actions.

Each audit report is sent only to the audited entities. The National Assembly receives a comprehensive annual report from BAI, and its committees can get individual audit reports only upon request.

As a result of 1999's audits, there were 6,624 resolutions--including 14 judgments, 453 disciplines, 4,135 corrections, 14 improvements, 56 accusations, and 1,952 recommendations. The resolutions involved about $270 million to be collected, withdrawn, or preserved.

Comparing Performance Auditing in Arizona and Korea

I asked the Arizona Office of the Auditor General (OAG) to assign me to an audit project so I could experience the whole process. When I arrived in Phoenix, I was assigned to an audit of private prisons. We have since completed the preliminary survey and mission analysis and entered the fieldwork stage.

I was delighted by my assignment because the topic was new, interesting, and challenging for me. But I was surprised because the size of the audit team was so small for such a mission - only two full-time members, including myself - and the timeline was so long (nearly a year from start to publication). If an audit project of this kind were planned in the BAI, a division of at least 12 auditors would be involved for four months at most. At BAI, a major focus is on completing audits quickly--to reduce the burden on the audited entity and to address issues in a timely way. A survey of Korean government employees once indicated that frequent, lengthy audits were a major obstacle to agency creativity, and this had a great impact on the current audit practices of the BAI.

The OAG and the BAI have almost identical goals but very different procedures for achieving them. BAI's approach is "individual-oriented," while OAG takes a "team-based" approach. OAG audits have several meetings where the team members, team leader, manager, Auditor General, and related staff share knowledge and ideas to improve audit quality and keep the work on track. OAG also uses checklists and audit programs to plan every activity, and every result is well documented in the working papers. In the BAI, it is usually up to the individual auditor to determine how to fulfill an assignment. BAI audit managers exercise leadership and supervision, but the auditor in charge has discretion in a wide range of functions and has full independence in performing field work. BAI auditors are evaluated based on the gravity and number of findings they make in their audits.

Some of the audit practices on the other side of the Pacific Ocean may not be easily understood. I think these practices are deeply associated with our tradition, culture, and government structure, as well as the public's unique expectations toward audit institutions. I may be able to provide you with a better comparison of Korea's and Arizona's practices when I complete the private prison audit in May 2001.

Closing Thoughts

I have learned far more than I expected during my first months in Arizona. My experience here will benefit not only Korea but also other Asian countries. An Asian subpart of INTOSAI launched a long-term regional training program in 1997, and I am one of its training specialists. During a recent ten-day workshop I was instructing in Bangkok, I felt the need for more knowledge of systematic audit practices to make a truly qualified auditor and instructor. What I have learned and experienced here in Arizona will be a great asset for me personally, and it will give me a lot of things to share with fellow auditors of Korea and other Asian countries.

I want to thank Debbie Davenport (Arizona Auditor General), Bill Thomson (Deputy Auditor General), and every colleague at this office for their kindness, support, and tolerance of my poor English. I am especially grateful for this valuable opportunity to share my experience with legislative program evaluators and performance auditors in NLPES.
 

A Field Report From Our Men (and Woman) in Hong Kong

Jane Thesing (SC)

Note: In late 2000, Max Arinder (MS), Craig Kinton (TX), and Jane Thesing (SC) represented NLPES in a legislative staff exhange program. This is an account of their visit in Hong Kong.

With the support of the U.S. State Department, the National Conference of State Legislatures organized a legislative staff exchange program with the Legislative Council (LegCo) of the Hong Kong Special Autonomous Region (HKSAR). In the first phase, a 12-member delegation from LegCo toured the United States in July 2000. The delegation attended NCSL's annual meeting in Chicago and met with legislative staff in California and Washington.

In the second phase of the exchange, a ten-person U.S. delegation traveled to Hong Kong in November 2000 for a series of professional dialogues. The American delegates included representatives from three NCSL staff sections: NLPES, the Research and Committee Staff Section, and the Legal Services Staff Section. NLPES's own Max Arinder had the honor (and responsibility!) of being leader of the entire U.S. delegation. Our visit included six days of meetings and observations at the Hong Kong LegCo and a one-day visit to the Macau Special Administrative Region (a former Portuguese colony) for meetings with legislative staff.

Hong Kong's Government

The Hong Kong Special Administrative Region (HKSAR) is a former British colony that was returned to Chinese sovereignty on July 1, 1997. The area of the HKSAR is approximately half the size of Rhode Island and includes Hong Kong Island, the Kowloon peninsula, the New Territories, and other islands. Its population is 7.1 million, of whom 98 percent are ethnic Chinese. Official languages are Chinese and English. Deliberations of LegCo are primarily conducted in Chinese with simultaneous translation into English. The U.S. delegates found the LegCo staff to be highly proficient in English (humbling, given our total lack of Chinese proficiency!).

Hong Kong's government is established by the Basic Law, a "mini-constitution" approved in March 1990 by the Chinese National People's Congress. This law provides that Hong Kong can retain its pre-handover social, economic, and legal systems for at least 50 years after 1997. This arrangement is commonly described as "one country, two systems." Hong Kong has autonomy except in the areas of defense and foreign affairs. As described by U.S. officials with whom the delegation met, Hong Kong has remained a free society since the take-over, with free speech and press much in evidence.

A chief executive chosen by a Beijing-appointed panel heads Hong Kong's executive branch. The government has about 15 policy bureaus whose employees are civil servants.

The Legislative Council is a unicameral legislative body. It has 60 members (24 directly elected by voters, 30 elected by functional groups, and 6 elected by an election committee). Though this may seem to be a limited democracy, it is important to note that there is little history of democratic government in Hong Kong; prior to 1985, all legislators were appointed by the British governor.

LegCo's primary function is to approve budgets and other bills proposed by the government (70 to 80 bills per year); it has little authority to propose legislation itself. It can amend the government's spending proposals only to decrease them. LegCo also handles citizens' petitions and complaints about various topics. LegCo has both standing committees and committees created to consider individual bills. It has weekly sessions on Wednesday afternoons throughout the year.

LegCo has about 300 staff, headed by Secretary General Ricky Fung. Staff are employed on renewable three-year contracts. The LegCo building is a former court building in the heart of Hong Kong island that has been beautifully renovated for its current purpose.

LegCo staff-led by Deputy Secretary General Kam-sang Law-provided a full agenda of informative briefings for the American delegation. The staff told us how they support the work of LegCo committees and members (and we, in turn, discussed how we work with our state legislatures). In addition, we observed sessions of the LegCo and its House and Finance committees, and we heard presentations from government officials, local academics, and U.S. consulate officials. These meetings improved our understanding of the work of our hosts, and our respect for the professionalism and skill of our Hong Kong counterparts increased daily.

Performance Auditing in Hong Kong

The NLPES participants spent one day shadowing our direct Hong Kong counterparts-the staff of the Audit Commission. In Hong Kong, the Audit Commission is part of the executive branch of government. The Director of Audit is appointed by the Chief Executive, but he can only be removed "for cause." The audit ordinance that governs the work of the Audit Commission was enacted prior to the Basic Law and the handover to China. It is still in effect, as are any laws established in Hong Kong prior to the Basic Law that do not conflict with it.

The Audit Commission follows the British model of auditing and conducts regularity (financial) and value for money (performance) audits. Its staff of 200 includes 100 professional auditors. The Audit Commission submits two reports a year to LegCo, each containing 10 to 12 value for money audits. The Audit Commission selects its own topics for audit, and its methods of conducting the audits were very familiar to the NLPES members.

The Audit Commission is tied into legislative oversight through LegCo's follow-up activities. Specifically, LegCo's Public Accounts Committee holds public hearings on the audit reports, and it then publishes reports containing evidence obtained in response to the audit. (These are some fat reports!)

We Americans felt right at home because our visit coincided with the semi-annual release of value for money audits. The next day's newspaper had an audit-inspired headline proclaiming that expatriates who work in Hong Kong are avoiding taxes to a substantial degree!

A World of New Experiences

The LegCo staff made great efforts to acquaint us with some of Hong Kong's cultural and culinary attractions. We were treated to a guided tour of the Hong Kong Art Museum and a concert by the Hong Kong Chinese Orchestra (which had instruments that this westerner had never seen or heard before!). We feasted on many interesting and delicious cuisines (Vietnamese, French, and regional Chinese specialities). We experienced Hong Kong's excellent public transportation systems as we traveled by subway, train, tram and ferry. (Hong Kong at night from the ferry is truly an amazingly beautiful sight!) Our hotel was great, once you figured out that you had to insert the room key into a slot in the room in order to turn on the electricity. Our only complaint was that our rooms came equipped with bathroom scales, when we wished to disregard the consequences of our feasting!

If they haven't already, the folks in charge of tourism marketing for Hong Kong should recruit the LegCo staff! They were uniformly informed, gracious, friendly, patient, fun-loving and indefatigable-just the people you would like to be your friends when visiting their home. They were also intelligent, well-informed and well-organized, exemplifying all that we strive for as we think about models of public service.

Concluding Observations

Those of us who participated in the Hong Kong exchange learned that something can be far and yet near; foreign and yet familiar. Though the culture in Hong Kong is unmistakably Asian, and thus foreign to those of western orientation, we found that the work of the auditors and other legislative staff there was familiar. They share

our orientation toward work, our striving to provide excellent service to our legislatures, and similar auditing methods and practices.

We can all profit by making more efforts to communicate with our Hong Kong colleagues. LegCo and Audit Commission publications are available at their excellent web sites, and the staff can all be reached via e-mail. Although personal meetings are no doubt preferable to electronic communication (we highly recommend Hong Kong as a travel destination!), we urge other NLPES members to make the acquaintance of our friends across the world. They are as near as your fingertips at www.legco.gov.hk (LegCo) and www.info.gov.ht/aud (Audit Commission).

An unanticipated side benefit of our experience was the opportunity to meet and interact with our NCSL colleagues from other staff sections. Those of us who work for separate audit agencies often have limited opportunity to interact with other legislative staff colleagues in our own states. It was valuable to exchange ideas about the services we provide and the issues that concern us; perhaps our experience can help us implement more fruitful communication and cooperation on the home front as well as halfway around the world.

From the Trenches: 
Confessions of a Recovering Academic

I have kept a cartoon in my office for many years that serves to keep me humble about my advanced degrees and the perception of some about their value to the world beyond the university. The cartoon shows a person applying for a job in a stable and the employer asks, "Do you have any qualifications?" "A Ph.D. in political science," the applicant responds. The employer says, "I take it that's a no."

In June 1999 I left my position as Associate Professor of Political Science and began a new career as a legislative policy analyst with the Florida Legislature's Office of Program Policy Analysis and Government Accountability (OPPAGA). It took me a while to find just the right position that would enable me to use my legislative expertise and research skills. I eventually found out about OPPAGA, the work that it does and its employment opportunities, via the Internet while I was living and working in another state.

Someone asked if there was anything that surprised me or failed to surprise me about making the transition from a faculty position to a legislative staff position. After years of teaching, plus some state government experience early in my career, I would say that little has changed about the games that bureaucrats play. After little better than a year in Florida, I have enough examples of agency bureaucratic gamesmanship to last me another dozen years, had I any desire to teach public policy or government again. On the other hand, most surprising is how easy the transition has been from a system where people are tenured for life to a position where one serves at the pleasure of the director, who serves, in turn, at the pleasure of elected officials. While I would not want to minimize the value of academic tenure as it concerns academic freedom, there is a level of personal and professional accountability present in my current position that seemed to me to be lacking in many academic departments.

Thinking that my departure from the university signaled the end of my involvement with organized political science, I let my various association memberships lapse when I left teaching. But in the spring of 2000 I attended the annual meeting of the Midwest Political Science Association and participated in a panel about alternative careers for political scientists. We had a lively discussion about graduate education in general and career paths for political scientists in particular. Alas, while my decision to let my professional memberships lapse may seem short sighted, it reflects what often appears as the chasm between academic life and "the real world."

The divide between academe and the rest of world is reflected in (1) how we educate our students, and (2) people's perceptions about the usefulness of a doctoral degree in the social sciences. One graduate student who attended the panel discussion last spring said that faculty in her department discouraged her from pursuing a Ph.D. if she wasn't intent on teaching and a tenure track position. I found her story, while disappointing, quite representative of my own experiences. The panel participants suggested that there are a great many political scientists in a variety of careers outside the university and that the discipline would be well served to expand the options presented to doctoral students.

Academic types have a lot more to offer than a cursory glance at their credentials or newspaper cartoons might lead you to believe. Beyond the obvious research skills required to complete a Ph.D., I believe there are two things about an academic career that facilitate my current position as a legislative policy analyst that might apply to others with similar backgrounds. First, there is the sheer diversity of experience. Second, there is the need to be creative problem solvers.

In a university setting, you are called upon to serve in a variety of ways. Over the years I spoke to student groups and classes, appeared on television for local news broadcasts and political discussions, and spoke to various civic groups such as the League of Women Voters.

The diversity of experience ensures that many academics can multi-task with the best of them. You may be teaching new classes (that always require immense preparation) while simultaneously writing, re-writing, and revising journal articles or conference papers, and preparing/grading exams and term papers. You might also be organizing panels for an upcoming meeting, serving on departmental and university committees, counseling students about study habits, grades, personal problems, and career choices, and writing letters of recommendation for students going to law school, graduate school, or elsewhere.

Creative problem solving comes with the territory of an academic career as well. You must search continually for ways to solve data collection and variable measurement problems. Often, the rigorous scientific method yields minimal findings in the way of statistically significant relationships, so you must look deeper to see what is missing from the equation.

Or data that you thought would be readily available isn't, and you must find another way to measure the relationship you're trying to explore. You look for new ways to present complex concepts to students in ways that help them learn or keeps them interested in classes they are required to take (i.e., American Government). You also look for new methods of testing to reduce the likelihood of cheating.

Evaluation agencies might wish to recruit academics through the Personnel Service Newsletter of the American Political Science Association (it has a list of applied research positions) or the newsletter for the Legislative Politics Section of the American Political Science Association.

Overall, we recovering academics are much more than the sum of the classes we've taught and the articles we've published. Teachers are often, if not always, committed to values beyond their pocketbooks, and they are often seekers of excellence in all things.

A Stranger In a Strange Land

How do you develop a strong legislative oversight function in a country just emerging from 16 years of military rule? How do you develop performance measures in a country with no accountability infrastructure? How do you implement legislative intent when the president views appropriation acts as advisory documents rather than law? How do you develop computer literacy among staff that must share only 20 or so aging PCs among hundreds of legislative employees? How far should you trust the Geckos living in your hotel room to eat any dangerous insects that try to come in?

These were just a few of the challenges that faced our four-person NCSL technical assistance team that spent seven days working with staff of the Nigerian National Assembly in December. It was a wonderful opportunity to help a young dynamic nation move towards democracy, and brought home several truths.

1. We are incredibly fortunate to be living in the USA.
2. A well-functioning democracy relies both on a system of formal laws and on a system of informal rules that elected officials, government officials, and citizens follow. While it is easy to create the former, it is very difficult to develop the norms, understandings, and procedures that underpin our system of separation of powers.
3. There are few shortcuts to experience, and our ways of doing things must be adapted to match the realities facing different nations.

Nigeria faces many challenges. It has the largest population in Africa and great natural resources, including huge oil reserves. However, it also has a history of military intervention in the government, ethnic conflict, and rampant corruption. Nigeria is highly dependent on oil sales, which provide over 90 percent of public revenues; the wide swing in oil prices over the past few years have played havoc with its budget. The nation has good roads but an absence of any apparent traffic laws. It has rural life that hasn't changed in a thousand years but also modern cities; bountiful electric power generation but a crumbling distribution grid that causes power outages many times each day; cultural richness but subsistence farming and incredible poverty.

To put yourself in the position of Nigerian legislative staff, imagine that your legislature passes a budget for its operations, only to have the governor withhold half of the funds. Imagine that revenues increase by 300 percent over budget estimates during the year, but the governor refuses to release information on how the extra funds are being used. Imagine trying to analyze the impact of a proposed bill when no compilation of previously passed legislation exists. Imagine that over half of all government revenues are lost to corruption. Imagine that per capita income is $200 a year, there is no electricity or clean water in much of the country, and there are long-standing ethnic divisions that must be managed to avoid civil unrest. Imagine that you are 18 months into the Fourth Republic, knowing that each of the three previous democratic governments failed within a few years and that each failure resulted in a military coup.

Nonetheless, there is a great deal of optimism in the nation. There is a free and vibrant press that readily criticizes the government. There is a firm commitment to make democracy work this time. There is a great deal of friendliness towards the United States, which is truly seen as the shining example of democracy and development. There was a great deal of amusement about the ongoing recount of the presidential election (much of it directed towards me, the unfortunate Floridian in the group), but also the recognition that our electoral system was working.

We certainly didn't change the world during our trip. Progress in a developing nation is measured in small steps. However, I'm confident that National Assembly staff will start being more proactive in conducting bill analyses and policy research and providing this practical information to Assembly members. With luck and time, in the future we may be able to broaden NLPES to serve legislative program evaluators in Nigeria as well as the rest of the developing world.

Note: Other participants in the Nigeria delegation included Jeremy Meadows (NCSL), Joyce Honaker (committee staff administrator of the Kentucky Legislative Research Commission), and Frank Caggiano (clerk of the South Carolina Senate).

Using the NLPES List Serve

To subscribe: Address an e-mail message to NLPES-L-REQUEST@NCSL.ORG. Leave the subject area blank. In the message area, type: SUBSCRIBE. You will get a "Welcome" message back if you are successful.

To send a message to all NLPES subscribers: Send an e-mail to NLPES-L@NCSL.ORG. Leave three blank lines at the beginning of your message to allow room for the listserve's automatic warning statement.
 

Calendar

A Cross-Reference Guide to Auditing and Evaluation Standards

(Note: Rakesh serves as the NLPES liaison to the American Evaluation Association. Rakesh notes that this article is based partly on a presentation by James Carpenter, audit manager for the City of Tallahassee, Florida, which compared the Yellow Book and Program Evaluation Standards.)

The majority of NLPES member offices conduct their audit/evaluation work in accordance with the Government Auditing Standards,¹ commonly known as the Yellow Book. Depending on the type of work, members may find it useful to supplement the Yellow Book with two other sets of standards:

• The Program Evaluation Standards²
• American Evaluation Association's Guiding Principles for Evaluators³

For those interested in how the standards compare, the accompanying matrix provides an easy way to cross-reference the three sets.

Matrix Layout. The Yellow Book is used as the anchor for this matrix.⁵ Using their own numbering systems as shown below, the other two sets of standards are cross-referenced to the Yellow Book's three categories-General, Fieldwork, and Reporting.⁶

The Program Evaluation Standards are divided into four groups:
 

The Guiding Principles are divided into five groups:

For the sake of brevity, the description of the standards above and the actual standards listed in the matrix are paraphrased.

_______________________

¹ U.S. General Accounting Office, Government Auditing Standards (June 1994).
² Joint Committee on Standards for Educational Evaluation, The Program Evaluation Standards, 2^nd ed. (Thousand Oaks, CA: Sage, 1994).
³ American Evaluation Association (www.eval.org). For a detailed discussion on the Guiding Principles, see Guiding Principles for Evaluators, ed. Shadish, W.R., Newman, D.L., Scheiver, M.A., and Wye, C., New Directions for Program Evaluation, no. 66 (San Francisco: Jossey-Bass, 1995).
⁴ Mohan, R., "Legislative Evaluators: A Diverse Group of Professionals," NLPES News, no. 68 (1997).
Also see Ensuring the Public Trust: How Program Policy Evaluation Is Serving State Legislature. This report contains the results of a nationwide survey of legislative program evaluation offices, jointly conducted by two NCSL organizations-NLPES and the Fiscal Affairs Committee of the Assembly on State Issues. http://www.oppaga.state.fl.us/ ppesurvey.html
⁵ It is assumed that the readership of the NLPES News is familiar with the Yellow Book, and therefore the matrix does not contain descriptions of any Yellow Book standards.
⁶ Specific Program Evaluation Standards and Guiding Principles are cross-referenced only to the relevant category of the Yellow Book, and not to a specific standard

Office Happenings

Arizona