• Mission & Governance
• Services Overview
• How to Get Involved
• Standing Committees
• NCSL Foundation

• Issue Areas: A-Z
• State-Federal Relations
• NCSL Standing Committees
• News from D.C. and the States

• About State Legislatures
• Elections, Campaigns & Redistricting
• NCSL's StateConnect Directory
• Web Sites

• Online Guide to NCSL Services
• Staff Sections and Networks
• Meetings & Professional Development
• Staff Directories

• Meetings Calendar
• Online Registration
• Meetings Home Page
• Legislative Summit

• Bookstore
• State Legislatures Magazine
• NCSL's StateConnect Directory

NLPES News
National Legislative Program Evaluation Society

Inside the NEWS

Chair's Corner
Calling New Legislative Staff to Seattle, WA
Surveying Via the Internet
Hawaii Profile
Interviewing Basics...Winging It Won't Work
There is No Time to Waste
Mentoring...Wisconsin Style
Legislative Staff Management Institute
Intern Programs...From Textbook to Reality
Liaison's Corner
Strategic Issues for Legislative Oversight Organizations
Office Happenings
Calendar of Event

Chair's Corner

For a lot of us, however, I suspect that legislative program evaluation seemed less important on September 11 than it did on September 10. Interesting topics that people in my office were evaluating-such as state employee health benefits, the economic status of current and former welfare recipients, water pollution regulation, and the solvency of the state's unemployment insurance fund-suddenly looked a little smaller to me. In the hours after the attacks occurred, legislative program evaluation appeared to be far removed from our country's immediate challenges.

Before long, however, I started to see things differently. Maybe it was the repeated references I heard to a GAO report that had exposed lax airport security practices before the terrorist attacks occurred. Or maybe I started to hear more questions raised about the effectiveness of America's immigration policy, border security, intelligence operations, and public health system. I was reminded of the importance of having independent fact-finders looking at public programs and how they have been implemented.

Or, closer to home, perhaps it was the economic recession that reminded me of our offices' continuing relevance. Not many months ago, Minnesota's executive branch budget director spoke of the state having a "boatload of money;" these days, the boat is on a sea of red ink. When state money is tight, legislatures need sound information and advice about program efficiency, program results, and policy options. We hope and expect that our offices will be an important source for this.

(Unfortunately, even highly regarded evaluation offices often have to face budget cuts during economic downturns, as some NLPES member agencies have painfully experienced in the past few months.)

Legislative program evaluation offices are far from the front lines of the war on terrorism, but recent events might lead legislators to request evaluations in topic areas that did not previously seem very pressing. Perhaps some NLPES offices will be asked to evaluate the security of hazardous materials transportation, the adequacy of state emergency response plans, or the way that state agencies have responded to previous emergencies or threats.

Even if we are not asked to study topics directly related to terrorism, our "regular" evaluation work is important at a time like this. Our reports, ranging from compliance reviews to broad policy studies, can be building blocks for lean, focused, effective, responsive, accountable governments-and, for governments that are ready to address issues needing attention.

When parts of our government scramble to respond to crises that were hard to imagine months earlier, there is still a need for the trains to run on time elsewhere in government. Our offices are a small part of helping that happen-and a no less important part than they were before September 11 or the recent recession.
 

---

Calling New Legislative Staff to Seattle, Washington

Through a combination of lectures, hands-on case studies, questions, and discussions, participants develop an understanding of program evaluation. Max and Jim typically describe program evaluation as a problem-solving process. Although no two states may be alike in their approach to evaluation, there is a common ground for all evaluations. Evaluation is to be systematic. Evaluation involves a continuum of activities from careful, accurate description of a program or process to determining the worth or merit of some program or activity. What makes it hard to generalize about evaluation is that there are no rigid rules. Evaluation is often creative design, and there is no ideal standard. Max and Jim use the term "disciplined inquiry" to help explain this unique aspect of program evaluation. In its basic concepts, program evaluation is determining "what is" and comparing it to the "what should be." The evaluation process gets us from where we are now to where we want to be.
The materials for the Track revolve around actual legislative requests for program evaluations. The Track examines the entire process of a program evaluation from intake of the request, to background research, to surveying and planning, to fieldwork and analysis, to finding and conclusion development.

Bringing together individuals from different states and backgrounds is essential to the success of the Track. These people interact, ask questions of each other, and provide practical information. The contacts they make are invaluable in future evaluation work. What makes this Track very beneficial is that Max and Jim are not talking theoretically-they are talking from practical examples with practical problems and practical solutions. Once the participants understand that program evaluation is an art as well as a science, they understand the important role they play in program evaluation. That's because the most important part of program evaluation is the person doing it. These new staff members are bringing to the table a toolbox full of different skills. Max and Jim try to relate how they can use that toolbox in doing a program evaluation. The easiest way is to relate it directly to problem solving, because that's what evaluators are---problem solvers!

This year's NCSL Skills Development Seminar will be held in Seattle, Washington, on August 14-17, 2002. Executive directors of legislative program evaluation and audit shops should consider sending their new evaluators and auditors to this worthwhile training experience.
 
 

---

Surveying Via the Internet

When surveys are called for during a program evaluation, evaluators typically attempt to design questionnaires that are easy for respondents to complete. This increases the chances of receiving a good response rate.

A relatively new technique for enhancing survey responses is allowing the population under study to respond electronically, via an Internet survey. For respondents, on-line surveys can lessen the amount of time needed to complete and return responses. For evaluators, on-line surveys can offer quick turn-around times and accurate response data. They save administrative time by eliminating the need to enter data and check for data-entry errors.
In the Minnesota Office of the Legislative Auditor, we began using on-line surveys in 2001. Here are a few things we learned from having a couple of these under our collective belt.

Know your population. On-line surveys are only useful if the intended recipients have easy access to computers and the Internet. For some of our on-line surveys, we knew that a share of the likely respondents would not have computer access. Consequently, we prepared both a traditional paper survey and an on-line one asking the same questions. In the cover letter mailed out with the surveys, we let respondents know they had an option to use their computer or return the paper survey via regular U. S. mail.

One group we surveyed this past year was facility managers from local governments; however, in many of our state's small cities, facility managers were head custodians who rarely used computers or for whom city policy proscribed use of the Internet while at work. No surprise, then, that of facility managers in 200-some cities, only 9 percent responded using the on-line survey.

On the other hand, in a survey of county technology directors and school district superintendents, 57 percent of respondents replied using the Internet. With more than half of the questionnaires automatically posted to a database, we saved considerable time avoiding redundant data entry and data clean-up.

Determine whether on-line surveys will add value. Mountaineers can get away with climbing mountains merely because they are there. Evaluators need better reasons to do most things. Although on-line surveys are fun, somewhat new, and offer some real benefits, remember that they are just one tool in an evaluator's toolbox.

Only use on-line questionnaires when your preparatory work indicates that you will improve your survey process by doing so. On-line surveys are the equivalent of a mailed survey, but with a different medium -- the Internet and computers instead of paper and a No. 2 pencil. If a telephone survey or face-to-face interviews are better suited to answering your research questions, reserve the use of an on-line survey for another project.

Planning yields good results. While on-line surveys can be real time savers during data preparation and analysis, they require up-front planning. Set aside sufficient time in advance of launching the survey to plan, design, and test the on-line instrument.

Rely on technological experts. Several software products are available for evaluators looking to begin Internet-based surveys. (See, for example, the February 8, 2000 PC Magazine for a review of a half-dozen such products.) Our office purchased one of these applications, but we still needed to rely on our information-technology (IT) staff to prepare the survey-related Web pages and database.

Design the survey instrument with the user in mind. On-line surveys have design advantages over traditional paper surveys. Good design will enhance the ease and quickness of completing the survey. Evaluators can design the Web pages so that respondents do not mistakenly fill out questions they are supposed to skip. Respondents with a certain response pattern can be directed (behind the scenes) to one set of questions while other respondents go to a different set. For example, facility managers who have automated building-monitoring systems could respond to questions about those systems, but others who do not could avoid reading those questions altogether.

Designing a survey for ease of use goes beyond the survey instrument. Instructions for gaining access to the on-line survey should be simple and straightforward. Nothing will frustrate a potential respondent more than trying to open a Web page but receiving instead an error message indicating the page is unavailable. Submitting the on-line survey upon completion should also be easy. Make it a "one-click" operation.

Assume that at least some of the respondents will have minimal Internet experience. Then make your design decisions based on what will work for the least experienced.

Test before pretesting. As with any questionnaire, the evaluator should pretest the on-line survey instrument with a subset of the population to be surveyed. If both a hard copy and on-line survey are to be used, some of the testers should fill out the paper version while others complete the on-line version.

Even before that, though, on-line surveys require thorough in-house testing. After working with IT staff to complete the survey design, evaluators should test the survey instrument under various scenarios. Use different Internet browsers and screen sizes to see how well the survey can be read by those who have old versions of browsers or small monitors. Recruit someone in your office who is not involved in the project to fill out the survey as a way to test navigation around the site. If that person reaches a dead end or encounters problems, you can make changes before the real users see the questionnaire.

Track the on-line usage. During the time the on-line survey is available to users, someone should track Web site traffic daily. Among other things, this will reveal the number of users who tried unsuccessfully to reach the on-line survey.

In tracking the first day or two of activity for one of our on-line surveys, we observed a large number of hits that had received "page not found" error messages. A quick examination of the uniform resource locator (URL), or Web page address, entered by these users revealed the problem. All of these users had entered a period after the URL. They did this because in the written instructions accompanying the survey, we listed the URL for the questionnaire as "www.auditor.leg.state.mn.us/ecity.htm." -- including a period after the "htm" term because it marked the end of a sentence. Inserting the period at the end of the URL in their browser, however, prevented users from viewing the page. Because we had been monitoring on-line usage, we could make some changes to prevent the problem from recurring.
 

---

Update: NCSL Bill Status Project

The goal of the task force is to develop a methodology for what will be requested from the states, as well as oversee the development of the web application. In addition, this task force will work with the NCSL Special Committee on Information Management to establish relationships with the states that will allow NCSL to collect bill status information daily. Users could then search the site to find bills of interest and track those throughout the year. Updates on the project will be provided in future issues of NLPES News.

---

Hawaii Profile

What is the price of paradise? One of the worst things about Hawaii is its high cost of living. In December 2001, a gallon of regular unleaded gasoline was $1.75; a loaf of bread cost $2.50; and a gallon of milk was hovering just under $5.00.

Do people in Hawaii really live in grass shacks? No! We want the other 49 states to know that contrary to popular belief: (1) Hawaii is indeed a part of the United States--you do not need a passport to come here, (2) we use U.S. dollars as our currency, and (3) we do not paddle canoes to work.

What is one of the most interesting facts about our staff? More than a handful had never seen snow until adulthood and at least one has never experienced that wondrous winter wonderland phenomenon to this very day.

Why are other auditors jealous of Hawaii auditors? Occupying the fourth and fifth floors of Kekuanaoa Building, we are fortunate to each have offices with windows. Why is that so special? The view to the east is of historic Kawaihao Church, the majestic Koolau Mountain Range dominates the northern view, downtown Honolulu is to the west, and those with south views have the opportunity to gaze over the Pacific Ocean. We can all see palm trees doing a hula, French blue skies, and fluffy white clouds...365 days a year.

Got criteria? Other common office phrases: "What's GAGAS?" "What's for lunch?" "Did you check the MOG (Manual of Guides)?" "How many IR (independent review) points did you get?" "Where's pau hana this week?" (Note: Pau hana is loosely translated into "finished with work" and conveys the idea of Friday happy hour.)

Is Hawaii's State Auditor really Martha Stewart? Marion Higa taught Martha everything she knows! Marion is a couture seamstress (she made her daughter's wedding dress and sews her own suits), has a garden that produces very pretty miniature pink roses, makes her own jewelry, and is a talented baker to boot!

What's "Felix?" It's one of the most pressing issues currently facing the Hawaii State Legislature. The 1994 Felix consent decree resulted from a lawsuit alleging that qualified handicapped children were not receiving the educational and mental health services they needed and that the State was in violation of two federal laws. Legislative concern over unclear requirements and exponentially rising costs related to the decree (from $181 million to $302 million over five years) resulted in Hawaii's Auditor issuing two Felix reports and providing support to the legislative committee charged with investigating the status of the decree.

You gotta be kidding! We couldn't make this stuff up if we wanted to. Staff report that the following events actually occurred during the line of auditing duty: (1) conducting fieldwork in the heart of a high security prison with prisoners walking to and from their cells (while guards on duty were observed sleeping), and (2) interviewing a state official in her office while trying not to notice the nude portrait of that official on the wall behind her desk.

What's the second best thing about Hawaii? The diversity of its people. Our office is a true representation of a melting pot of ethnicities and cultures. Our staff is Japanese, German, Chinese, Irish, Hawaiian, Korean, Scottish, Native American, Russian, French, and Portuguese, but most certainly 100 percent American!
 

---

Interviewing Basics: Winging it Won't Work

Analysts at the Texas Sunset Commission rely on personal interviews as a key way to gather information when reviewing agencies for the Texas Legislature. Good interviewing skills are critical to developing meaningful and fully documented recommendations. Interviews can help establish a good working relationship with an agency up front, which is important preparation for when serious or controversial issues must be addressed later.

As part of the training for new analysts, Sunset staff developed the following 15 basic tips for conducting a well-organized and productive interview that should prove helpful for other analysts and auditors doing similar work. Of course, there is no single "right" way to ask a question or conduct an interview. Remember to keep an open mind, an overall positive attitude, and respect for the people you're interviewing.

Interviewing Do's and Don'ts

Do your homework. Background research is a key source of questions. An agency's enabling laws, prior reports, appropriation bill, and Internet research provide excellent resources to help prepare for an interview. Keep a running list of questions as you do your research.

Designate a lead interviewer and chief note taker. Interviews should be done with at least two staff, one should lead while the other takes notes and asks follow-up questions.

Outline what you plan to cover. All good interviewing requires some outline of what you plan to cover. This outline can vary from a list of questions to just general topics you want to discuss, depending on your experience and the nature of the particular interview. Initial background interviews tend to be more routine, while follow-up interviews often must be carefully planned to obtain the right information. Organize your questions in a logical sequence.

Have a variety of questions and know when to ask them. Open-ended questions are broad in nature and leave a lot of freedom to respond. Closed questions are more restrictive and usually ask for specific information. Generally, your questions should look like a funnel -- open to closed. However, it can also be good to close with a broad, open-ended question, assuming there is time for the interviewee to respond fully.

Get off to a good start. Make introductions first - exchange business cards. Orient the interviewee and build rapport, but keep it short. Outline the purpose of the interview. Explain how long it should take. Offer an overview of the purpose or scope of your review.

Actively listen. Don't do all the talking -- your goal is to get information, not give it. One helpful rule of thumb is the 70/30 rule in which the interviewee should talk about 70 percent of the time, and the interviewer not more than 30 percent of the time. Still participate in the conversation - don't just ask questions and take notes. Show interest in the answers by asking follow-up questions.

Allow appropriate silences. Some people will offer additional information if you don't jump to the next question. Also, an appropriate pause gives you and the interviewee time to gather your thoughts and keep on track.

Ask for clarification. Ask questions to ensure that you understand the discussion and answers given. Don't assume everything is OK just because the agency staff says it is. Question their approach. If you're still having difficulty, recap or summarize the discussion to ensure that you got it right.

Avoid snap judgments. Your job is to question how an agency is doing its business. You want to approach what you are told with an appropriate level of skepticism, but it is safer not to express your immediate opinions until you have had time to give them careful consideration.

Be considerate of your interview partners. Don't contradict your fellow interviewers during an interview and don't answer questions they ask. Your colleagues' interest lies in hearing the perspective of the person being interviewed, not yours.

Be on time and don't run over. Nothing gets things off to a worse start than if you are late - the interviewee may be irritated and you may feel rushed or frazzled. Also, watch the time. Try not to run over the amount of time you originally allotted.

Close correctly. Thank the interviewees for their time and effort. Ask if you overlooked anything. Ask for suggestions on who else to talk to for different perspectives on the topic. Avoid bringing up new topics at the end - you can always follow up later.

Recap requests. Clearly and concisely request information. Recap all requests at the end of an interview. Set reasonable but clear time frames. Keep a record of requests and follow up if needed.

Write it up. Decide ahead of time who will write a summary of the interview. Highlight the key information you gained without getting bogged down in details. Note any issues that you want to investigate further.

Ask for feedback. After an interview is a good time to seek out feedback from your project manager or fellow team members. Did they feel the interview went smoothly? Did they feel that you got the information needed? Could you have structured the interview more effectively?

Approaching Tough Questions in an Interview

A spoonful of sugar. Balance a challenging question with praise and reassurance. Word questions to show an understanding of the difficulties a person faces.
Example: "Overall it appears that your agency is doing a good job, but one concern we have is..."
Example: "Many agencies we have looked at in the past have had problems with...Have you experienced those?"

Test the water. Suppose you are considering a controversial recommendation and you want to see what the agency thinks. Discuss your idea hypothetically, preferably couched in a set of different alternatives.
Example: "I understand that this is how you currently operate. What if the statute were changed to..." "What if your budget was significantly increased (or cut back)? What would the impact be?"

Listen to their side. You can distance yourself from a challenging point of view by attributing the question to someone else and offering to hear the interviewee's side of the issue.

Top Ten Reasons Agency Staff Like to Have Evaluators and Auditors Around

10. Can split the coffee fund more ways
 9. Nice to have someone around to blame missing files on
 8. Never met ____ (fill in appropriate name of co-worker)
 7. Really believe we are there to help
 6. Chance to get even with co-worker
 5.Can claim that someone else now has worse working space
 4. More bait for the dating pool
 3. New audience for all those old jokes
 2. Can't help it, they like everybody
 1.Those government issued ID badges are a real turnon

---

Small Shop Talk
There is No Time to Waste

The West Virginia Performance Evaluation and Research Division (PERD) has the challenge of producing quality performance reports with a small staff of 20 employees, two of which are support staff. The West Virginia Legislature requires PERD to review between 30 to 35 programs in a given year. For the year 2001, we reviewed our largest workload of 38 programs, of which we were able to complete 36. The remaining two audits were carried over to the list of audits for 2002. These reviews include large multi-million dollar agencies and small licensing boards with budgets as small as $3,000. We also evaluate (update) agencies' compliance with our recommendations of past audits, and we review agencies not on the Sunset schedule by special request from the Legislative Auditor. In addition, we have a Sunrise process that requires us to review the need for a new licensing board.

In the past, we typically encountered the unpleasant experience of approaching the holidays towards the end of the year with a heavy workload still incomplete. This created a stressful environment and long evenings in the office. However, over the last few years we have been successful in dramatically reducing these stressful situations by making the evaluation process more efficient without compromising quality.

The management challenge that PERD has is to not only complete all of the audits that are required by statute, but to avoid overlooking something important by reviewing as much of an agency as possible with limited resources. PERD doesn't have the luxury of some large shops that may be able to place three or four analysts along with a manager on a large audit, and be able to spend one or two years on the audit. Typically, PERD assigns one analyst to a manager for an audit.
PERD has hurdled this obstacle by doing several things, some of which we are still working on. The most important of these is using an outcome-based approach to planning an audit. When we have not been told specifically what to look at and problems do not immediately surface, we try to determine the scope quickly by identifying the outcomes and then determine what must work well in the program in order for the outcomes to be achieved. When we look at what is critical to achieving outcomes, it reduces the risk of missing something important. We also use hypothesis testing by attempting to answer important questions concerning the program.

Another important measure to increase efficiency is the development of a comprehensive procedural manual to use in addition to the Yellow Book. Although the manual is about 75% complete, we are reaping the benefits of achieving uniformity in data collection and report format and content. In addition, analysts can work with minimum supervision on relatively small audits. We can also train new analysts better through the manual.

Our efficiency has also improved as we have taken the approach of having the agency take on a greater amount of the auditing work. For example, in the past when we updated an agency's compliance with our recommendations, we assumed the burden of proving the level of compliance by determining what information was needed and what questions to ask. Now, we place the burden of proving compliance on the agency by requiring it to provide us with what it believes proves its level of compliance. This dramatically reduces our work. Another example is when we audit regulatory boards.

We are required to look at several areas including whether the board is needed to protect the public. We have developed a system in which we request standard information to be made available at the entrance conference, including evidence from the board proving it is needed to protect the public. When we have left the entrance conference, we usually have everything we need to complete the audit.

This year it is our goal to provide the agency under review with a list of our audit objectives and areas of focus. We have not done this enough in the past. This is intended to promote cooperation and eliminate apprehension and surprises. By knowing the focus of our review, the agency may provide us with information that we may not have thought of requesting. Also, over the last year we have provided the agency with a draft copy of the report and given them several days to review it. An exit conference will only take place if the agency has any substantive changes to suggest. In some cases the agency is in complete agreement with the report and finds no need to have an exit conference. In other cases, the suggested changes can be resolved over the telephone.

Overall, PERD is making every effort to find ways to make the audit process as efficient as possible. We have the policy of avoiding open-ended deadlines for agencies to provide us with information; we have standardized certain correspondence; and, we try to make entrance conferences productive by requesting information be made available so that we can discuss it while we have their attention. We want to make the best use of our time given the amount of work that we have to do. Experience will continue to teach us the best methods of auditing as long as we are open to make the necessary changes.
 

---

Strategic Issues for Legislative Oversight Organizations

Public and private clock-cleaning sessions are a way of life for Directors or Auditor Generals, particularly if our reports are delivering timely and emphatic messages. Nothing is more disconcerting than receiving a phone call from an angry and powerful legislator hacked off about one of our reports. Usually, in hindsight, we could have diminished the criticism with a slightly different tactic during the project or if we had worded the report with a little more finesse. We know we are not in the public relations business. Such encounters occur irregularly and are generally temporary. Although they may receive disproportionate attention and cause lost sleep for a few days, we should not let such matters distract us from more important issues, which if not dealt with appropriately could lead to the abolition or major curtailment of a legislative oversight agency.

Isolation from the legislative institution. Independence is critical to the perception of our work and is required by our standards. However we should not abuse our independence and act contrary to the best interests of the legislative institution. We should respect the legislative institution and not distance ourselves from it.

Lack of diligence. We have to try our best to answer questions and work through challenges and not be daunted by them. We simply cannot respond repeatedly: "We were unable to answer the questions posed to us because our staff lacked sufficient expertise, time, and valid data." The public and legislators presume that we have the expertise and that we are ready to help solve problems in state and local government.

Failing to communicate. Our reports have to be well written, easily scanned, and on time. Presentations should consider the complexities of communicating within a political institution and should be delivered with the message up front and with the capacity to respond with details if requested.
Engaging in sophistry. Our organizations should be honorable. We should use our organizational reputations, expertise, and communications ability responsibly.
 
 

---

Mentoring....Wisconsin Style

Why did we develop our mentoring program?
In 1998 the Wisconsin Legislative Audit Bureau (LAB) implemented a mentoring program for our newest staff. Staff developed the program to supplement a two-day orientation program, a series of in-house courses, and the on-the-job training offered by immediate supervisors. Management agreed to implement the program, agreeing that by better acclimating new staff to their work environments, staff should be able to make earlier and more meaningful contributions to their audit projects.

How did we develop our mentoring program?
A small group of staff, including myself, first considered the needs unmet by current efforts that such a program could fulfill. We then met with individuals who ran mentoring programs in other state agencies, researched literature on mentoring, and received training from a representative of the International Mentoring Institute. Drawing from each of these varied resources, we developed a handbook to guide the program.

Who is involved and how does the mentoring program work in our office?
To oversee the mentoring program, and to match each protégé (new staff) with a mentor, we have a committee. The committee has a manager, a supervisor, and a staff person from each of the two primary divisions in our office: program evaluation and financial audit.

All new employees participate in the program, while mentors either volunteer or are asked to participate in the program. Protégés and mentors complete interest surveys, which the committee uses to match protégés and mentors. Each match is designed to last six months.

Mentors receive training to help them identify ways to be helpful to a protégé and develop ideas for activities to complete with a protégé. In addition, mentors learn what the program and LAB expects of them. For example, mentors learn longer lunches or a few emails back and forth with their protégé to address office-related questions are time well spent by the pair. In addition, mentors are encouraged to arrange group activities, such as capitol tours, or training sessions from our Legislative Reference agency.

Written goals and periodic evaluations prepared by the protégés and mentors create flexibility and accommodate individual needs, but also provide useful structure and feedback to the program.

What have we learned?
Through suggestions made in the final evaluations participants complete, and also through ideas generated by the committee that oversees the program, we made some modifications to the program.

Mentors and supervisors need to talk. Some staff and managers were concerned that the responsibilities of the mentor would supersede, or worse, contradict the directions of the audit supervisor. Therefore, each mentor meets with his or her protégé's direct supervisor to discuss tasks or subjects that the supervisor hopes to personally address with the protégé, and identify any tasks with which the mentor might be able to offer assistance. For example, a mentor and protégé could attend a legislative hearing together, but a mentor would not inform a protégé about how to write a workpaper or document an audit finding.

Make matches as soon as possible. We have begun to include the form used for matching protégés with mentors in the packet of information that is sent out to new employees before they begin work.

The matches may not always work. Much of the literature indicated that due to personality differences, or other conflicts, mentor/protégé relationships might occasionally fail. So far, we have been lucky enough to avoid this problem, or maybe just really good at matching people! But, the one-month evaluation form does inquire about the satisfaction of both the protégé and the mentor--giving staff the opportunity to be re-matched.

What do participants say?
The success of each relationship depends largely on the individual effort of each participant. Generally, staff who have participated in the program indicate that it helped to give them an early sense of their role at LAB. In addition, staff have offered the following comments:
"The mentoring program is valuable even for new employees with previous professional experience. The unique nature of legislative program evaluation often requires even experienced employees to gain new knowledge, formally and informally. Mentors can provide invaluable assistance in this process."

"My mentor introduced me to others and helped me get to know LAB. Now that I am a mentor I feel that I owe that to a person I may be mentoring."
"Additionally, a mentor can be a resource for questions with which a staff person does not feel comfortable asking (or does not want to 'bother') their supervisor."
 

---

Legislative Staff Management Institute

Since 1990, when the first class of thirty-four legislative staffers convened in Minneapolis, the Legislative Staff Management Institute has grown to be a uniquely valuable resource, serving to train, motivate, and inspire those engaged in supporting the work of the state legislature.

Program Specifics

Each session of the Institute brings together about 25-30 senior legislative staff for a concentrated, intensive 11-day course of instruction on the campus of the University of Minnesota. The Institute is held in July or August each year, scheduled at a time that does not conflict with the NCSL Annual Meeting. Faculty is drawn primarily from the Hubert H. Humphrey Institute of Public Affairs, one of the leading public policy and planning institutions in the nation. For all but the first year of its existence, LSMI has been under the direction of Dr. John Brandl, who previously served in both houses of the Minnesota Legislature.

Prospective participants apply for admission and are selected on the basis of aptitude, commitment, and staff position. Preference is given to applicants who have significant management responsibility or are likely to assume such responsibility in the near future.

Tuition is $2,300 and includes lodging, most meals, all instructional materials, and access to many University facilities. Partial scholarships are also available. Financial support for the Institute is provided by the NCSL Foundation for State Legislatures.

Curriculum and Instruction

The LSMI curriculum is designed to help senior level staff managers with a commitment to the legislative process develop their abilities in dispute resolution and conflict management, management of the legislative process, leadership, policy analysis, strategic planning, and use of information systems. It is intended to provide participants with opportunities to enhance their management skills, become exposed to new ideas and techniques, and forge relationships with colleagues with both similar and different legislative responsibilities.

There are five primary curriculum areas:

Managing conflict, bargaining effectively, and getting to "yes."
Using the research and teaching skills of the Humphrey Institute's Conflict and Change Center, conflict management simulations will build the participants' ability to successfully "get to yes."

Managing people and organizations in the legislative setting.
Drawing on teaching cases and materials set in state legislatures, this part of the curriculum will consider how top-level legislative staffers organize and deploy their resources, develop systems for control and information, and deal with the inevitable conflicts among political and non-political staffs, elected officials, legislators and executive branch personnel.

Leading, motivating, inspiring, and creating.
Drawing on the research and teaching skills of the Humphrey Institute's Reflective Leadership Center, the curriculum will help the participants examine and improve their leadership skills.

Thinking and acting strategically in order to overcome the inevitable obstacles and limits of the legislative arena.
Planning exercises will help the participants think strategically about their own careers and futures, and about the legislative process and policy agenda setting.

Managing and using technology effectively.
Employing proven techniques in the computer labs of the Humphrey Institute, this instruction will enhance participants' ability to use technology to manage information in legislatures effectively. Participants will work on cases from their personal experiences with respect to design and management of computer systems.

LSMI is more than a skills development or management training seminar. Student participation is not just encouraged; it is required. Real-life work experiences of participants provide not only a frame of reference, but also the raw material for class discussions, case study topics, and group exercises. Faculty members facilitate discussion by providing analytical templates that are used to frame discussion. It is this interactive process which fosters an intense learning process and leads students to a more complete understanding of the dynamics of the legislative process and their role in supporting the process.

The LSMI Experience

The physical setting for the Institute is the west bank of the University of Minnesota's Minneapolis campus. The approximately two dozen or so students comprising each LSMI class are housed in Middlebrook Hall, one of the more comfortable campus dormitories. Classes are held a short distance away in the impressive Humphrey Center. It is the dorm, however, which becomes the focal point for the participants. Most meals are taken in the cafeteria, and the participants gravitate to the common areas in the evenings to socialize, continue the day's classroom discussions, or just "hang out."

It may not seem that spending two weeks (one weekend) living in a college dormitory would be a particularly enjoyable experience, particularly for someone who may be years or decades removed from campus life. However, a strong case can be made that both the dorm experience and the 11-day duration of the program are integral parts of LSMI and add much to the richness of the experience. Most participants have found that living in Middlebrook Hall provided an opportunity to get to know their fellow participants, identify common concerns, and share experiences. The camaraderie and collegiality that result carry over to the classroom and help create an atmosphere of trust, which, in turn, facilitates the climate for learning.

Similarly, the program length could be viewed as a barrier to attracting more participants. Although 11 days is a considerable period to spend away from work, home, and the outside world, it is time well spent. That amount of time is required both to cover the essential course material and to allow participants enough time to shift gears, put their daily work and personal responsibilities aside, and become fully engaged in learning.

The LSMI experience is not all work and no play. Although students are expected to come to class prepared to discuss each day's reading assignment, there is still ample free time to relax, unwind, and enjoy both the campus and the Minneapolis-St. Paul area. In past years, participants have had the opportunity to choose from a wide variety of social events, including an evening at the Guthrie Theatre or Orchestra Hall in Minneapolis, attending baseball games played by the Minnesota Twins or taking advantage of the many entertainment options in the culturally vibrant Twin Cities area. For die-hard shoppers, the proximity of the mammoth Mall of America has proven to be an irresistible lure. All of these activities become part of the lasting memories of the LSMI experience.

Since the first session in 1990, at least thirty-one legislative program evaluators and auditors have attended LSMI.

Since attending LSMI, I have incorporated a number of the approaches I have learned at LSMI to enhance our agency's training efforts, including incorporating these approaches into formal training courses for new staff. LSMI provides a unique educational experience for attendees that is not available through any other program with which I am familiar.Paul Stuiber, Program Evaluation Director, Wisconsin Legislative Audit Bureau

I feel strongly that LSMI was an important experience. I thought it was so valuable to have the time for a few days to step back and reflect on the meaning of government and government service, learn from persons with similar jobs in other states, and experience renewed commitment to our profession. Kathy McGuire, Florida OPPAGA

Since attending LSMI, I have applied the oval mapping process in a number of situations. The technique, among others that we tried at LSMI, was particularly relevant when presented in the legislative context. LSMI has provided some basis for me to assess where I am from a career perspective and in which direction I want to head. Van Lee, Hawaii State Auditor's Office

I am a strong supporter of LSMI and really enjoyed my two weeks there. I found Bob Terry's leadership sessions to be especially helpful. Leadership is what our jobs as agency managers is really all about. John Brandl, LSMI director, is an excellent thinker and his perspectives as a former legislator make his contributions to the program invaluable.Don Bezruki, Wisconsin Legislative Audit Bureau

I have, on occasion, revisited the specific strategies for conflict management and dispute resolution presented at LSMI. But, the long-lasting parts of the training continue to be the insights I gained into my character and others---knowing oneself as a leader was a wonderful exercise. As a result of LSMI, I also have a continued appreciation for the differing roles of others in the Legislature. It was wonderful to have a broad, national vision of legislative service, if only for a short time. Janice Mueller, State Auditor, Wisconsin Legislative Audit Bureau

As a result of LSMI, I have learned how to conduct better meetings and be more effective when discussing system changes as a result of the John Bryson session on SWOT and the Thomas Fuitak session on negotiations. As a result of LSMI, I have a number of close friends in distant states that I can call upon for advice. Incidentally, most have been promoted within their legislatures! Most legislative staff enter the program a bit skeptical about the program's duration, dorm life, and discussion format, then become quickly convinced of the benefits. Ask any LSMI graduate!John Turcotte, Executive Director, Florida OPPAGA

For more information contact:

Kart T. Kurtz, Director of State Services
National Conference of State Legislatures
303-364-7700, extension 118
karl.kurtz@ncsl.org

John Brandl, Dean
Humphrey Institute of Public Affairs
612-625-0669
jbrandl@hhh.umn.edu

---

Intern Programs...From Textbook to Reality

While offices can certainly benefit from the work of interns, a primary goal of an intern program is to provide meaningful experiences to assist participants in developing usable and marketable skills for the future. This article summarizes the experiences of six interns and how reality differed (at least a little) from the academic environment.
 

Aaron David, Michigan Office of the Auditor General
During the summer of 2000, I interned with the Michigan Office of the Auditor General. I was very excited to get the opportunity to intern for the Auditor General because that was the internship that I had set all of my effort into getting.

The first thing about my internship that I was excited about was the fact that I would be doing the same work as the full-time staff auditors and not doing "intern" work. I got placed on a performance audit of the Michigan State Police, Motor Carrier Division. Before this internship, whenever I though of auditing, I thought of compiling and verifying numbers and projections. Performance auditing opened my eyes to something that I had yet to consider. I admit, at first I struggled to see how or why performance auditing was important or useful. As the weeks passed, I found the performance audit picture becoming less and less cloudy. I began to see our purpose and how what we did could improve the effectiveness and efficiency of the performance of state government operations. I also considered what state government might be like without agencies having the benefit of these audits. Sometimes the simplest things go overlooked because you only see things from one perspective.

Performance auditing brings a new perspective into the picture and helps the public determine if their tax dollars are being maximized. One of the most useful lessons that I learned about performance auditing was that our primary goal was to help the agency improve its operations.

Terry Dykhouse, Michigan Office of the Auditor General
I am majoring in accounting and minoring in finance. The most surprising thing that I encountered during my internship is how different my assignments are in comparison to my college classes. By being on a performance audit, I have dealt very little with the financial aspects of the organization that we are auditing. In school it seems that the financial aspect of accounting is stressed. During my internship, I seemed to do a lot of reading and writing. Not to say that I do not do a lot of that in college, but in my classes I deal almost exclusively with income statements, balance sheets, etc.

Overall, I had a good experience during my internship. It gave me a perspective that is not taught in college.

Richard Gurley, Tennessee Offices of Research and Education Accountability
I really feel that my college degree prepared me for my work at the Offices of Research and Education Accountability. However, I believe that I learned as much about research methods and policy areas during my internship as I would have in graduate school. In fact, I often thought that my internship was just like being in school-I study areas, write reports, and develop presentations---except, I get paid for it instead of having to pay.

One observation that I made during my internship is the poor condition of data collection in state government. Sometimes I enjoyed the adventure of scurrying around all over creation looking for needed data. That experience reminded me of traipsing from floor to floor in my college library to find more material for a project. But, in college I would always tell a professor that I could not find information on a topic and we would simply find a new subject for that research paper. That is not an option when legislators are making law regarding a specific policy area.

I learned during my internship how quickly priorities change. A committee asks our office to study an issue and report back to them as soon as possible. By the time we get halfway through with the assignment, they permanently table all the bills dealing with the issue that we were researching. I did not realize how much legislative researchers do that never makes it into print.

I believe that I am a lot smarter as a result of having interned with the office. It was gratifying to see the Legislature turn to our office, listen to us, and enact legislation based on information we had developed.

Mike Henthorn, Colorado State Auditor's Office
I obtained a bachelor's degree in criminal justice with a minor in accounting. Although some of my criminal justice courses gave me some helpful skills, such as the proper way to conduct an interview, the accounting classes that I took for my minor are totally opposite of the accounting that is related to my internship. In college, the focus seemed to be on debits and credits and how to do the "books" for small businesses. When I encountered accounting issues during my internship, they were related to governmental accounting.

Overall, my college experience did not prepare me for the assignments that I encountered as an intern. I believe that what you learn on the job is more beneficial, in most instances, than what you learn in the classroom. However, I do think that college instills the tools a person needs to be successful in the "real" world. Completing college takes a lot of hard work and dedication, as does your career. For myself, this is the area that college assisted me in preparing for the "real" world.

Roxanne Matthews, South Carolina Legislative Audit Council
I have learned that "theory" and "practice" are very different. Sure, theory is necessary in academia, but once you leave those hallowed halls of the university, it really amounts to a "hill of beans." Real world application-such as that provided by an internship-is much more practical because it is hands-on and actually quite interesting.
There has never been a dull moment during my internship at the council because I am constantly working somehow with almost all of the ongoing audits. I have truly enjoyed being an intern and have learned immeasurable amounts because of this wonderful experience.

Robbie Vance, Mississippi PEER Committee
My internship was a very good learning experience for me. The work has really related to the classes that I have taken in college. On many occasions during my internship, I used skills that I thought I would never use. The things that I thought were not important when presented in class proved to be important in the work environment. I have also developed contacts within state government that I can use when I graduate.
 

---

Liaison's Corner
NLPES Liaison Committee Reaches Out to Other Professional Organizations

Who is on the Professional Liaison Committee and what is the charge of the Committee?
Three of the current members of the NLPES Executive Committee, Kate Wade (WI), Ken Levine (TX), and Heather Moss (WA), are serving on this committee. In shorthand, the charge of the Committee is to periodically review the list of entities with which NLPES has established liaison relationships and, in addition to enhancing NLPES's understanding of the other entities' activities, communicate NLPES's activities to the other entities.

What is the current Committee doing to carry out this charge?
The Committee has been in the process of contacting some of the organizations with whom NLPES has professional liaisons; the complete listing of these organizations is shown on the NLPES website at http://www.ncsl.org/programs/nlpes/about/liaison/liaison.htm. In addition to being alert to projects of interest for the NLPES membership within those organizations, we are speaking with the liaisons about professional development and training opportunities of mutual interest.

What is the Committee finding?
Some of our newer NLPES members may have a special interest in learning about local chapters of organizations with a national structure. For example, the American Society for Public Administration (ASPA) has a formal local chapter structure, and NLPES members may find local training opportunities by contacting their local chapter. NLPES members may also be interested in the Center for Accountability and Performance (see the following website: http://www.aspanet.org/cap/index.html) established in 1996 and the source of "Performance Measurement: Concepts and Techniques," a collection of materials designed to help practitioners manage for results.

In contrast, the Joint Committee on Standards for Educational Evaluation is a coalition of some 18 national organizations, including the NLPES, with an interest in the quality of evaluation. The JCSEE has published two sets of standards for evaluation ("The Red Book") and continues with the development of a standards to guide teachers' evaluations of students (newsfl99.htm#Developing_Standards_for_Student_Evaluat )
"Best practices" seems to be a hot topic these days, and NLPES members can tap into information on this topic from a number of sources. Member shops within the National State Auditors Association are working to develop a set of "best practices" questions that could be used to guide a performance audit. In contrast with the best practices materials on the NLPES website that relate to "best practices" within NLPES member shops (see the following site  http://www.ncsl.org/programs/nlpes/survey.htm), the NSAA membership has begun to develop questions that could guide our work on some of the most common issue areas that arise in our audits and evaluations. These questions also remain distinct from the "Best Practices for Good Management," developed by the Utah Office of the Legislative Auditor. This award-winning document (news1001.htm#Utah) is aimed at helping managers of new programs establish sound management processes, and could also be used by managers of existing programs to improve their programs.

The topics addressed by the NSAA members in their best practices project are varied, and the questions for each topic are intended to provide a solid starting point for a performance audit. Sets of questions are under development for the following common audit issues: Complaint Handling; Consultants; Fees and Fines; Staffing and Personnel Utilization; Goals and Objectives; Grants; Inspectors/Inspections; Laws and Rules; Licensing; Record keeping; and Budgeting and Financial Recordkeeping. The questions related to the use of consultants, for example, include:

• Determine why the program needs to use the services of outside consultants. Is the use of outside consultants justified?
• What procedures have been established for selecting the outside consultants?
• Determine the rate of pay for the consultants. How was this rate determined? Is the pay rate in line with the amounts paid by other programs for similar services provided by consultants?
• Has the program established written agreements with its consultants?
• Have internal controls been established for ensuring that consultants do not receive final payment until they have provided all of the services outlined in their contracts?

---

Office Happenings

Minnesota
The Minnesota Legislative Audit Commission recently reappointed Jim Nobles to a six-year term as Legislative Auditor. Roger Brooks was also reappointed as Deputy Legislative Auditor for Program Evaluation.

Mississippi
Kelly Kuyrkendall has recently joined the PEER Committee as an evaluator.

Missouri
Jeanne Jarrett has recently resigned as Director of the Missouri Oversight Division. Mickey Wilson has been named Acting Director.
Pat Brown, Carla Mitan, Emily Donaldson, and Melaine Ott are recent hires at the Missouri Oversight Division.
Jill Talken recently left the Oversight Division for another employment opportunity.

Nevada
Gary Crews, Nevada's Legislative Auditor, recently retired after thirty years in financial and performance auditing. Paul Townsend has been appointed as Gary's replacement.

Ohio
Alicia Mitchell, Cortney Rowland, Chris Woolard, and Molly Peacock are recent hires at the Ohio Legislative Office of Education Oversight.

West Virginia
Susannah Carpenter has recently joined the Program Evaluation and Research Division as a Research Manager. She replaces Harvey Burke who resigned due to illness.

Wyoming
Kelley Pellissier has recently been promoted to Senior Program Evaluator and Johanna Brukardt has been promoted to Program Evaluator with the Wyoming Legislative Service Office.
Emily Quarterman and Noah Miller have recently joined the Wyoming Legislative Service Office.
 
 

---

More From Hawaii

---

2002 Calendar of Events