• Mission & Governance
• Services Overview
• How to Get Involved
• Standing Committees
• NCSL Foundation

• Issue Areas: A-Z
• State-Federal Relations
• NCSL Standing Committees
• News from D.C. and the States

• About State Legislatures
• Elections, Campaigns & Redistricting
• NCSL's StateConnect Directory
• Web Sites

• Online Guide to NCSL Services
• Staff Sections and Networks
• Meetings & Professional Development
• Staff Directories

• Meetings Calendar
• Online Registration
• Meetings Home Page
• Legislative Summit

• Bookstore
• State Legislatures Magazine
• NCSL's StateConnect Directory

NLPES News
National Legislative Program Evaluation Society

Inside the NEWS

Chair's Corner
Small Shop Talk
Utah Office of the Legislative Auditor General Profile
Interviewing Basics: Three Simple Ideas
2002 NCSL Annual Meeting and Exhibition
Arizona Division of School Audits
Using the NLPES Listserve
Occupational Licensing Agency Evaluation Blues
Evaluator Certification: Can We Ignore the Debate?
New Directions for Evaluation
Professional Development Opportunities with Liaison Organizations
NLPES Fall Training Conference
Lessons Learned: Three Models
GASB Research Project
Office Happenings
2002 Calendar of Events

Program evaluators spend a lot of time looking backwards-analyzing and interpreting things that have already happened. But put aside these instincts for a moment, and look at some of the highlights for NLPES members yet to come in 2002.

The NCSL annual meeting will be held July 23-27 in Denver, Colorado. As always, this meeting will be a place for lively discussions of current issues by legislators, legislative staff, national experts, and political pundits. NLPES holds its annual business meeting at this event, and it will also sponsor sessions on topics of particular interest to program evaluators.

While you're marking dates on your calendar, set aside September 18-21 for the NLPES fall training conference. The staff of the Arizona Auditor General's Office have worked overtime in recent months to plan an exciting conference in Scottsdale (just outside Phoenix). Conference highlights will include national evaluation guru Michael Quinn Patton, a session featuring award-winning NLPES studies, and the wonderful Arizona weather! Most of all, though, nothing compares with the fall conference for its opportunities to talk informally with your peers about the work you do and the challenges you face.

In addition, the NLPES Executive Committee is piloting another form of training, and a prototype of this effort will be available later this year. With a small grant from NCSL, we asked the Florida Office of Program Policy Analysis and Government Accountability to develop a training module in a CD-Rom format. The idea is to explore how NLPES offices can share training information in portable, flexible ways-that could be used by individual evaluators at their desks, or by entire offices at their convenience. More details on this later.

Another "coming attraction" is a new Executive Committee survey of NLPES member agencies on several topics of general interest. You may recall that we tried this last year and got a lot of interesting responses-on questions related to topic selection, staff appraisal, helping new staff get "up to speed," and others. (To see last year's questions and responses, go to http://www.ncsl.org/programs/nlpes/survey.htm) We'll aim to have responses to the new survey posted on the NLPES website by fall.

Also lurking in the near future (July) is the end of my term as chair of the NLPES Executive Committee. It's been a true privilege to have this opportunity. As in our legislatures, however, the real work of our group gets done in the committees, and I want to especially thank the NLPES committee chairs for their many hours of hard work behind the scenes during the past year: Jane Thesing, South Carolina (training), Shan Hays, Arizona (communications and technology), Martha Carter, Nebraska (awards), James Barber, Mississippi (membership and nominations), Heather Moritz, Colorado (annual meeting), and Kate Wade, Wisconsin (professional liaisons). And, of course, thanks to the other Executive Committee members for their contributions, too.

Meanwhile, I know that Heather is looking forward to assuming the NLPES chair duties (although mostly she's looking forward to the completion of a politically-sensitive and lengthy audit of the Colorado Lieutenant Governor's office!). And I'm hoping that we can all look forward to a year in which our state budgets will be under a little less stress.

Jim Pelligrini, Deputy Director, Montana Legislative Audit Division

Quality control is essential in any size evaluation organization. In Montana, with 12 staff members, we equate quality control to multi-tasking. The makeup of the evaluation team is intended to facilitate ongoing quality control. Over the last three years we have been using a team process to provide timely quality control over evaluation projects.

Each team is made up of a project leader, staff team members, a project oversight, and an off-site quality control reviewer. While these team members may be assigned to a particular project, they also take on different roles on other projects. For example, the project oversight person on our contract claims audit will be the off-site quality control for another project. Likewise, the off-site quality control on the contract claims audit is also serving as the project oversight on another project and the project leader on a third project. The quality of planning is enhanced since we have increased the knowledge and skill pool by including staff from other areas. Their insight and help can help focus objectives. The brainstorming brings in ideas from other points of view. All levels of staff are involved. Yes, even the deputy legislative auditor does field work.

The roles of each member of the evaluation team are important. The Project Leader is responsible for all aspects of getting the audit planned, completed and report out. He/she is responsible for pulling all written material together into a comprehensive report; ensuring working papers are complete and referenced; evaluations completed; team progress is maintained; and milestones are met. Decisions on the audit are the Project Leader's responsibility.

Project Oversight is responsible for keeping the audit work on track, timely, and according to standards. He/she is responsible for timely review of the work and discussion of audit progress and findings. He/she provides a resource for the project leader and team members for brainstorming and project direction. Project Oversight is responsible for major scope/budget approval and review of the Project Leader's work.

Offsite QC is an efficiency and effectiveness resource for the audit. It provides an independent review of the progress and direction of the audit work. He/she can be used to provide consultation to many aspects of the audit. The main purpose is to provide a timely quality control check of the audit. A major responsibility is providing input on scope/progress/findings.

Each team member is involved in controlling the quality of the audit work and report. Each report goes through an internal review with the Legislative Auditor, Deputy Auditor, and office legal counsel.

The biggest key to this type of quality control is that it is ongoing. We have gotten away from quality control being "a review of the working papers" and something that occurs at the start and end of the project. By the end of the project the issues have been thoroughly discussed, the direction of the work has been clearly established, resources have been shared to meet project needs, questions have been answered or restated, and the objectives addressed. Evidence is reviewed and determined to be competent.

The quality control process is based on using the experience, knowledge, and skills of as many staff members as possible to produce a report that is timely and relevant. With a structure in place that facilitates communication and sharing of knowledge and experience those standards that call for adequate planning, supervision, reviews of compliance, understanding of management controls, evidence and support, reporting, professional proficiency, independence, and due professional care are easy to meet. Bring on that external quality control - "feed me".

Overall, quality control in a small shop is not its own system. It is part of the structure of doing your work.

Our Lovely Acronym: OLAG (pronounced O-Lag) for Office of the Legislative Auditor General.

Facts About Our Staff: 16 men and 7 women. Professional certifications: 2 CPAs, 3 CIAs, 4 CFEs, 1 JD, 3 working on a certification, and 4 thinking about it. Median number of years with OLAG: 7.6. We aren't completely inbred. Besides Utah, we hale from Idaho, California, Montana, Kansas, Alabama, Michigan, Pennsylvania, Oregon, Oklahoma, North Carolina, Florida, Germany, Italy, and Bulgaria.

Common Office Phrases: (1) "What's the effect?" (2) "What are your levels of evidence?" (3) "This is the assignment that could cost us all our jobs." (4) "Who brought treats?"

What We've Grown to Love (Because We've Had To): Construction, construction, construction. First, it was four years of expanding Interstate 15, light rail, and any other road construction conceivable "to get ready for the Olympics." Now, just when we thought it was over, it's our Capitol Building restoration project. This one is scheduled to last 6 years. After two "wing" buildings are built, the Capitol will be closed AFTER you all have a chance to visit when we host NCSL in 2004. You're invited to tour our new digs in "The West Wing" in July 2004.

Least Flattering Descriptions of Our Staff: One of the (then Big 8) CPA firms called us "Unqualified, Unprofessional, and Unethical." Since we had a four-person audit team, we added "Unattractive" and called ourselves the "Four Ewes." Another time, the U.S. Attorney for Utah said we must be "creatures from outer space" for suggesting funds spent on the war on drugs should be redirected.

Dream Audit Assignment Gone Wrong: One staff person went to great lengths to get approval to accompany the pilot on a fish stocking flight over the High Ulntas Wilderness Area. It was an old dive bomber that dropped fish in the lakes high in the mountains. The trip was spectacular until the first dive and he got so sick he lost his lunch all over the cockpit. Of course, we were never invited again.

Strange Audit Experiences: (1) Being in the interior of a dam and being told not to worry; all the water pouring through the cracks inside was normal. (2) Being inside a prison cell when a guard demonstrates how the cell locks work, but then cannot get the cell door open again for three hours.

Entrance Conference Attention Getter: When the Executive Director of Corrections puts his gun on the table, saying it would make him "more comfortable."

Unusual Exit Conference Happenings:

In the midst of a tense meeting, a former Auditor General turned and said, "I have some questions for the auditors."

One agency director couldn't attend the exit because he had already been fired. He later went to prison.

We thought an agency director was eating breath mints throughout a two-hour exit. When he had a heart attack later that day, we learned he'd been popping nitro pills for his heart. Fortunately, he made a full recovery.

A director stormed out three times threatening to sue if the report was released, but returned each time trying to intimidate us to drop certain findings.

Most Intriguing Description of an Audit Report by Our Committee Chair: "Very frankly, this audit reads like a cheap novel. Unfortunately, it's true."

Time to Question Your Audit Steps: When wildlife resources officers offer you a bulletproof vest after inviting you to accompany them on a raid of a poacher's camp.

True Stories about (Mostly Former) Staff:

The FBI came to interrogate one staff member as a suspect in the "mute bank robber" case. (He was exonerated.)

A former intern punched out an Assistant AG in a parking space dispute.

Staff were once rebuked on the House floor for raiding hot chocolate from the Legislators' kitchen.

One staff member spent parts of 3 days adjusting ceiling tiles trying to get better ventilation and then found out they were made with asbestos.

One staff member used to call the Governor's office at the first sign of snow, suggesting that weather leave be considered.

One staff member forgot that he had put some work papers in his trunk and sold the car. The work papers were never recovered.

How We Reacted to the Tornado of 1999: Never having ever seen a tornado, most of us rushed to the windows get a good look (probably not the best idea, but no windows broke). Meanwhile, our Auditor General was outside in his jogging clothes and just kept running. We understand he set a personal best.

Some of Our Favorite Things About Being an OLAGer: Casual Friday, flextime, the tough assignments (usually), donuts at staff meetings, we make each other look normal, and (most of all) knowing we make a difference.
 
 

In this article, I offer three simple ideas to help the experienced interviewer hone their skills: First, interview the right people; second, develop your own, authentic, style; third, share knowledge, don't dig for it.

Don't interview too many people; interview the right people.

One of most common questions I am asked as the resident methodologist at the Arizona Office of the Auditor General is "how many people should I interview?" Without fail, my response is never accepted on its face with anything but shock and disbelief, maybe even a sense of horror that this so-called expert is telling me something that CLEARLY IS WRONG. Well, it definitely flies in the face of their beliefs and that answer is: one. One person is all you need to interview as long as it's the right person.

The myth that more is better probably comes from survey research methodology where numbers mattered for scientific validity. No such standard exists in interviewing because interviewing as usually done in an audit/evaluation environment is a qualitative enterprise not a quantitative one. As a result, numbers really don't matter. They act only as a placebo for the auditor to make him/her feel better about their work. From my perspective, bigger numbers take the burden off the auditor to do it right. The auditor can make mistakes and still go to the next person. I don't really have a problem with this except that in an era of tight budgets, nobody can afford to over-interview. Some in our own office say: "But it gives the auditor some start-up time; lets him get his feet wet in the agency and figure things out." This point of view undermines one of the most basic premises of any method including interviewing, namely, that the method should have value to all who participate. We would never think of letting a new staff person "get his/her feet wet" with a survey or a regression analysis. Interviewing deserves the same methodological respect. The new auditor is better to try observation or even a structured observation tool to learn about the agency firsthand and the processes he/she is reviewing. Save the interview for when it can be useful to both interviewer and interviewee.

So interviewing a lot of different people may make the auditor feel better but it is a placebo and has no real added benefit especially to the experienced auditor/evaluator. In fact, too many interviews can be a detriment to the his/her own skill development and places an undue burden on the agency. There are other downsides to interviewing too many people besides the inefficiencies and ethical legerdemain. Multiple interviews require excellent synthesis skills and so unless our legislative offices are more committed to building the synthesis skills, multiple interviews will inherently fail us. All you need to do is read interview writeups and then ask for the dreaded ''summary work paper"-you will be met with blank stares and a glazed look will come over the auditor's eyes before he/she asks invariably: "what do you want me to say in that summary." No, more is not always better.

Using the wrong style for interviewing.

I think the single most problem area for auditors and the reason for the vast majority of failed interviews is that the interviewer used or applied a style that does not fit him/her. In this method, genuineness and authenticity matter; you are one-on-one, the person has a close conversation with you about things that matter to him. Because we are auditors or independent evaluators, we try to appear aloof and independent. Independence reinforces a kind of 1960's behavioral model known as stimulus-response. The first performance audits under sunset were most likely similar to police investigations, maybe even focused on fraud, waste and abuse so it is not surprising that the central technique of fact finding: the interview, was used in a way that reflected interrogations: question/answer; question/answer. Indeed, auditors developed checklists and "control questionnaires" to collect the "responses". My first foray into teaching interviewing to the auditors in our office started with the deconstruction of the controls questionnaire in wide application at the time. I would ask the auditors: How do you implement this instrument in the field?" They answered: "we ask the questions on the questionnaire." The audit teams felt that as long as they sent somebody new in every year with the instrument, they were minimizing bias and maximizing independence.

So, one important idea I can covey to auditors in the field today is to abandon stimulus-response notions of the interview method. Instead, think of it as many qualitative methodologists characterize it, a conversation with a purpose. Forget about developing skills to help you ask the perfect question, word a phrase perfectly, or pop the question at just the right moment in the interview. Rather, develop listening skills, learn to direct conversation effectively, and learn the intricacies of a good probe. Most importantly, the auditor must develop his/her own authentic style of interviewing. Develop a style that people know in their hearts is the real you and no matter how difficult the question, I guarantee they will respond out of respect for you. Interviews fail when the interviewee loses respect or trust in the exchange, usually because it lacks authenticity. One simple exercise I give our auditors is to go home every night for a week and "interview" your spouse about his/her day. You have 10 minutes to get everything he/she did and its import. In order to do that you must listen, direct, probe and most of all really be interested in what they are saying.

Share knowledge; collect the essence of what is meaningful.

Modern philosophers tell us that there is no true knowledge. Rather all knowledge is simply constructed by the experiences we have, the language we use and the shared images and concepts of our culture. Most of the time, interviewing as currently done does not reflect this point of view. Rather, interviewing assumes "they" have the knowledge and I have to get it out of them via this method. Somewhat influenced again by interrogative interviews, audit/evaluation interviewing can tend towards objectifying truth rather than constructing it through the interview.

When knowledge is socially constructed however, a few new conditions apply. First, we need to respect the person we are interviewing and the contribution they make as equals. "They" don't have anything but together they and we can create a picture of the process that we both agree appears to be a reliable description of the issue at hand. Second, our ethics must balance the confidentiality of our workpapers against the interviewee's need to know how we are thinking about the issues. We need to treat their participation with more value, more as an "exchange" of views. Third, the interview should have meaning to both parties and not occur from obligation (because you are being audited). It falls on the auditor to make it a meaningful exchange and not simply exploit his/her position of authority to "require" answers. The interviewee can expect your behavior to reflect these values of mutual respect and worth. Obviously, under these rules, lying and even deliberate omissions (common in interrogation) are out of the question.

Think about it as if the auditor is a traveler, the experiences of his foray into the agency or program should cumulate into a feel for the agency and its people. The auditor must feel like it is a trip he has saved up years to take and is looking forward to with excitement and the anticipation of finding out new and interesting things. S/he has prepared himself by learning the conversational expressions; S/he respects the culture, the history, the people. S/he will travel among them, observing, experiencing and asking questions where appropriate to better "engage himself into the culture". The auditor's critical eye is simply to improve his own understanding and not to judge the agency. S/he may ask questions about paradoxes he observes. This may cause some consternation among the agency the same way a probing question asked by a therapist may unloosen some old truths to create space for the new. The auditor should leave the interviewee with a real sense that he has actually gained valuable knowledge about him/herself and the agency by being audited-isn't that a radical idea. Ultimately the new, shared understanding between the auditor and the agency is the litmus test of a successful interview.

Arizona Division of School Audits
Sharron Walker, Director, Division of School Audits, Arizona Office of the Auditor General
In most states, K-12 education represents a significant portion of state spending. In Arizona, this is about 48 percent of the state's General Fund budget. Despite that level of effort, however, Arizona frequently ranks near the bottom of the scale in national studies on the amount of K-12 education funding, leading to criticism of the Legislature's efforts to fund the schools. However, some legislators have been concerned about simply increasing funding for schools when there are questions in their minds about how effectively and efficiently the schools are using the existing funding. Legislators have historically been concerned about what they perceive to be high administrative costs in the schools. However, a particularly telling point with legislators has been that national studies show Arizona school districts spend a lower percentage of each dollar on instruction than most other states. This 'classroom dollar' issue further heightened legislative concern about how efficiently school funding was being used.

Although there has been legislative concern about the uses of school funding, schools have not historically been subject to the same level of legislative oversight as other programs with significant state funding. In Arizona, only one performance audit of school districts has been previously performed, and that was in 1996. When Governor Jane Hull, a former schoolteacher, asked for increased funding for schools, the Legislature agreed to raise the statewide sales tax if the voters approved. In supporting the increased funding, legislators who were concerned about perceived inefficiencies demanded increased oversight in two ways - monitoring classroom dollars and conducting performance audits of each of the 230 districts once every 5 years. In July 2001, the Auditor General established the Division of School Audits to carry out the mandates resulting from Proposition 301. The subsequent budget crisis, a projected $1 billion shortfall for fiscal years 2002 and 2003, has since dictated that we will not staff to a level to audit each district every 5 years. However, the 5-year requirement shows the degree of legislative interest in ensuring efficient school operations.

Before beginning individual district performance audits, the Auditor General decided to lay a foundation with two baseline studies. First, to facilitate measuring the effect of the increased school funding, we determined the classroom dollar percentage, at the statewide level and for each school district, prior to the addition of Prop 301 sales tax revenues. Second, we reported on school district plans for spending the new education sales tax monies.

To determine the percentage of dollars spent in the classroom, we used the National Center for Education Statistics' definition of dollars spent on instruction. This allows comparisons to be made to other states' statistics, the national average, and Arizona's past performance. Our first baseline study reported that Arizona school districts in fiscal year 2001 spent about 57.7 percent of their dollars in the classroom. This compares to the most recent national average of 61.7 percent, which was based on 1999 data. Individual district percentages ranged from 32 percent to 89 percent, however, the vast majority of districts fell within five percent of the state average. Most of the reported results were as expected -- districts with higher classroom spending typically had larger student populations, higher average teacher salaries, and more experienced teachers. And more dollars spent on food service, transportation and administrative costs meant lower classroom spending percentages. An unexpected result was that additional funding provided for desegregation programs and voter-approved budget overrides (tax rate increases) appeared to have no effect on the classroom spending percentage.

In our second report, we looked at how each school district planned to use its new Proposition 301 sales tax monies. Prop 301 specified three allowable uses for the money: teacher base pay increases, teacher performance pay, and 6 specific maintenance and operation purposes ('menu options'), including additional teacher pay increases. With teacher compensation as the main component, several questions arose regarding the definition of teacher and the allocation of pay increases. The Attorney General determined that for purposes of Prop 301 pay, the teacher compensation increases are not limited to traditional classroom teachers. Instead, employees who 'provide instruction to students on matters related to the school's educational mission' are eligible for the pay increases. Thus, districts have to negotiate with their employees and define in their plans which classifications of employees are eligible. This lead some districts to include librarians, aides, speech therapists, and nurses as eligible to receive those raises.

Overall, the districts plan to use 97 percent of their Prop 301 monies to increase teacher compensation (including all classifications they determined eligible). Most districts plan to distribute base pay increases equally among eligible employees. Further, most districts were developing new performance-based pay plans as they did not previously provide incentive pay. And while the maintenance and operations allocation could be directed to any of six 'menu' options, 75 percent of the districts indicated they were putting some or all of this money toward additional teacher compensation. Besides reporting the statewide trends, we also compiled an individual report page for each school district detailing its planned uses of the new Prop 301 monies, along with other comparative data and the classroom dollar percentage.

The new sales tax dollars are apportioned directly to school districts; the money is not subject to appropriation from the State General Fund. This also means districts are not guaranteed to receive the projected amounts. With the downturn in the economy, sales tax collections have not met the estimates, leaving some districts without sufficient revenues to fund their contractual obligations to teachers. Some districts are renegotiating teacher contracts, supplementing with district general funds, or terminating teachers early. The Legislature is currently considering a bill to allow districts to spend these dedicated monies on a budgetary basis rather than the current cash basis.

With these first two baseline studies completed, we are now preparing for the school district performance audits. We are evaluating various types of information on the 230 Arizona school districts to develop our audit schedule for the upcoming year. We have visited some school districts to get a realistic picture of what we may find once we begin conducting our first audits. Based on our report on classroom dollars, our initial performance audits will focus on school district food service and transportation operations and administrative costs. With our current staff of 18, we expect to complete 10 to 12 school district audits the first year.

To read our first two reports, you can visit our web site at www.auditorgen.state.az.us. Also, if you would like further information, contact Sharron Walker, Director, Division of School Audits at (602) 553-0333 or swalker@auditorgen.state.az.us.
 
 
 

Do any of you get the occupational licensing agency evaluation blues? Well we have a tool that may make your job a bit easier. The Sunset staff have recently completed an update of our Occupational Licensing Model. While not a terribly exciting title, Sunset staff have found the document quite helpful in our occupational licensing evaluations. And after 20 years since our last version, an update was certainly in order.

The Model covers every aspect of an occupational licensing agency or program from the need for the agency to how licensing enforcement should work, based on common standards we have observed over 25 years of evaluations. For example, under the area of licensing, we have a subject category called "experience." One of the standards is that experience requirements should be set to ensure competency and not limit entry to the profession. This standard was developed based on evaluations where time and again, a profession or trade would adopt experience requirements that had little to do with ensuring the public receives quality services, and a lot to do with limiting competition. I doubt that this experience is limited to Texas.

The Model also delivers a more detailed explanation for each of the standards to help you develop criteria to measure whether the program meets the standard. For example, in the area of licensing enforcement, one subject area is "inspections". The standard is fairly broad, stating that the agency should have clear procedures, rules, and statutory authority for conducting inspections that help ensure consistent treatment of licensees and timely compliance. The explanation section then lays out what policies the evaluator would expect the agency to have that define records, facilities and other matters subject to inspection. The model gives additional explanation on the use of announced versus unannounced inspections, risk based scheduling of inspections, and the processes in place to follow up on compliance issues.

Since the Model was, of course, designed for our staff, a few of the explanations may be a bit "inside Sunset," but most are relevant for any evaluator. For example, The Model refers to the term ATB in several places. ATB means a Sunset Across-the-Board recommendation that we apply to all agencies that pass through the Sunset process. (Information on Sunset ATBs can be found in the Guide to the Sunset Process on our Website.)

Internally, we have also built a Microsoft Access database around the Model to enable staff to query the information in the model. For example, one could query "enforcement" to see all the standards that relate to licensure enforcement. Our database also contains a couple of enhancements that are not in the printed model. First, the database includes lists of appropriate questions to ask an agency in each of the subject areas. Second, the database links each standard to related recommendations that staff have made in earlier reports. In this way, we not only can see the standard, but also examples of how and where we have previously made a similar recommendation. While we would love to take full credit for this fancy database, I must admit we used a consultant to help us develop it.

Now that you know all about the model, I'm sure you are ready to read it. The Model is available online (www.sunset.state.tx.us/sunset). If you prefer a printed copy, please call 512.463.1300 and request the Occupational Licensing Model or contact us via email at sunadmin@sunset.state.tx.us. We would love to hear your comments and suggestions about the model. Please feel free to email me at: ken.levine@sunset.state.tx.us
 
 

In the Fall 2001 issue of NLPES News, I asked for some ideas as to what kinds of questions you would like to have addressed by members of a panel titled, Certification of Evaluators: Do We Really Need It. In response, I received suggestions that were helpful in facilitating the panel discussion at the annual meeting of the American Evaluation Association in St. Louis last November. This article provides a context in which the discussion took place, and then the key points from the lively discussion involving members of the panel and the audience.

My sincere thanks to James Altschuld (Professor, School of Education Policy and Leadership at Ohio State University), Nancy Zajano (Director, Ohio Legislative Office of Education Oversight), and Linda Triplett (Evaluation Division Manager, Mississippi Joint Legislative Committee on Performance Evaluation and Expenditure Review) for serving on the panel.

Legislative oversight staff working as analysts, evaluators, and performance auditors know of certification requirements only in relation to financial and internal auditors, fraud examiners, and government financial managers. Some of us maintain legal credentials if we hold a law degree. Other than that, it is primarily the quality of our work that certifies our credibility.

Context. Unlike those of us in the legislative environment, the larger evaluation community has been debating the issue of evaluator certification for some time. In recent years, the debate has come to the forefront in the form of panels, surveys, publications, and informal discussions on Internet listserves. The debate has focused on defining the evaluation profession and differentiating between evaluators and other professionals that include methodologists, research analysts, auditors, and yes, legislative analysts.

In his presidential address at the 1997 annual meeting of the American Evaluation Association (AEA), Will Shadish caused a stir among the audience members when he asked them to self-assess whether they are evaluators by answering 10 questions that he posed to them as essential to the evaluation profession. His questions related to logic, validity, causal inference, evaluation use, metaevaluation, and the nature of the program being evaluated. These questions have subsequently been linked by some in AEA to the idea of evaluator certification.

An AEA taskforce on certification facilitated discussions at its 1997 and 1998 annual meetings, as well as conducted a survey of AEA membership in 1998. The survey results showed a varied mix of interests and opinions about the desirability, necessity, effectiveness, and feasibility of certification (174 respondents, n = 431). Along with the survey results, five other related articles were published in the fall 1999 issue of the American Journal of Evaluation.

Two of those articles were by James Altschuld, who chaired the AEA task force on certification. He has proposed credentialing of evaluators as the first step toward setting some boundaries for defining the field of evaluation. The idea of certification is moving forward as evident by yet another recent study, which looks at the extent to which evaluators from different backgrounds could agree on essential evaluator competencies. [Other articles on the topic of certification appear in the 1998 American Journal of Evaluation, 19 (1), pages 1-19, and 1999 American Journal of Evaluation, 20 (3), pages 495-506.]

St. Louis Panel. No clear position on evaluator certification emerged from the 90-minute panel discussion. The three distinguished panelists-James Altschuld, Nancy Zajano, and Linda Triplett-presented a broad range of views on the idea of certification.

As expected, Altschuld advocated the need for some type of certification for evaluators. He argued that we need to define our profession and agree upon a minimum level of competency for evaluators-as it is now, anyone can call himself or herself an evaluator. He suggested credentialing as the first step, which is different from certification. Credentialing involves completion of minimum core competency requirements (academic courses or field experience), while certification requires testing the individual's knowledge and skills to demonstrate a minimum level of competency.

Both Zajano and Triplett, on the other hand, expressed concerns about the unintended consequences of credentialing. In particular, they said that the process might leave out competent practitioners who do not meet the credentialing requirements. They, however, recognized the need for defining the field and ensuring that competent evaluators represent the profession. One way to achieve that, they said, is through on-going evaluator training.

After covering both the pros and cons of credentialing in her opening statement, Zajano concluded by recognizing the need to start the process of identifying the common skills and knowledge needed by evaluators across subject areas and practice settings, no matter how difficult this may be. She noted that as program evaluators we often require program staff to articulate what they do and develop performance measures. We need to be able to articulate what we as evaluators do as well.

There was considerable discussion on the importance of a common language and professional standards. Because of its interdisciplinary nature, it is difficult to get one's arms around the field of evaluation and an agreement on core competency requirements for evaluators. The audience members brought up two key points. First, many evaluators like the professional diversity that AEA represents. The second point is that it is difficult to define the profession and it can be a challenge for those within the profession to market themselves as "qualified evaluators." Some in the audience asserted that having a certification would enhance the profession and make it easier for them to establish their credibility among consumers of evaluation services.

Conclusion. My purpose for writing this article is not to persuade you for or against certification. However, I would like to evoke a debate within NLPES, and encourage its leadership to keep an ear and an eye open toward what is happening outside of NLPES that may define our profession and judge the credibility of our work. Not all of us call ourselves evaluators, but we use the "e" word (evaluation) in our work frequently, including in the name of our society (NLPES). A decision by AEA on evaluator certification can have a significant impact on all of us, since AEA is one of the largest evaluation organizations with 2,969 members from the United States and 55 other countries.

In essence, we cannot ignore the debate on certification that is already taking place. It is up to us-either we participate in it, or react to the outcome of the debate later when the likelihood of changing the outcome is less. So here is my proposal to you, read at least one of the articles I have mentioned in this article. Happy reading!

Editor's note: The views expressed in this article are those of the author and panelists mentioned in the article and do not necessarily represent the views of the Washington State Joint Legislative Audit and Review Committee.

In an era of tightening budgets and increasing demand for our work on an expanding range of evaluation topics, many NLPES-member offices are looking for ways to enhance staff skills using locally available resources. In the Liaison's Corner of the Winter 2002 NLPES News, local chapters of the American Society for Public Administration were cited as potential sources of training. From the NLPES website, at http://www.ncsl.org/programs/nlpes/about/liaison/liaison.htm, one can link to several organizations served by an NLPES liaison and learn about professional development opportunities available throughout the United States.

The American Evaluation Association (AEA) homepage includes 14 affiliates throughout the United States, linking to websites for most of these affiliates. While there may not be an affiliate in your immediate area, you may find useful articles in newsletters produced by affiliates, or identify a contact by reviewing the membership roster for an affiliate.

The Meetings/Events link from the AEA homepage displays a lengthy listing of events through December 2002. The topics are varied and so are the locations: our own NLPES Fall Training Conference in September in Arizona is listed, as are conferences on "Benchmarking Best Practices in State and Local Government Information Technology" in Chicago in October and the AEA's own annual conference, scheduled for November in Washington, D.C.

Local chapters of the Association of Government Accountants (AGA) can be identified by linking to the AGA's Membership page. Monthly luncheons and evening meetings are common among the chapters, and all chapters identify a contact person. The AGA is self-described as a not-for profit educational organization, so, not surprisingly, daylong seminars, teleconferences, and other professional development opportunities are featured on chapter websites. Because membership extends beyond accountants to include auditors, budget analysts, chief financial officers, information systems managers, finance directors, inspectors general, professors, and students, luncheon topics are wide-ranging. In the Southern Wisconsin Chapter, for example, this month's luncheon speaker, the State Budget Director, is playing to a sold-out house!

The Institute of Internal Auditors (IIA) has developed an extensive website. From the homepage, one can gain information on IIA affiliates by clicking on the "Membership" tab at the top of the site. The affiliate page notes that local chapters are the backbone of the organization, and websites for the local chapters tend to include information on meetings and training events within the chapter. For a broader view of issues in internal auditing, links on the IIA homepage provide information about Enron-related activities within the IIA and allow you to peruse The IIA Bookstore.

The IIA site, will give you the top 12 benefits of joining the IIA as a governmental auditor. The site notes the IIA's awareness of the tight budget --high visibility--environment in which governmental auditors function.

Also the IIA site contains information designed to meet the broad range of informational needs of the governmental auditor. The information is wide ranging, including, for example, an "Evaluation Cookbook," described as a practical guide for educators/lecturers interested in evaluating materials for their effectiveness in achieving specific learning objectives. Also included is a link to the Intergovernmental Audit Forum home page where you can learn about conference opportunities in your part of the country.

While travel and training budgets may be tight, visit the websites of the NLPES liaison organizations to see what resources may be available in your own backyard. And while you're visiting, don't hesitate to reach over the backyard fence, so to speak, and invite your colleagues in related organizations to join us at the 2002 NLPES Fall Training Conference in Arizona.

Sometimes it seems like we encounter the same challenges in every project we undertake. A few states have developed procedures to try to capture and share the lessons learned in each project.

The Arizona Model. Arizona's "lessons learned" process began in 1997, and begins with a team meeting after the report is published. Team members discuss a list of questions in six general subject areas: budget, staffing, issue selection, methods/audit steps, writing, and agency relations. About once a year, a committee reads through the write-ups of all the year's meetings and summarizes the findings for everyone to read. We also keep a copy in the permanent file we maintain on every agency we audit, to assist the next team.

Although the process has helped to identify and institutionalize some improvements, we believe it's time to take it a step further. We're looking for ways to make the information more accessible, so future teams are more likely to apply the lessons learned. At the same time, we're re-examining our questions and our process, to ensure we are openly discussing problems and solutions, not griping and finger pointing. To improve our process, we've been looking to our peers for help.

The Florida Model. OPPAGA uses a facilitator to debrief teams after some of its projects. Team members complete an extensive questionnaire that explores all aspects of the project (purpose, timelines, planning, fieldwork, writing, quality assurance, interactions, and project value). When this process began about four years ago, results of the debriefings were posted on a shared drive for everyone to read, which was useful but created some anxiety among staff, and the process was discontinued. Now, these debriefings have resumed at each team's option, and the results are not summarized or published unless the team wishes to do so. Some teams have held brown-bag sessions to share the lessons learned.

The Kansas Model. In Kansas, we've tried various approaches for memorializing the hard-won lessons others have learned, and, just as important, making those lessons available to others. We've tried providing oral reports in staff meetings on lessons learned, and listing those lessons on a static web page on our intranet. However, neither of these techniques met the twin criteria for such a system:

• it must make it easy to capture staff's experiences; and,
• it must be easy to search for and find lessons relevant to a particular part of the audit process.

A routine lessons-learned meeting. A few days after each audit is presented and made public, the supervisor convenes a meeting of the Legislative Post Auditor, audit manager, audit team, and others involved in the audit. Staff are instructed to think about the activities/processes that worked well during the audit and helped us accomplish our goals, as well as areas that didn't work so well, and to think about potential solutions to the problems encountered.

A public forum for discussing the lessons learned. The items listed and refined during the lessons-learned meeting are discussed with the whole staff during our bi-monthly staff meetings. This ensures that everybody hears the ideas and has a chance to discuss them.

A method for recording and preserving the lessons efficiently. In theory, the list items could be recorded and cataloged on index cards, but a computer database is simple to set up, straightforward to operate, and it allows users to conduct searches by audit topic, audit phase, or audit title from the convenience of their desktops.

A systematic way of ensuring that everybody records the lessons learned on each audit engagement. An item on the supervisor's 13-page performance audit checklist reminds the supervisor to oversee formulation of lessons learned by the team during the audit, and makes the supervisor responsible for entering those lessons into the database The database allows each entry to be categorized by audit phase (planning, fieldwork, etc.), and by activity (interviewing, surveys, report outline, and the like).

Currently, our management team is discussing the system's biggest drawback: there's no easy way to ensure that staff members avail themselves of the lessons others have learned. Options we've discussed include the following:

Periodically summarize the entries and distribute to staff. Summarizing the entries for the last 6 months, say, or for the last 2 rounds of audits, would ensure that people get the benefit of others' experience even if they miss the staff meeting where the lessons were discussed the first time. On the other hand, staff would get only the two exposures (the initial staff meeting presentation and the subsequent update), and probably neither exposure would be contemporaneous with users' need for the information. For example, it's unlikely that an admonition about survey design would happen just as a staff member was in the process of designing a survey.

Periodically use the entries as the basis for training. This option would go beyond just distributing recent entries to staff for discussion. Frequent or related entries would be taken as a sign that training in those areas was needed, and used as a basis for designing that training on the assumption that lessons learned over and over probably indicate a general knowledge deficit. In such cases, the database would be helpful in pinpointing those deficits. As with the previous option, it's unlikely that a staff member who has a need for hints on interview techniques will be preparing for interviews at that particular time. Although some training is well retained, much is lost if it isn't immediately put into practice.

Periodically "clean up" the database to eliminate duplicative or overly specific entries, and then require staff to refer to it at the beginning of an audit or audit phase. The idea here would be to add an item to the supervisor's checklist reminding them to review the database at the start of the audit, and to write in audit steps requiring team members to review the database before, say, creating a survey instrument or writing interview questions. This option would put the information in front of the staff member at the time it's needed, although it would require somebody to periodically go through and make decisions about what to keep and what to consolidate or jettison. Otherwise, over time the database easily could become too large and unwieldy for this option to be practical.
 

Three Rules for Legislative Evaluators and Auditors

• You have no friends in state government other than each other. (Or, you're not paranoid if everyone really is against you!)
• If you don't know or aren't sure, don't guess or speculate. (Or, sometimes ignorance is bliss.)
• If you get in trouble, ask for help. (Or, when you're in a hole, stop digging.)

Have you ever wondered why you, a program evaluator, should be interested in the activities of the Governmental Accounting Standards Board (GASB)? After all, accounting rules and regulations are just for the "bean counters," right? Well, you might be surprised to learn that GASB has something to offer you too.

While the governmental accounting world has been focused on the issuance of GASB 34, a comprehensive change in the financial reporting model for governments, GASB has also been busy with a significant research project focused on the reporting of performance measurement data.

GASB's research project on Service Efforts and Accomplishment (SEA) reporting is designed to expand knowledge and improve practices related to reporting performance information. Ultimately, GASB will consider the appropriateness of establishing formal standards or guidelines for public reporting of government performance information. For now, the research is accumulating a wealth of information on trends and best practices for government reporting of performance information. Research into citizens' views of performance information will soon culminate in the publication of a research report on suggested criteria for use in reporting performance information.

Background

The GASB SEA project is funded through a grant from the Alfred P. Sloan Foundation. The original three-year grant, awarded in 1997, was extended in October 2000 for an additional three years.

The project is divided into six phases. The initial phases were designed to gather and share information on how performance measures are being used, and the effect of their use, at the state and local levels. Subsequent phases are directed towards determining user needs, developing suggested criteria for reporting performance measures, and encouraging experimentation with public reporting of performance measurement data. The final step is to determine the success of the suggested reporting criteria and assess whether GASB should consider adding to its active agenda a project to establish reporting standards or guidelines for performance information.

Status of Project

Many positive benefits have been realized from this project. There is already a significant amount of material available about how performance measures are being used by state and local governments. This body of knowledge is growing daily.

GASB has established a comprehensive performance measurement clearinghouse on the Internet at www.seagov.org. A tremendous amount of useful information is available at this site, including: case-studies of state and local governments using performance measures: the results of a GASB performance measurement survey: the results of citizen discussion groups on reporting performance information: contact information for individuals and organizations using performance measures: links to governments that are using performance measures: discussion forums; and links to related websites.

GASB has studied 28 state and local governments in depth to review and document their use of performance measures, and the effects of that use, in the decision-making process for strategic planning and budgeting, management, and reporting. This study included 14 state governments and 14 local governments.

What Next?

GASB's effort to promote the use of performance measurement information is continuing. As additional case studies are developed, they will be added to the GASB Performance Management for Government page. Five case studies are currently in production, including two state examples. Suggested criteria for reporting performance measurement data are currently being developed and tested, and are scheduled for release as a research report later this year. GASB will encourage experimentation with the suggested criteria once they are released. Finally, around 2005, GASB will consider whether to include the issue of reporting standards for performance data on their technical agenda. If a decision is made to add performance measurement reporting to the agenda, it will mark the beginning step for possible implementation of standards in this area for state and local governments.

For More Information

For more information on GASB go to their website at www.gasb.org. The Performance Measurement for Government site can be accessed from this page, or directly at www.seagov.org. Additional links are available for local, state, federal and international sites, current research, and other sites with information on performance measurement. One link even has relevant sites organized by service areas. Those looking for information on performance measurement for specific service areas, such as education, foster care, or transportation, should find this to be a helpful site.

Governments at all levels have struggled with ways to measure and report on their performance. The 1990s saw an explosion of efforts across the country to tie performance measures to planning and budgeting efforts. The next few years will likely see more experimentation with reporting performance information to citizens. Better clarity about program objectives, agreement on how program success will be measured, and consistency in reporting performance information will be beneficial to program evaluators. Visit the GASB web site for more information on this exciting project.

Colorado

Deputy State Auditor Larry Gupton will be retiring after 22 years with the Colorado Office of the State Auditor.  Larry, his wife Marcia, and their Labrador puppy are looking to relocate later this year to a warmer climate, possibly the North Carolina shore.  Larry plans to work part-time for the Office through the end of the calendar year and then spend his time sailing and enjoying a well-deserved rest.

The Colorado Office of the State Auditor announces the appointment of Sally Symanski to the position of Deputy State Auditor.  Sally has been with the Office since 1991.

The Colorado Office of the State Auditor extends a personal invitation to all NLPES members to attend the upcoming NCSL Annual meeting in Denver in July.  The weather’s great, the social events will be fun, and we are all looking forward to seeing you in the Mile High City.

North Carolina

Recent new hires for the North Carolina General Assembly, Fiscal Research Division include Jim Klingler, Doug Holbrook, Chloe Gossage, and Reyna Walters.

Montana

The Montana Legislative Audit Division had its first retirement from the Performance Audit function.  Mary (Reynolds) Zednick’s last official day of work was May 3.  Mary left her Audit Manager duties and moved to a small Montana town (population 490, plus 1) to be relaxed and content.  Her first official duty on the Monday after her retirement was to shovel 14 inches of snow.

With Mary’s retirement comes the addition of a new (younger) staff member, Angus Maciver.  Angus is the first hire in more than six years.