 |
Home | Contact Us | Press Room | Site Overview | Help | Login | Register |
 |
 |
| About NCSL | State & Federal Issues | Legislatures | Legislative Staff | Meetings | Bookstore | Legislators & Staff Only |
| NCSL Home > Legislative Staff: Staff Sections & Networks > NLPES > | Add to MyNCSL |
NLPES News
Fall 2002, No. 83
|
||||||||||||||||||||||||
PDF Version
Chair's
Corner
Congratulations
to 
Cheryl
Ridings Recognized for ContributionsThis year's recipient of the NLPES
Lifetime Achievement Award is Cheryl Ridings, retired Deputy Director of
the South Carolina Legislative Audit Council.
|
Please take a few minutes to browse through the results from the most recent survey of NLPES member agencies. We had a tremendous response to our 2002 survey-33 member agencies participated! This year's topics included (1) how our agencies have coped with recent budget challenges; (2) how our agencies conduct internal review of findings prior to issuing reports; (3) legal authority our agencies have to obtain agency records; (4) ideas for possible education-related evaluations; and, (5) comments on NCSL's electronic database of legislative audit reports. Go to the NLPES website (http://www.ncsl.org/programs/nlpes/research/survey/survey.htm) to view the survey results. While visiting this website, feel free to look through other years' survey responses. Thanks again to everyone who participated in the 2002 survey! |
State Profile
Connecticut Office of Program Review and Investigations
Welcome to Connecticut--We're from Connecticut, Hartford, that is...41.45'N -- 72.40' W, or about midway between New York City and Boston. No, we're not a bedroom suburb of either, though it's been reported some of our Gold Coast residents in the southwest believe Albany is their state capital.
We're known formally as the "Constitution State," derived from a compact written by the first settlers in the colony to establish their individual rights and responsibilities. It was the first such document in the new land. Informally, we're the "Nutmeg State," a reference arising from apocryphal tales of our early drummers peddling spices and medicinals, nutmeg in particular, to innocent homesteaders in the Western Reserve. The story goes it wasn't really nutmeg but oak bark and the Iowans were an easy sell.
Today we peddle jet engines, helicopters, submarines, pharmaceuticals, insurance, and small arms. Notable historical developments include Sam Colt's six-shooter that made John Wayne the man he was, Horace Wells's anesthesia so Wayne's target wouldn't have to bite a bullet when the doc removed the slug, and Elias Howe's sewing machine to stitch the hole in his jersey.
While high in income, middle in population, strong in Fortune 500 corporate headquarters, we're 48th in land area. For the latter, however, we've got Rhode Island (mostly water) and Delaware to kick around. And who cares how small we are when we're home to the national women's basketball champions--the UConn Huskies.
Modus operandi--Program Review and Investigations has been in the Capitol building since 1972 when it was statutorily created over a gubernatorial veto. The first director was none other than George Schroeder, who as E.D. of the Legislative Audit Council has been sharing what he learned here with the citizens of South Carolina for two decades.
The nonpartisan staff serves a twelve-member, bipartisan, joint committee of the General Assembly. The committee sets the agenda, reviews and typically adopts staff reports, and raises bills to implement legislative recommendations. It has the power to subpoena witnesses, documents and materials, and compel testimony under oath. The committee generally operates on a calendar year, selecting topics in January and adopting staff reports the following November and December.
The staff--a dozen nonpartisan professional plus four graduate summer fellows--are your typical well-educated analyst types who have never seen an agency, program, or policy they could not or should not improve. Each fall and winter, the staff--armed with the sword of righteousness and marching under the banner of truth--goes into the hearing and final report phases to lay before the committee its best thoughts on the six to ten projects conducted that year.
Occasionally windmills appear and we respond full tilt. Examples: "I'm sure the Majority Leader will understand and appreciate the wisdom of our recommendation to eliminate the deputy commissioner position. There are certainly other careers his brother can turn to. . . ." "I realize the County Sheriffs date back to medieval England, but they really don't do anything that can't be handled more efficiently somewhere else. It's perfectly reasonable to eliminate them especially now that Robin Hood is dead (and political fundraising can be done by PACs). . . ." "Just because one witness is the Chief State's Attorney and the other is Commander of the State Police doesn't mean we shouldn't put them under oath and question them at the public hearing. . . ." "Yes, we are recommending elimination of licensing and regulation of arborists. After all, we're only talking about a tree here."
And agencies are always delighted to see us. Each project begins with an entrance (forced) conference between the PRI executive director and the commissioner to introduce the project staff and discuss the scope, trajectory, and timing of the evaluation. The commissioner is always effusive in praising the PRI staff and delighted to have us spend four to eight months rummaging around his or her space. We say "we're happy to do it, we'll try to not be intrusive because we're here to help you." Then we add, "incidentally, we'll need complete access to all your confidential files, a list of every database, the password to your automated information system, and the name and phone number of every person on your staff."
At the end, generally half our recommendations are statutory and moved forward by the committee as bills during the subsequent session of the General Assembly, and half administrative (the agency can accomplish these without legislation). The PRI statute requires the agency to take corrective action in response to the committee's findings and recommendations. Often funding considerations (like our projected billion-dollar budget deficit for the current fiscal year) get in the way of this (along with any well-deserved bonuses for the responsible analysts).
Staff Resources--All staffers have advanced degrees--MAs, MPAs, MBAs, JDs--and substantial experience (e.g., as snow plow drivers, baloney slicers, potato pickers, and aides to recently indicted city officials). Half have been held in captivity by program review for more than twenty years and the latest hire came aboard almost seven years ago. Such little turnover must be the result of inspirational atmosphere created by top management.
The summer fellowships are an opportunity to exploit graduate students in the MBA, MPA, and JD programs of area universities. They get a few thousand dollars and we have four sets of hands to help gather data during a labor-intensive period. Further, they can experience the chase at windmills with us during the summer and slip away in September denying any knowledge of or association with the stuff we come out with in November.
Favorite excuse for a blown deadline--PRI is fully automated and has been for nearly two decades. The utility of it can't be overstated, particularly with blown deadlines--it's those damned people down in IT... the server is down again." Until this year, SAS was the statistical workhorse for large databases, but the office recently shifted to SPSS. With this new analytical resource, we hope to finally dispel the myth advanced by Mark Twain, a famous Hartford resident and sometime author, that statistics are the worst kind of lies (or at least produce tables that we can explain in less than one minute to our busy legislators).
Visit us--Finally, we're just off Interstates 95 or 90,
two hours northeast of Times Square and two hours southwest of Quincy Market.
Look for the golden dome atop the building many confuse with Cinderella's
castle at Disney World. All visitors are welcome, especially the folks
from Iowa - we'll freshen up your nutmeg. Office-bound can visit us at
our website, www.cga.state.ct.us/lpri
|
Legislative evaluators and auditors aren't exactly known for having "life of the party" personalities. In an effort to improve that image, the NLPES News offers some simple exercises to do around your office. Phone someone in the office you barely know, leave your name and say, "Just called to say I can't talk right now. Bye." Walk into a very busy person's office and while they watch you with growing irritation, turn the light switch on/off 10 times. After every sentence, say 'mon' in a really bad Jamaican accent. As in, "the report's on your desk, mon." Do this for one hour. Carry your keyboard over to a colleague and ask "You wanna trade?" Come to work in army fatigues and when asked why, say, "I can't talk about it." Speak with an accent (French, German, Porky Pig, etc.) during a very important conference call. Find the vacuum and start vacuuming around your desk. Hang a two-foot long piece of toilet roll from the back of your pants, and act genuinely surprised when someone points it out. |
Getting
Started on Designing Organization, Operations and Management Studies
Dr. Greg Rest (Greg is the Chief Methodologist of the Virginia Joint Legislative Audit and Review Commission.)
Sometimes legislators request that we "take a look at" the organization, operations, and management of some agency, without saying more specifically what should be examined. Consequently, our study mandate is very broad, leaving the study team to discern the key issues. In the absence of further guidance, the range of possible issues can seem daunting at first. Therefore, based on past studies, JLARC staff compiled a list of ten possible issue areas to consider examining when we get such a vague request for an organization, operations, and management study.
- Mission and Planning. This issue area often entails comparing "what is" to "what should be." "What is" are agency activities and performance. "What should be" are agency mandates, mission statements, plans, and other statements of intent.
- Relationships with Outside Entities. Sometimes an agency must rely on either public or private entities to get its job done. Other public entities include federal, local, or other state agencies. This issue area can also include the option of privatizing or contracting out services or functions.
- Organization and Structure. Is there a need for reorganization of the agency? Organizational structure components include: division of work (including job specialization); departmentalization of function; chain of command; span of control; and line and staff delineations.
- Authority and Responsibility. What is the source of the agency's authority, and is it sufficient for the agency to meet its responsibilities? Are there other external or internal obstacles that may prevent the agency from effectively carrying out its responsibilities?
- Staffing and Personnel Practices. A commonly asked question is: Are staffing levels appropriate, given the level of workload? Quantitative approaches for addressing this question can range from simple staff/workload ratios to more sophisticated data envelopment analysis and regression analysis. Turnover, recruitment, compensation levels, training, and morale can also be included in this issue area.
- Communication. What are the channels of communication in the agency, both vertical and lateral? How effective are these channels in providing needed information and coordination of agency activities?
- Financial Management and Funding. Are the necessary financial controls in place? Does the agency have sufficient funding to implement its mission fully? Is integrity of separate funds maintained? (For example, does the agency avoid using fees collected from one group to subsidize activities focused on another, different group?)
- Accountability and Oversight. What procedures are followed for internal monitoring or overseeing subunits of the agency? What procedures or structures are in place for external monitoring of the agency? What documentation requirements are in place and are agency staff adhering to them?
- Management Information Systems. What symptoms of an inadequate management information system are present, in terms of operational deficiencies, psychological deficiencies, and report content deficiencies? Issues to consider include: information costs and benefits; user satisfaction; technical capability (such as system response time and interfacing capability); operational characteristics (such as ease of data input and error rates); availability of internal controls; and availability of system support.
- Operations and Performance. This issue area includes using performance measures or unit productivity measures (which compare input measures to output measures).
Small
Shop Talk
I'm Not an Automaton!...and I Guess I'm Not Supposed to Be
Noah Miller (Noah is an Associate Program Evaluator for the Wyoming Legislative Service Office.)
may not have the experience of many other evaluators of working in a large environment. But, I do now have some experience working in a small program evaluation office. I work for the State of Wyoming, in the Legislative Service Office. Our section currently has four evaluators plus a manager, and one of those evaluators just started. I never imagined when I started less than a year ago that I would become one of the section's more senior staffers so quickly (read: lots of turnover recently).
When I first started, I thought the best idea would be for me to "roll with the punches," so to speak. I had just graduated from college and I was definitely the new guy. What I soon realized was that I was hired to bring not only another set of hands to the keyboard, but to bring new ideas into the office and the process of program evaluation. I was very nervous suggesting new things to other evaluators who had much more experience than I did. But everyone has different experiences and biases and therefore a unique perspective that is valuable.
Because all the evaluators in this office work very closely together and with the manager, one individual's suggestions can have a definite impact. This is one of the distinct advantages to working in a small office. Although most offices have an at least somewhat hierarchical structure, the greater the number of people, the greater the difficulty in bringing about bottom-up changes. The larger the work environment, the greater the number of people those ideas must filter through before changes can occur.
In our office, if a staff member makes a suggestion for changes in the evaluating process, whether it is something new this person learned at a training or developed themselves, that suggestion is honestly considered. We try many new approaches to evaluating all the time. Some things work very well for us, others don't. But the advantage is that we are able to attempt new practices in a short time with relatively few difficulties. When things don't work, it is pretty painless to revert to our old standards.
In my eleven months with this office, I have seen changes in the overall process of evaluating; we constantly fine-tune how we do what we do to meet the needs of our office, our committee, the Legislature and the state. As the process continues to evolve and the needs of our intended audience change, an office must have the ability to evolve with it. A small office such as ours has an advantage in this area, but flexibility is no less important for a larger office. Our reputation as offices that present reports that are clear, concise, complete, and unbiased (as much as is possible) is very important.
So, my suggestion to all evaluators is not to "roll with the punches," but rather to challenge the process of evaluating. If something in your office does not work or isn't the best it could be, don't complain about it, but rather figure out how to improve it. If your idea doesn't work or doesn't take root, try something else. Remember, we weren't hired as evaluators to be automatons or to have closed minds. We are evaluators because we can find and consider problems in our state governments and creatively inform our states through our reports what to be aware of and what may be fixable.
Assessing Audit Risk
Rick Riggs and Randy Tongier (Rick is an Administrative Auditor and Randy is an Audit Manager with the Kansas Legislative Division of Post Audit.)
As
auditors, we're concerned about reaching the right conclusion by deciding
whether to say a given program or operation is OK (has few or no
problems) or to say that the program is not OK (has significant
problems). Our conclusion is right if we say a program is OK when it is,
or not OK when it is not. By the same token, our conclusion is wrong if
we say a program is OK when it is not, or not OK when it is. The graphic
below shows these possibilities.
|
|
|
||
|
|
|
|
|
|
If the Program Is
Really OK... |
...and we say the program is OK |
|
...and we say the program is NOT OK |
|
|
|
|
|
|
If the Program is
Really NOT OK... |
...and we say the program is NOT OK |
|
...and we say the program is OK |
Generally, if our conclusion ends up being wrong, it is worse to say that the program is OK when it is not than to say the program is not OK when it is. That's because our audience expects us to find any problems that exist. If we don't, we've failed. On the other hand, if we think we see significant problems when none exist, we'll probably be seen as just overzealous, and that's more likely to be forgiven. For that reason, we want to avoid ending up in the lower right hand corner of the Box of Death.
MANAGING THE RISK THAT WE MAY REACH THE WRONG CONCLUSION
In planning and carrying out an audit, we can do things to increase the likelihood that we reach the right conclusion, and reduce our risk of being wrong. Here are the risks we can reduce:
Sufficiency Risk: the risk that we don't have enough evidence to reach the right conclusion.
Reliability Risk: the risk that we don't have competent evidence. We can reduce this risk by evaluating or testing the reliability of evidence or by using evidence from more reliable sources.
Relevance Risk: the risk that we don't have relevant evidence. We can reduce this risk by making sure that there is a direct logical link between the evidence and our conclusion.
Interpretation Risk: the risk that we don't understand or properly interpret the evidence we gather. We can reduce this risk by doing more thinking about what we've found, assigning more experienced staff, doing more supervision, and making greater use of experts.
Although we can control our risk that we reach the wrong conclusion, we can't change the risk that there really is a problem with the program. However, we can do things that will allow us to estimate that risk and assess its impact on our audit. For example, we can evaluate the operating procedures of the program; good procedures help reduce the risk of problems. We might, as a result of our assessment of the risk that there is or isn't a problem, focus our work on the areas of highest risk.
There are the two types of risk that the audited program has problems:
Inherent Risk is greater if the program is relatively new, is complex, or distributes cash or benefits. Agencies put management controls in place to manage inherent risk. (The greater the inherent risk, the greater the need for strong management controls.)
Control Risk (the risk that there still could be problems in spite of management controls) is greater if management has adopted poor practices, has less experienced staff, or has less educated or trained staff.
RE-ASSESSING RISK ONCE WE REACH A CONCLUSION
A conclusion that the program either is OK or is not OK is either correct or incorrect. If the program is likely to have problems (a high inherent or control risk), and we say there are problems, we find ourselves in the comfortable lower-left box. But if we conclude that a high-risk program doesn't have problems, that's not a comfortable situation. If problems really exist, we've just entered the lower-right Box of Death. (If you think the term "Box of Death" is overly dramatic, just look at Arthur Andersen.)
A final key point: we should assess audit risk on a regular basis during an audit--at the beginning, from time to time throughout, and especially at the end--as our understanding of the program changes. Periodic reassessment allows us to identify and make any needed adjustments to our audit work on a timely basis.
Liaison's Corner
Institute of Internal Auditors Expands Curriculum
The Institute of Internal Auditors (IIA) has expanded its government-related curriculum for the seminars area. These are offered at regular training sites and can also be provided at your office locations. The description of each course can be found on the IIA website: theiia.org. These seminars include:
- Assessing Risk in the Public Sector
- Audit Reports in the Public Sector
- Consulting and Other Professional Services for Government Auditors (NEW!)
- Evaluating Public Sector Controls Using COSO
- Management Accountability and Control per OMB A-123
- Objectives and Scope for Quick Response Audits (NEW!)
- Performance Auditing in the Public Sector
- Quick Response Auditing (NEW!)
Fall Training Conference Provides Training and Networking
Here's a fast fact quiz. What city boasts an average median temperature of 72.6 degrees Fahrenheit, 174 golf courses, 58 miles of bike paths, more than 500 restaurants, and more than 120 art galleries and has chili as its official food? Why, that's Scottsdale, Arizona, of course!
Scottsdale, a new city with an old soul, has blended the Native American, Spanish, and western pioneer cultures into an enjoyable experience for all visitors. From September 18-21, more than 192 legislative evaluators and auditors as well as faculty gathered in Scottsdale to learn new skills and network with colleagues from across the U.S. Sponsored by the Arizona Office of the Auditor General (and co-sponsored by the Arizona Evaluation Network), the fall training conference had a theme of "The Changing Climate of Evaluation-Responding to New Demands." From the opening plenary to the closing luncheon, the conference provided learning opportunities and enjoyable social activities. The following photos capture the atmosphere of the conference.
|
Did you miss the fall training conference? Well, all of the great information from the conference is only a mouse click away. Summaries of conference sessions are located on the NLPES website (www.ncsl.org/programs/nlpes). You may also access from the website summaries of NLPES sessions from the recent NCSL Annual Meeting in Denver, Colorado. |
Controlling
Project Scope on Uncontrolled Project Directives
Gary VanLandingham (Gary is the Deputy Director of Florida's Office of Program Policy Analysis and Government Accountability.)
What do you do if you are given a project with potentially limitless scope but also a set deadline for its completion? What do you do if such projects become a large portion of your office's regular workload? This is an ongoing issue with Florida's Office of Program Policy Analysis and Government Accountability (OPPAGA).
With the same law that created OPPAGA in 1994, the Legislature also mandates that OPPAGA conduct "Program Evaluation and Justification Reviews" of each state agency. The Legislature specifies that these reviews are to assess the efficiency, effectiveness, and long-term implications of state policies. Each review is to "be comprehensive in its scope" and at a minimum is to include the identifiable cost and purpose of each program; each program's progress toward achieving performance standards established by the Legislature under Florida's performance-based budgeting initiative; and alternate courses of action that would result in administration of the same program in a more efficient or effective manner. The courses of action to be considered must include whether the program should be reorganized, privatized, consolidated, shifted to another level of government, or eliminated. Whenever possible, OPPAGA reports must specify the potential savings that could be realized if the reports' recommendations are implemented. The intention of these reviews is to help the Legislature identify and eliminate programs and functions that do not add substantial value for taxpayers.
Between three and six agencies are up for review each year. Initially, the justification reports were due by December 1 of each year. However, the Legislature later provided that the reviews are to assist the Legislative Budget Commission in its examinations of agency budgets. To meet this time frame, the reviews will be completed by October at the latest.
You may be thinking "no big deal - a few of these studies are required each year." Well, consider the fact that Florida's constitution provides that the state is limited to a maximum of twenty-nine agencies. As a consequence, these are BIG agencies that run a wide range of programs. For example, the Department of Children and Families, Florida's human services agency, has about 35,000 employees (more than the total number of state employees in many states). The department's major functions include overseeing programs such as child abuse investigations, foster care and adoptions, public assistance, developmental disabilities, elderly abuse investigations, mental illness, substance abuse, and childcare. Because OPPAGA's justification review of this agency must include all of these major programs, the potential scope of a justification review is massive.
One overarching problem for OPPAGA is the fact that justification reviews are not the only thing OPPAGA must do. The Legislature has also mandated that OPPAGA examine all the state's school districts on a five-year cycle, conduct another twenty or so "regular" performance audits of specified programs each year, and conduct follow-up studies eighteen months after issuing any report.
The primary solution to OPPAGA's workload dilemma is the careful management of each project's scope. In applying this solution, we recognize several facts. The Legislature clearly intends for OPPAGA's reports to be substantive, comprehensive, and hard-hitting. Consequently, the office cannot do superficial reviews. However, it is not possible for us to look at every agency activity in depth within the time and resource limitations we faced. The fortunate answer to this dilemma is that there were certain programs and issues that the Legislature clearly cared more about than others. Thus, our path is to focus the studies on critical issues and programs within each agency while providing a general overview of all agency functions. In short, we put our resources on the issues and programs that have the biggest bang (legislative interest) for the buck.
In this risk assessment, we do several things at the beginning of each justification review. First and most importantly, we contact legislative stakeholders to determine what they care about in each agency and program. We meet with substantive and fiscal committee staff and, when possible, chairs and members, and keep in touch with them throughout the project. We give scoping priority to those issues with high legislative interest. Second, we look at our prior work in each agency to identify issues that we think are ripe for analysis. In our history, we have developed expertise (or at least general familiarity) in most agencies and have a lot of historical knowledge about prior and ongoing challenges facing the agencies. This is an advantage that larger offices enjoy--being able to cover a lot of government ground over time. Third, we look at other published research on similar programs in other states. As imitation is the sincerest form of flattery (OK, you can call it stealing), we will happily flatter another NLPES office that has come up with a great way of looking at a policy issue. Our goal is to do this risk assessment within a few weeks of starting each justification review, which enables us to plan our resource allocation (number of staff assigned to the project) on the risk/interest in the project.
While the law establishing the justification review procedure provides a checklist of items that our reports are to consider, we do not write them that way. Our experience shows that reports with a clear issue "storyline" have greater member interest than those formatted in the "garbage can--it's in there somewhere" style. Consequently, we focus our justification reviews around traditional finding issues. We place a brief discussion of the factors legally required to be included in a standard Appendix A if they are not particularly germane to the major themes of the report. However, we include a section in each report that addresses the bottom line questions of why each program was created, what would happen if the programs did not exist, and whether it would be in the state's interest to reorganize, merge, or privatize the programs or their component functions.
In short, the secret to success when dealing with very broad project scopes is to do what makes sense. You can't do everything, and thus you should spend your project time on issues that will produce the highest potential value to our client - the Legislature.
Understanding Performance Measurement
Rakesh Mohan (Rakesh is an evaluator with the Washington State Joint Legislative Audit and Review Committee. Rakesh also serves as the NLPES liaison to the American Evaluation Association.)
This article crystallizes the main thoughts that were put forth in a panel discussion on performance measurement that I moderated at the 2001 annual conference of the American Evaluation Association in St. Louis. The panel was titled, "Ask the Experts . . . Everything You Wanted to Know about Performance Measurement but Were Afraid to Ask." The panel members were:
- Joseph Wholey - Professor of public administration at the University of Southern California, a fellow of the National Academy of Public Administration, and a principal in the Council for Excellence in Government
- David Bernstein - Administrative Coordinator for Montgomery County, Maryland, a member of the board of directors of the American Society for Public Administration Center for Accountability and Performance, and an adjunct professor of public administration at George Washington University
- Ian Davies - Executive Director of the Capacity Development Network at the University of Victoria, Canada, and a member of the External Advisory Panel to the President of the Treasury Board of Canada on Managing for Results and Reporting to Parliament
- Jules Marquart - Director of Policy, Planning and Research at the Tennessee Department of Children's Services
Summary of Panel Discussion
Joe Wholey opened the discussion by outlining four reasons for having ongoing performance measurement systems: (1) increase management flexibility in return for accountability for results, (2) improve the quality of services through ongoing monitoring of outputs and outcomes, (3) communicate the value of services to stakeholders and skeptics, and (4) coordinate cross-cutting programs with complementary goals. Joe offered a number of examples where the use of performance measurement has been a success. His examples included the Chesapeake Bay Program, the Healthy People Program, the National Highway Traffic Safety Administration, and the Veteran's Health Administration.
Joe's opening comments set the stage for discussion on the conceptual framework and the "how to" of performance measurement. The following thoughts emerged from that discussion.
Performance, like beauty, is in the eyes of the beholder, suggested Ian Davies. To capture the idea of performance, causal or organizational models are used to develop measurable indicators, generate data, and communicate with stakeholders. Because these steps involve using judgment and making decisions, performance measurement is inherently a political and social process. Therefore, in addition to focusing on the technical side (that is, having a rigorous methodology) of performance measurement, it is important to pay attention to the other side of the equation that involves people. This means that we do more than just consult with stakeholders after a decision has already been made-they have a stake in the process and therefore want to know what is going on.
Ian also stressed the importance of agreeing on the language of performance measurement. Words like "performance" and "measurement" have different meanings to different people depending on their social, political, and professional contexts. David Bernstein echoed Ian's comments by saying that we need to define success so we recognize it when it walks in the door. For example, program officials might think that their program is successful if they have delivered services to their clients. Policymakers, on the other hand, may define the program's success in terms of making a difference in the lives of those it serves.
Performance measurement is all about process, commented Ian Davies. He said that the discussion on performance measurement is laced with words that focus on results. However, the success of the implementation of performance measurement practices rests on the process of how we do it and sustain it.
Using examples from the children's welfare program, Jules Marquart added that the performance measurement process must address three issues: (1) understanding the context and reasons for doing performance measurement, (2) ensuring the validity of outcome measures, which drives the collection and use of data, and (3) having the data available to measure outcomes.
On the issue of validity of outcome measures, Ian cautioned against behaving as if causalities between program efforts and outcomes are always known and they can be linked to the input stage for determining how to allocate money. In the real world, decisions about resource allocations are not only based on administrative rationality, they are made in a political environment.
Don't put a premium on meeting targets, advised Joe Wholey. He
said that setting targets is the least developed part of performance measurement.
We do not know how to set targets. David added that there is a need for
enhanced planning because we really do not know what we are doing in terms
of strategic planning and target setting. If we do not do a good job at
first in setting targets, we will either be disappointed because we did
not reach the pie in the sky, or we will be accused of creaming because
the targets were set too low in the first place. Instead, Joe suggested
that we should have multi-year performance goals (targets).
A single number is practically useless, warned David Bernstein. However, if a single number (as an indicator of performance) is gathered in the context of other numbers, it could be useful. In the absence of a context, a single number is dangerous because it paints an incomplete picture. Jules added that a single number collected over time could be meaningful for tracking performance or comparing one state with other peer states. Joe said that a single number could be meaningful if it is compared to a standard for driving performance improvement using an incentive system, which is necessary for bringing about a positive change.
Performance measurement should trigger questions, not necessarily answers, said David Bernstein. Performance measurement should not be used for evaluating programs. This does not mean that evaluators should not be involved in performance measurement, nor does it mean that performance measurement cannot be a useful component of an evaluation. What it does mean is that the best use of performance measures is for monitoring purposes-to see how a program is doing. Hence, such an exercise should trigger questions, not necessarily provide all of the answers.
Conclusion
As we reflect on these thoughts on performance measurement, we find that performance measurement is more than just counting inputs and outputs. Among other things, it requires understanding the political context, causal relationships, and stakeholder involvement. Wrong conclusions or recommendations about a program's performance can have serious policy and fiscal implications for the program, waste scarce public resources in collecting and analyzing data for the wrong measures, and affect the credibility of the audit/evaluation office making such conclusions and recommendations.
Resources on Performance Measurement
The panelists recommended the following useful resources:
- Audit Commission of England and Wales website: http://www.audit-commission.gov.uk/
- Bernstein, D. J. "Comments on Perrin's 'Effective Use and Misuse of Performance Measurement.'" American Journal of Evaluation, 20 (1), 1999, 85-93.
- Center for Accountability and Performance, American Society for Public Administration website: http://www.aspanet.org/cap/index.html
- Divorski, S., and Mckay, R. (U.S. General Accounting Office). "Studying Organizational Culture in a Federal Agency." Presentation at the annual conference of the American Evaluation Association, St. Louis, November 8, 2001, panel session #307.
- European Commission's evaluation website: http://europa.eu.int/comm/budget/evaluation/index_en.htm
- Governmental Accounting Standards Board's (GASB) performance measurement project, including Oregon's Multnomah County project (http://www.gasb.org/ or http://www.seagov.org/index.html).
- Melkers, J., and Willoughby, K. G. "Budgeters' Views of State Performance-Budgeting Systems: Distinction across Branches." Public Administration Review, 2001, 61(1): 54-64.
- Melkers, J., and Willoughby, K. G. "The State of the States: Performance-Based Budgeting Requirements in 47 out of 50." Public Administration Review, 1998, 58(1): 66-73.
- Shipman, S., and Vaurio, E. (U.S. General Accounting Office). "Integrating Program Evaluation and Performance Measurement." Presentation at the annual conference of the American Evaluation Association, St. Louis, November 8, 2001, panel session #307.
- U. S. General Accounting Office (GAO) website: http://www.gao.gov/
|
NCSL's Denver office moved to a new headquarters
building on October 1, 2002. The new address is 7700 East First Place,
Denver, Colorado 80230. The new telephone number is (303) 364-7700 and
the fax number is (303) 364-7800.
|
Argentina and NCSL: Building Legislative Institutions
Max Arinder (Max is Executive Director of the Mississippi Joint Legislative PEER Committee.
In my twenty-plus years of experience with the Mississippi Joint Legislative Committee on Performance Evaluation and Expenditure Review, I have had the opportunity to view state government from just about every angle imaginable. The experience has been challenging, eye-opening, rewarding, frustrating, exhilarating, depressing, entertaining-you name any point within the typical range of human emotion and I can likely describe a situation or event encountered in state government that has prompted that feeling. I would quickly note that I'm not complaining! On balance, the diverse challenges and experiences have made for a very personally rewarding career. It has also given me a great respect for the core institutions and processes of representative democracy and an appreciation of just how finely balanced a government must be to truly carry out the will of the people, while protecting a wide range of interests. I would further note that the knowledge I've gained at PEER has been magnified through my involvement with the National Conference of State Legislatures.
It is one such recent NCSL experience that is the subject of this commentary. In June of this year, I had the distinct honor and pleasure of being asked to serve as a team member for a series of seminars between the United States and Argentina entitled "Building Legislative Institutions." My role on the team was to talk about the rich diversity and heritage of state legislatures, giving the Argentines a glimpse of just how broadly the states have interpreted the legislative institution, its resources, and its processes. Senator Randall Gnant, President of the Arizona Senate, had the assignment of providing insights into the demands of legislative leadership--not a simple task by any measure. Finally, Mitch McCartney, current Chair of the Legislative Information and Communications Staff Section and Legislative Information Director for the Nebraska Legislature, served as our unicameral expert and shared his thoughts on the importance of civics education and the role of youth in maintaining the health and vitality of a representative democracy. For me, the experience was tremendously enriching, and I believe the other team members would agree with that assessment.
The seminars were held in three cities in regions representing a cross section of Argentine regional government. We started and ended our work in Buenos Aires, the capital and largest city in the country. Its beautiful boulevards and the distinctly European flavor of its architecture intrigued us. It is a city with a rich and colorful history and its citizens are justly proud of its reputation as one of the world's most exciting cities. From Buenos Aires we moved up the Parana River to the city of Santa Fe, a delightful city with even more delightful people. Our rewards for two intense work days exchanging ideas with members of the regional council were a lunch of local river fish prepared to perfection and an authentic Gaucho barbecue, direct from the Pampas. From Santa Fe it was back to Buenos Aires for a flight to, perhaps, the most exotic place I've ever been, Ushuaia, the capital of Tierra del Fuego. Billed as the southernmost city in the world, Ushuaia, backed by the foothills of the Andes chain, was blanketed in snow and bordering sky blue Beagle Bay. South of the Strait of Magellan, Ushuaia, with its physical isolation and natural grandeur, is indeed la cuidad fin del mundo. However, its people were anything but isolated. Billed throughout the city as a civics education opportunity, our presentations were attended by a broad range of citizens from many walks of life. Their questions were insightful and their interest in better government was palpable. But this article was not meant to be a travelogue, so enough about the places. What did I learn?
First, I need to provide a very big caveat to preface my comments. It's very risky to draw conclusions about something so complex as the political future of a nation like Argentina, even under the best of conditions. Add to that my admitted lack of knowledge and experience relating to the subject and one may easily conclude that my opinions are of little practical value regarding so serious an issue. However, it may be that my lack of sophistication regarding the political and economic woes of Argentina allowed me some advantages as well. For instance, I believe it allowed me to hear the frustrations and fears of the people with whom I had contact without the "analytic" baggage that so often accompanies subjects we know too well.
What I heard was, at once, eye-opening and disturbingly familiar. Amidst confusion, frustration, and loss of confidence people are looking to the government for answers, but they are not sure they can trust the answers they receive. Among the people I talked with there was a deeply rooted sense that the country should be doing better than it is doing. Looking around, they see an educated populace, adequate natural resources, a moderate climate, an appreciation of the benefits of a market economy, and a basic desire for personal liberty, freedom of expression, and the benefits of democratic institutions. Why, then, is their country plagued by disappointing system failures such as the one they are currently experiencing? Why do they seem prone to losing their government to powerful or charismatic figures? Why, with such strikingly similar attitudes on a number of core values, should the U.S. apparently be so much more successful in making the grand experiment of representative democracy work to keep the nation strong and economically viable? Why should the U.S. apparently be so much more successful in weathering the periodic trials that threaten the vitality of its democratic institutions? The people with whom I had contact are searching for answers to these questions.
What is disturbingly familiar in all this is that discussions of politics and government here in the U.S. are often punctuated with expressions of confusion, frustration, and loss of confidence. What then makes us politically different--if anything at all? I certainly do not have a clear answer to this question, but I might offer several possibilities that seemed to resonate with those whom I talked with in Argentina.
First, many people seem to feel that the party system, as currently practiced in Argentina, acts to remove government from the will of the people. There is a sense of disconnection between the people and the government. Though all are required to vote, they feel they are voting for a party ticket in which they have no true voice.
Second, parties in Argentina represent a much wider and more extreme range of political ideology with regard to how governments should function relative to the will of the people. With so many strongly held, politically viable ideologies competing for control, deal making and coalitions have become a way of life. For whatever reason, the majority of people in the U.S. tend to marginalize the extremes. Over time, centrist politics tend to win the day and both major parties are firmly committed to the basic ideology of representative democracy and keeping a finger on the pulse of the people. We have our share of dealmaking and coalition building, but in basic ways the psychology of the U.S. political process is quite different. Rather than being best characterized as deal making and coalition building among clearly defined ideological extremes, the U.S. political process can better be characterized as finding balance. It is this ability to find balance and marginalize the extremes that seems to give the U.S. the ability to weather difficult times.
Third, because of the role parties play in selecting candidates for office in Argentina, there appears to be a greater loyalty between an office holder and his or her party than there is between the office holder and the voters. In the U.S., of necessity, those elected to political office must keep the will of their constituencies in clear focus. In cases where the need for political action is at odds with constituent opinion, most U.S. politicians become quite adept at educating those constituencies and swaying opinion in their direction. If they cannot, or their judgment proves wrong, they tend to pay the price at the ballot box. There does not seem to be the same motivation to educate and create consensus with the voting public in the Argentine system, again creating a disconnect between the voters and elected officials that appears much more pronounced in Argentina than it is in the United States.
Fourth, Argentineans seem to be much more concerned, at least at this point, with finding the "proper" size and shape of government. Based on my conversations, the uncertainties that mark Argentinean life and politics have lead to a basic questioning of whether they have gotten it "structurally" right. Are their governments the right size, are they properly staffed, do they provide needed checks and balances without promoting gridlock? These are questions that concerned many I talked with. I believe these questions reflect a basic concern about the health of their core institutions and an uncertainty about the future of those institutions that does not currently exist in the States.
Finally, there seems to be less awareness of the importance of protecting minority interests to maintaining the overall health of a democracy. Though we continue to flirt with forms of direct democracy, most people in the States see a deliberative process as important to finding a proper balance for majority/minority interests. Again, in Argentina, there is currently such an apparent disconnect between the "will of the people" and government action that the focus of many is clearly on making government more directly accountable. There is less talk of the dangers of allowing the "will of the people" (read the "will of the majority") to direct decision making totally. Government by poll is not the best use of the opportunities presented in a democratic society. I would hope that this message does not get lost in their struggle for change.
As stated earlier, I certainly do not have answers to address the issues highlighted above. In fact, I may not even have the issues right. Someone with more knowledge of the complexities of Argentine political life would be a better judge than I. But I do know something about people, and the people I met are concerned about Argentina's future and are very interested in improving their government.
As to this particular exchange, I would make several comments. Foremost, I hope we were helpful in some way to our host provinces. With the language differences, there were times when I was not sure I was really providing the audience with the information they wanted and needed. At other times, I was concerned that I was not making myself clear in my responses and would have liked to have been able to spend time exploring the audience's understanding of what I was trying to say. The small group question and answer sessions were generally more gratifying in this regard. In some ways, the rather free form approach to our presentations from venue to venue was liberating. It kept us on our toes and got us focused on what we really came to say. I thought we jelled quickly as a team, worked well together at times of uncertainty, and were mutually supporting in our presentations.
I find these working exchanges to be thought provoking and personally enriching. I support the wisdom of bringing legislators and staff from the states into direct contact with their counterparts in other countries. Most of the world views U.S. government from the perspective of the national congress and the presidency. It is good to expose them more directly to the variety that is seen in the states and to help them to understand better the relationship states have to the federal system. Dialogues such as the one in Argentina can pay great dividends to the participating states and nations. I can only hope that we provided benefits to our host provinces; I know I bring back perspectives and insights that will help me to be a better steward of government for the State of Mississippi.
Office Happenings
Colorado
David Kaye recently joined the Colorado State Auditor's Office as a deputy state auditor, replacing Larry Gupton who retired. David formerly worked for the Colorado Attorney General's Office.
Mississippi
Oona McKenzie recently resigned from the PEER Committee staff to become an employee of the Mississippi House drafting office.
South Carolina
Dr. Elizabeth Oakman, Associate Auditor, retired in June after more than ten years of service with the South Carolina Legislative Audit Council. In addition to a plaque, staff presented her with a personalized calendar featuring LAC staff in photos appropriate for each month.
Tennessee
Congratulations to Dr. Phillip Doss who was promoted to Principal Legislative Research Analyst and to Emily Wilson who was promoted to Senior Legislative Research Analyst.
Texas
Ginny McKay, one of two Policy Directors for the Texas Sunset Commission, recently received the annual "Excellence in Government" award given by the Foundation for Texas Children.
West Virginia
Chris Nuckols recently resigned from the West Virginia Performance Evaluation and Research Division to accept a job with Wake Forest University's Medical School as a Prospect Research Analyst. Chris was with PERD for more than seven years, which is a long tenure since the office has been in existence for only eight years.
More From Argentina
© 2008 National Conference of State Legislatures, All Rights Reserved
Denver Office: Tel: 303-364-7700 | Fax: 303-364-7800 | 7700 East First Place | Denver, CO 80230 | Map
Washington Office: Tel: 202-624-5400 | Fax: 202-737-1069 | 444 North Capitol Street, N.W., Suite 515 | Washington, D.C. 20001
