 |
Home | Contact Us | Press Room | Site Overview | Help | Login | Register |
 |
 |
| About NCSL | State & Federal Issues | Legislatures | Legislative Staff | Meetings | Bookstore | Legislators & Staff Only |
| NCSL Home > Legislative Staff: Staff Sections & Networks > NLPES > | Add to MyNCSL |
NLPES Training Resources
Fall Training Conference
1997 NLPES Conference Notes Evaluators: Truth Seekers, Bunglers, Hucksters and Acrobats
Evaluators: Truth Seekers, Bunglers, Hucksters and Acrobats University of Maryland, College Park Dr. Smith discussed the changes she has seen in the profession of program evaluation. She sees a move away from objectivity to advocacy and abandonment from the original commitment. She described four types of evaluators: the truth seeker, acrobat, huckster, and bungler.
Dr. Smith suggests that bunglers and hucksters survive because of a lack of clear standards and lack of proper review or oversight. She stated that evaluation has a simple goal-to render judgment on what is being studied using empirical data, which is studied and analyzed. Legislative Oversight: Past, Present, Future
The purpose of this panel was to explore the changes in the role of legislative staff in assisting legislators, the breadth of state-level activities subjected to oversight, and the tools and methods of those engaged in oversight. Karl Kurtz began this session with a discussion of three types of legislatures: those that are full-time, well paid and have large staffs (nearly all of the ten largest states); those that are part-time, low paid, with small staffs (mostly small population states in mountains and New England); and hybrids between these two types. Trends in state legislatures in the past 25 to 30 years include: growth in staff, better ability to make decisions independent of executives and lobbyists, a significant drop in turnover in lower houses, and more independent leaders with less centralized control. These trends held true until about 1990. New developments include term limits, staff cuts, ethics and campaign finance laws, anti-government attitudes, and a simple decline in trust in government. When panelists were asked whether their reports today were better or worse than 20 years ago many responded that they are better. Reasons given included: clearer objectives and writing style, more explicit criteria, more detailed recommendations, better technology, and more impact through better communication with the legislature. When asked about changes in dealings with executive agencies, Mississippi described a movement toward a random selection of reviews that could result in more positive findings. Florida stated that they have learned the art of political science and have become better at defending themselves and teaming up with constituents. Exploring the Foundations of Legislative Oversight: New Currents in the Stream of Evaluation Claremont Graduate University, California Dr. Scriven discussed many aspects of the field of program evaluation and evaluation methods in general. In the past, program evaluation was considered to be the determination of success or failure of meeting goals. This is monitoring and is not sufficient because nothing is said about side effects, no costs analyses are performed, and there is no analysis of efficiency versus effectiveness. There are many related fields of evaluation, such as product evaluations (e.g., Consumer Reports), personnel evaluation, policy analysis and performance assessment. Evaluation is a reflective discipline defined as the systematic determination of merit, worth or significance. Merit is the quality of the product. Worth is the value of the product in relation to its cost. Significance is defined as the importance of the product compared to alternatives. Scriven discussed four types of investigations: grading, ranking, scoring, and apportioning or allocating. Grading (A-F) is the allocation of entities into categories that have distinctions. Ranking is sorting high to low. Scoring is numerical grading or ranking. Apportioning is budgeting. Problems with each type of evaluation were discussed in depth. Scriven stated that indicators cannot be relied upon to make judgements because indicators may be invalid. One must look to outcomes. There is a great fight in program evaluation now about program indicators. They should be monitored sharply for abuse and indicators should be changed to keep a program fresh. Dr. Scriven stated that a program's "goal achievement" should not be used as the one and only criteria because it may miss potential program side effects and does not include other considerations such as the strength of the goal. Working on Interdisciplinary Teams
Discussion Topics Overcoming the Professional Stereotype Danny Miller entertained participants with jokes about stereotypes in different professions such as law, accounting and computer specialists. He discussed the elements of communication including the source, message, channel, receiver and feedback. He also mentioned the effects of individual personalities and individual perceptions on interdisciplinary teams and how perceptions and stereotypes may affect the ability to communicate well with team members of other disciplines. Group dynamics and the stages that teams go through were also discussed including forming, storming, norming and performing.The Lingo Barrier in Interdisciplinary Environments John Simpson discussed "lingo," why we use it, how it prevents communications and how we can overcome the lingo barrier. He gave examples of how acronyms and other words specific to certain disciplines can act as barriers. He suggested that interdisciplinary teams educate other team members on the meanings of words specific to their professions and that teams use terms understandable to all team members, when possible.Working on Interdisciplinary Teams/Elements of Total Quality and Key Features of Successful Teams Cheryl Ridings discussed elements of total quality management which should serve as the focus of interdisciplinary teams, including: employee involvement, focus on customers, product/service excellence, data based decision-making, and continuous improvement. She also discussed key features of a successful team: a clearly defined mission and goal; expertise in various disciplines; authority to do the job; management support-visibly and administratively; defining a finite life span for the team; retain team models that work and build on them; open communication between team members; and management facilitation, not dictation. Interdisciplinary teams appear to work well for the South Carolina Legislative Audit Council.Working on Interdisciplinary Teams Jan Bush discussed the reasons why an interdisciplinary team structure is used: to bring together the knowledge, skills and abilities necessary to accomplish a project; to improve communication among staff working toward the desired result; and to enable better decisions that improve the final product. To achieve these organizational goals and create an effective interdisciplinary team each team member must be able to identify that member's role on the team. Team members must know that they have a valuable contribution to make to the team and that their contribution will be recognized and accepted by other team members. She also discussed some problems that interdisciplinary teams encounter. Potential problems include:
Sometimes it is not efficient to have interdisciplinary teams with full-time interdisciplinary team members. For example, a lawyer on an interdisciplinary team may be expected to conduct routine legal research-within the ability of analysts-just because s/he is the legal expert on the team. Some ground rules for interdisciplinary teams: stick to the agenda; stick to the topic; stick to the time limit; make suggestions positive; discuss items under your control; all members participate; no discussion of other employees; divide work into parts; don't interrupt; practice alternative decision-making if no consensus can be found; and decide in advance how to deal with unproductive members. Data Collection: Using Active Listening to Maximize the Productivity of Interviews Brenda made the following points on the art of gathering information through interviewing. People withhold information when they:
Successful questions are:
Critical listening requires:
Barriers to effective listening:
In evaluating for relevancy and validity, did the speaker:
Meta-evaluation: Evaluating Our Own Work
Florida presented an approach they use to evaluate the effectiveness of their audit work. At the end of an audit, the audit team has a formal "de-briefing". The audit team formally responds to the following questions:
Florida uses this information for training and identifying areas where future improvements would be possible. |
© 2008 National Conference of State Legislatures, All Rights Reserved
Denver Office: Tel: 303-364-7700 | Fax: 303-364-7800 | 7700 East First Place | Denver, CO 80230 | Map
Washington Office: Tel: 202-624-5400 | Fax: 202-737-1069 | 444 North Capitol Street, N.W., Suite 515 | Washington, D.C. 20001
