|
NLPES Awards Program
Submission Information and Award Criteria
NLPES Excellence in Research Methods Award
NLPES Excellence in Evaluation Award
Impact Awards
Outstanding Achievement Award
NLPES Excellence in Research Methods Award
Submission Instructions
The NLPES Executive Committee will present this award to an office or offices which have demonstrated outstanding use of evaluation methodology in a report released during calendar year 2007. Up to three submissions, at the judges’ discretion, may be selected for the award in a given year. Submissions involving contracted work must demonstrate that office staff were involved in all aspects of selecting, developing, and applying the evaluation methodology, and participated in making decisions involving the analysis of data and the findings contained in the evaluation report.
Submissions for the Excellence in Research Methods Award must include a copy of the report and a narrative that identifies the methodology category for which the office wishes to be considered (see below) and describes how the methodology achieves the selection criteria. The narrative should be no longer than four pages. If the report does not contain a detailed description of the evaluation methodology, you may submit any additional documentation necessary to explain the analysis as an appendix.
Submissions must fall into one of the following three categories:
1. Technical difficulty and sophistication–such a submission should demonstrate the appropriate and successful use of advanced statistical techniques or software to efficiently and effectively address issues covered in an evaluation.
2. Innovative or unusual applications of established methodologies–such a submission should demonstrate the successful application of established qualitative or quantitative techniques in new areas or ways.
3. Exceptional breadth, depth, and scope of fieldwork–such a submission should demonstrate the successful application of a methodology that, while maintaining efficiency, takes the evaluation to a more comprehensive level.
Please also include a one-paragraph summary of the report and the name, phone number, and e-mail address of a contact person. (If your office receives this award, this information may be used in a handout at the NLPES awards luncheon at the NCSL Annual Meeting.) If you wish, you may also include the names and addresses of up to five individuals or offices you would like us to notify if you receive the award.
Selection Criteria
Judges will evaluate each application based on the information presented in the four-page written narrative.
Each submission must address the following criteria:
• usefulness – that the methodology was appropriate for the context in which the evaluation took place and was properly designed
• application – how the analysis was carried out
• the results it produced – the development of quantitative or qualitative data which, in turn, were used to develop relevant findings and/or recommendations
A few comments about the types of information to be contained in each section, as well as their relative importance, are provided below.
Methodology Usefulness/Design (35%) – The submission should demonstrate the selected methodology’s applicability to the evaluation’s objectives. The section should:
• Clearly state the desired objectives
• Detail the appropriateness of the selected methodology
• Identify the tie between the selected methodology and the objectives
• Identify why the method or the application of the method is unique or innovative or how the methodology advances the field of evaluation
Methodology Application (35%) – The submission should demonstrate how the selected methodology was applied during the fieldwork phase to meet the evaluation’s objectives in the most efficient and effective manner possible. This section should:
• Clearly explain methodology procedures
• Demonstrate the methodology is appropriate for the intended outcome
• Demonstrate how methodology design and work performed addressed the evaluation’s objectives
Evaluation Results (30%) – The submission should demonstrate that the selected methodology and the resulting conclusions were appropriate for the evaluation and were appropriately interpreted for the reader. This section should:
• Clearly demonstrate the selected methodology addressed the evaluation’s objectives and contributed significantly to development of the evaluation’s findings and/or recommendations
• Describe how the results of the methodology were central to identifying a problem, demonstrating the scope and or materiality of a problem, and/or were used to identify a solution to a problem
• Identify that the methodology’s resulting conclusions were appropriately and clearly interpreted for the reader.
Frequency of Submission
Any program evaluation office may apply for the Excellence in Research Methods Award. An office must refrain from applying for this award for one year after receipt of the award.
Deadline for Submissions
All submissions must be postmarked by Friday, April 25, 2008. Please provide a copy of your office’s submission to each of the judges listed below.
Information about Judges
Each year, the NLPES Executive Committee will select a panel of three judges to review application materials. These judges will be selected to minimize the potential for conflict of interest or bias. An office that receives an Excellence in Research Methods Award may be requested to make a staff person available to be a judge for the following year. Judges for the current year are:
James Barber
Legislative PEER Committee
P.O. Box 1204
Jackson, Mississippi 39215-1204
(601) 359-1226
Levi Bowles
Kansas Legislative Division of Post Audit
800 SW Jackson, Suite 1200
Topeka, Kansas 66612
Dan Jacobson
Minnesota Office of the Legislative Auditor
140 Centennial Building
658 Cedar Street
St. Paul, MN 55155
651-296-1230
NLPES Excellence in Evaluation Award
Submission Instructions
The NLPES Executive Committee will present this award to the office which is determined to have contributed the most to the field of legislative program evaluation during the four-year period beginning January 1, 2004 and ending December 31, 2007. One winner will be selected each year.
To be considered, each submission must consist of the following:
- A written narrative of no more than five pages which describes the office’s body of work, the impact it has made, and how it has contributed to the field of program evaluation during the application period.
- Attachment A – An organization chart that generally represents the four-year application period. Also provide a table that shows the number of performance/program evaluator FTEs by their relevant position in the organization for each year. Also provide summary information about the total annual expenditures, by year, for the application period.
- Attachment B – Summary information about the number of performance evaluation reports, special reports, briefing memos, and other work products issued during the four year period. Note: Do not include financial audits and other work performed primarily by financial auditors as this type of work will NOT be considered when evaluating each application. If possible, please include a complete listing of all work products to allow the judges to obtain an understanding of the topics covered by each type of work product. If such a list is unavailable, please provide a general description of the types of information provided and topics covered by each type of work product.
- Attachment C – Copies of up to 3 reports so the judges can become familiar with the appearance and visual quality of your work products. You may provide instructions for how other reports can be found on your Internet website.
- Other Attachments – are optional, and may provide any information that helps show your office’s results in meeting the selection criteria below. Note: Because of the time commitment required to properly review all applications, we ask you not overwhelm the judges with miscellaneous information. If other attachments are included, please be sure they are specific and clearly describe how they address the selection criteria.
Please also include the name, phone number, and e-mail address of a contact person in your office. (If your office receives this award, this information may be used in a handout at the NLPES awards luncheon at the NCSL Annual Meeting.) If you wish, you may also include the names and addresses of up to five individuals or offices you would like us to notify if you receive the award.
Selection Criteria
Judges will evaluate each submission based on the information presented in the written narrative and the attachments. The 5-page narrative should describe the office’s achievements in three specific areas. Those areas and their relative importance are listed below:
- Makes an Impact – 50%
- Body of Work – 30%
- Furthering the Field of Legislative Evaluation – 20%
A few comments about the types of information to be contained in each section are provided below.
Makes an Impact – Include information about the significance of your work and how it has positively affected state operations and policies. For example, you may show how your work has saved the state money, increased revenue, changed the way a program or service is provided, or led to laws passed or rescinded, etc.
Body of Work – Include information about what your office did and how that work was accomplished during the four-year period. Any comments you want to make about the information provided in Appendices A and B should be contained in this section. You should describe how subject areas were selected for evaluation. You may also want to describe whether you used contractors to perform special work or reports, whether any unique circumstances in your state impacted the selection of evaluation topics, whether a series of reports on a specific topic area were issued, etc. Other items you may want to consider would be to describe how your reports clearly and effectively communicate your message to legislators and other readers. If applicable, you may wish to provide information about the number of presentations made to legislative committees and other bodies.
You may also describe any noteworthy accomplishments or extraordinary actions taken by your office. Examples may include additional staff or appropriations given to increase to the amount or breadth of evaluation work performed, initiating new processes within your evaluation shop which have proved highly successful (and might be useful to others to emulate), and other accomplishments.
Furthering the Field of Legislative Evaluation – Include information about your participation in NLPES activities, providing training for other shops, or working with or sharing information with other states, local governments, and state agencies, etc. You may also consider including the use of unique or innovative methodologies to conduct evaluations.
Frequency of Submission
Any program evaluation office can apply for the Excellence in Evaluation Award. However, an office which wins the award in a given year may not be permitted to apply for the award for the following two years, and as described below, agrees to provide a judge in the two following years.
Deadline for Submissions
All submissions must be postmarked by Friday, April 25, 2008. Please provide a copy of your office’s submission to each of the judges listed below.
Information about Judges
Each year, the NLPES Executive Committee will select a panel of three judges to review award submissions. The offices which have won the Excellence award in the two preceding years must provide one judge each. The other judge will be selected by members of the NLPES Executive Committee. These judges will be selected to minimize the potential for conflict of interest or bias.
Judges for the current year are:
Kathy McGuire
Florida OPPAGA
111 West Madison, Room 312
Tallahassee, Florida 32399-1475
(850) 488-0021
Leslie McGuire
Georgia Department of Audits
Performance Audit Division
270 Washington Street, SW, Rm1-156
Atlanta, GA 30334
404-651-8927
Rakesh Mohan
Office of Performance Evaluations
Idaho State Legislature
700 W. State Street, Suite 10
Boise, ID 83720-0055
(208) 334-3880
Impact Awards
Impact Awards will be presented to offices that released a report during Calendar Year 2006 or 2007, which had documented public policy impacts. Submissions involving contracted work must demonstrate that office staff members were integrally involved in all aspects of the evaluation and report.
Selection Criteria: Successful applicants must document that the report submitted meets two or more of the following criteria:
• Dollar savings from implementing audit recommendations. Includes documented annual or one-time savings resulting from the implementation of the evaluation’s recommendations. To qualify, applicants must demonstrate that the recommendations were implemented and dollar savings resulted. It is not sufficient for the applicant to simply list the expected savings from implementing recommendations.
• Program improvements as a result of implementing audit recommendations. These are improvements to the organization made as a result of implementing the recommendations. These could be documented through various methods including:
- evidence of changes in an organization's structure, training, procedures, etc. or
- documented improvements in agency outcomes as determined by reasonable outcome measures
• Impacts from the legislature's perspective. These are documented results of the evaluation on legislators. Some measures of legislative impact may include:
- changes in statute
- additional funding and
- requests for presentations to legislative committees other than your oversight committee.
• Impacts from the public perspective. These are documented results of the evaluation on the public. This could be measured through various methods including:
- media attention specific to the report's impact; (media attention that simply discusses the report is not sufficient) or
- advocacy and special interest group responses.
• Impacts from other organizations' perspectives. These are documented results of the evaluation on similar organizations. This could be measured through the following:
- evidence showing that the evaluation methodology has been copied by evaluators in other organizations or
- documentation that the evaluation work is being used to improve the operations of other organizations that provide a similar service.
Submission Process
To apply for an Impact Award, provide (1) a copy of the nominated report; (2) a narrative of no more than two pages in length describing how the report has met two or more of the selection criteria; and (3) documentation of the report's impact described in the narrative. Please also include a one-paragraph summary of the report and the name, phone number, and e-mail address of a contact person for the report. (If your office receives this award, this information will be used in a handout at the NLPES awards luncheon at the NCSL Annual Meeting.) If you wish, you may also include the names and addresses of up to five individuals or offices you would like us to notify if you receive this award. (Note: Each office may submit only one report for consideration.)
Deadline for Submissions
All submissions must be postmarked by Friday, April 25, 2008. Please provide a copy of your office’s submission to each of the judges listed below.
Information about Judges
Each year, the NLPES Executive Committee will select a two judges to review award submissions. Judges for the current year are:
Trish Bishop
JLARC
Suite 1100, General Assembly Building
Capitol Square
Richmond, VA 23219
(804) 371-4563
Dale Carlson
Bureau of State Audits
555 Capitol Mall, Suite 300
Sacramento, CA 95814
916.445.0255
Outstanding Achievement Award
This award will be presented to an individual who has made outstanding contributions to the field of legislative program evaluation. The award will be presented subsequent to the recipient's retirement or exit from the legislative program evaluation/performance auditing field.
Selection Criteria
The recipient of the award will be selected on the basis of the following criteria:
• Long-time involvement in the field of legislative program evaluation and/or performance auditing.
• Active participation in professional organizations related to the field. For example, the recipient may have actively served on the NLPES Executive Committee, or may have been active in another professional organization like the National State Auditors Association, AEA, etc.
• Other contributions that have advanced the field of program evaluation (e.g., he or she was responsible for specific innovations, etc.)
• Reputation for high ethical and professional standards.
Submission Process
There are no formal nominations. See below for more information.
Selection Panel
NLPES shall establish a five-person select committee for this award, consisting of the Chair of the Executive Committee, the Vice-Chair, and three other persons with a significant history of both active participation in NLPES and experience in legislative program evaluation. The Chair will make appointments to the select committee, who will serve one-year terms but may be re-appointed by the next Chair.
The select committee will confer throughout the year, and determine recipients of the Outstanding Achievement Award each spring. The select committee will not seek formal nominations but will make its decisions based on members' knowledge of state and individual contributions. The select committee may, at its discretion, seek outside input with regard to eligible individuals. The select committee may, at its discretion, determine to grant one, multiple, or no awards each year. The select committee will make its decisions based on consensus.
The Chair of the NLPES Executive Committee will serve as chair of the select committee and will report results to the full Executive Committee and arrange with NCSL for the award to be given at the Annual Meeting.
Questions
Should you have questions regarding the NLPES awards and selection process, please contact Leslie McGuire, Chair, NLPES Awards Committee, at the Georgia Department of Audits, (404) 651-8927, or by e-mail at mcguirel@audits.ga.gov.
| 
