 |
Home | Contact Us | Press Room | Site Overview | Help | Login | Register |
 |
 |
| About NCSL | State & Federal Issues | Legislatures | Legislative Staff | Meetings | Bookstore | Legislators & Staff Only |
| NCSL Home > Legislative Staff: Staff Sections & Networks > NLPES > | Add to MyNCSL |
NLPES Training Resources
Fall Training Conference
Moderator Janice Mueller set the stage for this interesting workshop by describing a dilemma often faced by program evaluators. On the one hand, the media are key contacts who help to publicize evaluation reports and highlight findings, but on the other, the media may portray findings in a manner that antagonizes the auditee and complicates the relationship between the evaluator and the audited agency. Mueller asked the panelists to talk about what makes an audit newsworthy, what the media think of program evaluations, and how legislative staff can help the media to report evaluation findings in an accurate and timely fashion. Steven Walters, the Madison Bureau Chief for Wisconsin's largest newspaper, opened the panel discussion by stressing the importance of evaluations, data, and accountability in our current political climate. He listed eight items that print journalists need from audit staff:
Andy Moore of Wisconsin Public TV said that the cumbersome names of audit agencies take up too much air time (and asked workshop participants to say National Legislative Program Evaluation Society.) Moore gave the audience some pointers on getting TV coverage of an audit:
Laurel Steffes, from the Wisconsin DNR, talked about the importance of organizing information releases and shared her agency's 10-step guide to creating a public information plan. Submitted by: Christina Tsuei
A number of states have adopted a variety of programs intended to increase work requirements and in other ways reform Aid to Families with Dependent Children (AFDC). With the passage of the federal welfare block grant, it can be expected that the states will develop additional reform proposals. In doing so, there will be increased interest in evaluating which new program ideas are effective. As evaluators are called upon to develop evaluation strategies for these new initiatives, there may be some lessons to be learned from evaluations already completed in Alaska, South Carolina, and Wisconsin. Jim Griffin discussed the evaluation of employment and training programs in Alaska. He emphasized that with unemployment in the double digits and particularly high in the winter, approaches that work in other states will not necessarily work there. He pointed out that different groups have different expectations for what a program will do, and what appears as a success to one group will not to another. He also stressed the difference between job search programs, which help people look for a job, and job training programs, and suggested that the more expensive training programs may not be more effective than job search programs. Kathleen Snider discussed the South Carolina Family Independence Act that was passed in 1995, and implemented statewide in 1996. The Legislative Audit Council is currently completing its first evaluation, with additional evaluations due every two years. The Council's review found that most case management staff were in favor of welfare reform, because it gave them tools to remove from AFDC people who were not complying with education and training requirements. Kathleen mentioned one problem in implementing these reforms was finding effective and appropriate training programs. Good programs are rare, tend to be expensive, and often require a GED, which many AFDC recipients do not have. Another implementation issue was that case managers gain a new, pivotal role, and many eligibility workers had to be trained and refocused on new responsibilities while still carrying out eligibility functions. Kathleen described some of the major obstacles to effective evaluation. First, when information systems are federally funded, they are developed to be responsive to federal reporting needs, rather than on providing data for performance measures. In South Carolina, the computer system had inaccurate and incomplete data that limited the questions that could be answered. In addition, different programs and information systems had different definitions for such basic program components as the number employed, the number participating, and the definition of a job. Another obstacle is having multiple state agencies involved in a program. In South Carolina, ten agencies were mentioned in the Act. While it is good to have universal goals that go beyond the social service agency, the number of agencies involved made coordination difficult, and evaluation time-consuming. Judith Frye discussed three welfare reform programs in Wisconsin: Learnfare, Two-Tier, and Wisconsin Works (W-2). Learnfare is designed to encourage school-age AFDC recipients to enroll in, attend, and complete high school. The Audit Bureau is completing evaluations of the statewide Learnfare program, targeted to teenagers, and of Learnfare Expansion, piloted in four counties, and targeted to 6 through 12-year-olds. Judith stated that one important lesson to be learned from these studies is to plan the evaluation as part of the reform program. Teenage Learnfare was already fully implemented when evaluation began, making the process much more difficult. Two-Tier is a 3-year demonstration project that modifies benefits paid to newcomers in four counties near the Illinois border to the benefit levels of the state they moved from. After six months, benefits return to Wisconsin levels. The evaluation is looking at whether this program affects families' decisions about whether to move to Wisconsin, and which county to move to. The evaluation consists mainly of surveys designed to determine what factors are important in choosing a place to live, and why people moved to Wisconsin. The evaluation was planned as part of the project, and baseline data was collected before the demonstration began. Unlike Learnfare, this evaluation is being completed by a contractor, and thus is using fewer Audit Bureau resources. However, this situation leads to other challenges, in having to deal with a contractor. W-2 will replace AFDC in Wisconsin when it is implemented next fall. Recipients will be placed in jobs at one of four levels. The Audit Bureau has been involved in specifying what data needs to be collected, so that these needs can be incorporated into the new system as it is implemented. The Audit Bureau is required by the Legislature to complete or contract for an evaluation of W-2. Submitted by: Emma Caspar Addressing Fraud During Program Evaluations
While many of us in program evaluation may not routinely encounter questions of fraud, perhaps we should be better prepared to handle it. The three panelists provided insight on how to handle these questions, based on their experiences with fraud. Allen Brown presented a practical overview of fraud, including its common elements, warning signs, and what to do if one suspects fraud. Fraud is a "deliberate act (i.e., requiring intent) to secure unfair and/or unlawful gain." Fraud contains three common elements: (1) motive, (2) opportunity, and (3) rationalization. Fraud is often difficult to find because, according to Brown, "fraud and stupid can look alike." Generally, fraud involves altered documents, repetitive acts, transactions with no reason other than to conceal, and lies. Fraud warning signs include the following: paying from copies instead of actual receipts, unreasonableness of quantities, unusual variances from year to year, generic invoices that are not numbered, excessive overtime, expenditure shifts to grants or contracts at the end of the fiscal year, and substitution schemes, such as substituting checks for cash. If an evaluator suspects fraud, Brown suggests securing the documents and contacting a supervisor. An office may also need to consult its legal staff. Ann Campbell provided an overview of California's experience with investigating allegations of improper governmental activities committed by state employees. California limits its investigative function in the Bureau of State Audits to state employees and departments. The main source of allegations is a 1-800 hotline. The office receives approximately 400 calls per month and typically investigates 10 percent of allegations. Before beginning a formal investigation, the office must establish predication, that is, a reasonable basis for believing that there is merit to a complaint. Charles Krueger emphasized management's responsibilities in addressing fraud, which include deterrence, detection, and investigation of fraud. Krueger recommends that every office have a fraud policy that enlists management support, institutionalizes the approach to fraud, reduces the impact of emotions and personalities, reduces the risks of investigative mishandling, and reduces the risk of successful civil litigation by suspects. A fraud policy should include the following statements: a prohibition of illegal activity, a clear definition of responsibility for conducting investigations, suspected wrongdoing will be investigated fully, management is responsible for establishing controls and procedures to detect suspected wrongdoing, management will cooperate fully with law enforcement and regulators, and a prohibition on cover-up and retaliation against witnesses. In summary, program evaluators and auditors need to be aware that they might have to address the issue of fraud while conducting studies. With evaluators who are aware of fraud and management that supports a consistent approach to investigating fraud, offices can address fraud during program evaluations in an efficient and effective manner. Submitted by: Jenn Moenck Feige Fraud Investigation Techniques
Although the focus of evaluation work is often on the issuance of public reports, this is not the goal. The goal is to get something done. We are in the business of information sharing -- sharing with peers, legislators, and citizens. Our business has many standards. SAS 53 (financial audits) says it is our responsibility to detect and report on errors and irregularities. Generally, errors are unintentional and irregularities are intentional. Similarly, the Yellow Book 4.12 (financial) 6.26 and 6.28 addresses irregularities and illegal acts. In all cases, it is important to know how to find fraud, what you are looking for, and what tools to use. The presenter offered case studies and examples of fraud on subjects such as medical fraud, stamp expenses, JTPA grant money, financial aid overcharges, and travel billing. Often (80% of the time), the detection of fraud comes from an agency employee who is attentive and responsible enough to recognize and report an unusual situation. In many cases, there are red flags going up. Administrators should be encouraged to do their job by being skeptical and reporting what appears to be unusual. Many times it is a matter of applying common sense. We should establish a relationship with employees who may be able to provide valuable information to support our suspicions. THINK, do not just process information. Petty cash is a frequent area in which fraud occurs. If you are examining financial records, be suspicious of copies rather than original documentation because copies can be easily manipulated. Pay close attention to the integrity of management by reviewing the managerial decisions that are made. Once in awhile, take the time to call a vendor and confirm receipts and other records. If you are examining payroll information, look at the overtime records specifically and use common sense on the number of hours employees are being asked to work or are claiming to work. Look at the composition of deposit information (the cash/check mix) and not just the balance. Perform an analytical review whenever you approach financial records. For example, when examining travel expense vouchers, look at expenditures over a period of time and examine the percent change over time. Use common sense. Be suspicious if, prior to an audit, crucial papers are lost in a fire. Lessons:
Submitted by: Bruce Prenda
Texas and New Mexico have developed formal review processes for larger information technology projects in response to concerns that some prior projects had exceeded their budgets and others failed to deliver the intended results. The legislatures in both states wanted to made changes to ensure accountability and coordination in statewide information technology expenditures. In 1989, Texas created an executive branch department to oversee an information resources planning process. Recently, New Mexico initiated its own information technology review process within the Legislative Finance Committee. While both states' efforts were motivated by run-away costs and failures of past projects, the structures they have created also establish a statewide strategic plan which defines a vision for the future use of technology. Texas's experience in the review and analysis of information technology projects relies on a cooperative review process that includes the Department of Information Resources, state agencies and universities, and a quality assurance team. The planning process is based on a state strategic plan developed by the DIR as well as individual agency strategic plans. These plans seek to lay the foundation for each agency's biennial operating plan for information technology projects. In addition, a quality assurance team reviews projects to identify those with a high risk of exceeding their budgets or failing to produce the intended results. These projects receive additional scrutiny and may undergo an independent review to determine risks and develop a risk management plan. The Texas State Strategic Plan for Information Resources Management provides a context for agencies' planning efforts and individual projects by defining a vision of a future Texas. The plan establishes goals, such as the integration of government services through a statewide information resource infrastructure. In addition, the report sparks the reader's imagination through short stories featuring Texas "residents of the future" using, or being served by the state's information technology system. Copies of Texas' strategic plan are available from the Texas Department of Information Resources. New Mexico's information technology review process is a relatively new effort. Although it bears some similarity to Texas' approach, the review takes place in the legislative branch with a greater emphasis on the budgetary process. Agencies are required to compete for funding for their information technology projects. Legislative staff review and score these projects, even if external funding sources, such as federal grants, have already been identified. Scores are based on criteria established by the Legislative Finance Committee and designed to ensure consistency in the state's use of information technology, effective project and risk management, and justification of each project based on a needs assessment and cost/benefit analysis. The panelists agreed that scoring projects is difficult. Rather than assigning numeric scores to each project for risk assessments or funding recommendations, both states appear to rely on a simpler system of categorizing projects into three groups: high, medium, and low. Panelists from both states distributed handouts that describe their information technology review process in more detail. In addition, several related publications, including the State Strategic Plan for Information Resources Management, are available from the Texas Department of Information Resources. Submitted by: Ingo Bensch Governmental Investment Practices
The investment practices session highlighted many of the problems that can lead to substantial investment losses. In recent years, governmental entities throughout the United States have reported losses caused by risky investment practices. Orange County, California incurred a $1.7 billion loss, primarily as a result of borrowing funds for investment purposes. In Wisconsin, the State Investment Board incurred a $95 million loss when it neutralized a number of interest rate swaps and structured notes. In Texas, the Teacher's Retirement System incurred substantial losses from loan defaults, which were secured by real estate that has lost much of its value. Generally, losses were due to investment strategies that were inappropriate for governmental funds either because of their sensitivity to interest and currency rate fluctuations, or because of the length of their maturities. Individuals making investment decisions based on poor judgment or imprudent action are primarily responsible for losses. To a lesser extent, the sellers of some of these investments are at fault for misrepresenting the risks. Further, management controls and oversight proved inadequate in most cases to prevent or detect the losses. A number of other contributing causes were identified, including the following: Investments were leveraged by borrowing funds to make investments or by including a multiplier in interest rate formulas. Investment staff that did not have adequate technical expertise, training, and continuing education. Complex investments risks were not identified or understood. Statutory guidelines and local investment policies were lacking or open to interpretation. Pressure was exerted by a governing body to increase investment earnings without any consideration of the corresponding increase in risk. The fundamental principles of public funds investment have not changed. Investment professionals should ensure that public funds remain safe and that investments mature when they are needed for governmental purposes, such as capital construction or day-to-day operations. Only after ensuring the safety and liquidity of public funds should alternative earnings rates be considered. One method to maintain an adequate investment system is to require governing bodies to annually review investment policies. These policies need to clearly define investment goals, acceptable levels of risk, and expected rates of return. Policies also need to identify allowable investments, investment maturities, and diversification objectives. In addition, procedures need to be developed that will promote effective management oversight and provide the checks and balances necessary to ensure compliance with policies. Questions related to governmental investment practices can be directed to Craig Barkelar , phone, (608) 264-6988. Outsourcing and Contracting of Performance Evaluations: Alternatives or Squandered Resources?
The three panelists presented an overview of their experiences in outsourcing and contracting performance evaluations. Each state had various reasons for contracting these performance evaluations. The reasons generally involved the need for specific expertise that was not available internally or a perceived need for an "outside" opinion. Idaho's Office of Performance Evaluation, however, established outsourcing as a method of achieving a political compromise when the office was initially established in 1994. The Office was directed to meet the following objectives: Provide independent performance evaluations. Involve the private sector. Avoid the expansion of state government. The first step for Ms. Van Maren, Director of the Office of Performance Evaluations for the Idaho State Legislature, in meeting this legislative directive was to determine the availability of resources in the area. She concluded that without hiring permanent employees her resource pool was limited to two Big Six firms, a purchasing agency, one university, and its graduate students. The next step was to "create credibility." She focused on ensuring that the performance evaluation reports would be accurate, verifiable, and consumable. Consumable meant designing a recognizable product. She also wanted to ensure that the evidence supporting the reports was verifiable and that the Office would retain all rights to the working papers. The Office found that the benefits acquired through the outsourcing experience were very dependent upon the firm or individual hired. Ms. Van Maren cited the following pros and cons for specific program evaluators contracted with by the Office:
The Office learned that outsourcing the majority of its work had limited pay backs because outsourcing required a high investment of its in-house staff to recruit, train, and oversee the performance evaluations. The in-house staff also spent a substantial amount of time reviewing working papers, revising the reports, and presenting the results. In addition, the Office found it challenging to respond to legislative questions after the contract was completed. In some cases, in-house staff spent time researching the working papers and the report in an effort to become knowledgeable enough to respond to legislative questions. Tennessee and Hawaii recommended the following as a result of their outsourcing contracting experiences: Communicate with individuals and groups that may be affected by the program evaluation results to obtain an understanding of the expectations for the final product. Ensure that you are familiar with the state's contracting requirements. Design the contracting process to ensure that the contractors are not only qualified but unbiased. Effort spent up-front designing the process will be rewarded throughout the contract. Research the contractors' vitae, publications, and previous employers. Allocate substantial time to oversee the contract, particularly the report phase. Establish at the outset the report style, organization, and purpose. Ensure that the contractor understands the expectation that technical jargon must be translated into layman's terms and /or clearly defined. Insist on one contact person, preferably someone local, who is responsible for the final product. This will minimize the time spent on the telephone, faxing, and sending overnight mail. Overall, the three panelists thought there were benefits derived from outsourcing and contracting. However, they did not think it was a viable means of avoiding hiring full-time program evaluators, unless there was a specific specialization necessary to complete the evaluation. Submitted by: Carol Ann Smith, CPA
A well-constructed database can be a powerful audit tool. With database software, the auditor can easily manipulate and analyze a large volume of data. However, as with any data gathering instrument, the usefulness of the database is only as good as its design. To be truly effective, the database must be designed to solve a particular problem or to answer a specific question. Databases have four types of data fields--alpha/numeric, logic, date/time, and memo. The memo field is unique in that its size is unlimited. It can be used to highlight concerns or draw conclusions. This has the advantage of allowing you to include your detailed comments right in your database. Thus, this information is available at your fingertips and can be accessed without going to a separate file or work paper. Databases can be used on a variety of audits. Scoff and Craig demonstrated how they used databases on three different audits. On two audits they constructed their own databases and on the third they used an existing database. One of the audits involved a review of family services case files. For this audit they created two files--one included data gathered from the input process and the other one included data gathered from investigation. Using the query function, they were able to identify cases displaying one or more similar characteristics. In another audit, they used an existing database to analyze the auditee's purchasing patterns. In both instances, it was apparent that using the database enabled them to analyze the data much more quickly and effectively than they could have using other manual or automated tools. Scott and Craig closed the session with a hands-on demonstration. Specifically, we created a simple database and performed various functions using Microsoft Access. I was amazed at how easy it was. Overall, I was really impressed with what can be done with databases. I have frequently used automated spreadsheets to gather and sort data, and I have been satisfied with the results. However, I can see that, in certain situations, database software would allow you to better analyze data in a much more efficient manner. Submitted by: Sylvia Hensley, Audit Principal
Best practices evaluations document the outstanding services that are cost effectively provided by local government units. State evaluation offices have assisted local government units with the means of communicating "how they did great work" with other units. As local units begin adopting service techniques with demonstrated cost effectiveness, the state, which transfers a large portion of its budget to local government, is expected to benefit as well. Things to Consider/Lessons Learned: Recognize the importance of local associations, experts, and local units of government. Local units of government are the audience. Use technical panels and consultants. Limit size, scope, and focus of review. Recognize the importance of surveys. Information from state and local agencies is inadequate. More research is necessary for best practices review than is necessary for program evaluation. The conceptual model is important. Keep in mind what is feasible. The following criteria are important in selecting topics for a best practices review: Importance of the service delivery area. Is the service considered important to the public? Potential for service delivery improvement. Is there a high potential for improving the effectiveness of service delivery? Potential for cost savings. Is there a high potential for saving public money? Number of jurisdictions. Are there many jurisdictions delivering the service? Availability of data. Are there adequate data available to conduct a useful study? Research feasibility. Is the service area amenable to research and measurement, given available staff and research methodologies? Balance among topics. Among all topics chosen, is there a balance between county and municipal issues and a balance across functional areas of local government? Timeliness. Is this a good time for a study of these topics? Some advice on how to deal with the discovery of bad practices was also provided. Although there is a responsibility to report it to the legislature, it was recommended that there be no finger pointing at specific jurisdictions. The absence of a best practice can be used to encourage its use. Information was provided on the American Productivity & Quality Center, a nonprofit education and research organization that helps business, government, health care and education change and improve. The Center can be reached by phone at 1-800-776-9676. Submitted by: Kristin Breen
Perhaps the word "marketing" is not properly descriptive for the actions of evaluation staff interested in ensuring that reports both meet needs of legislative bodies and are given adequate attention by them. Marketing carries a sales connotation, and all three presenters cautioned evaluation offices to recognize the fine line between promoting a product and lobbying for its use. Respect for the legislature is key. Some of the marketing obstacles worth avoiding are: blind siding critical staff and members with reports; being untimely; being seen as having "own agenda"; self-promotion; and public press releases (which may be seen as self-serving). Gary VanLandingham's advice on report marketing was to always remember that the legislative process is chaotic and distracting to legislators and, "If they don't read it/remember it, it doesn't exist." To cut through this noise, 80% of OPPAGA reports are less than twelve pages, and all of the information is "hard-hitting" (i.e. no program descriptions or auditing standards reviews). Reports are very readable, using sidebars and graphics, and are widely distributed. All key staff are briefed on their publication. OPPAGA also keeps report findings in circulation through committee testimony and biannual report synopses. Publication of 18-month follow-up reports assess whether problems are resolved. Every three years OPPAGA presents a report of the cost savings due to their work. Marketing the product means making it conveniently available on demand. Toward that goal, the Florida Government Accountability Report (F-GAR) is being developed, a one-stop source of information on over 400 state programs, scheduled to be on-line in July, 1997. Brief (4-5 pages), searchable, and linked to other relevant information, F-GAR will enable users immediate access to indicators of program performance and accountability. Max Arinder and his staff do strategic planning with their committee once a year, reaffirming the mission and purpose toward which PEER reports should strive. Every four years a staff/committee retreat is held to discuss long term evaluation considerations. It was suggested that evaluation products be marketed to a need and contribute to the policy making debate, without taking the place of it. Reports should also stress education value and quality, as well as utility in problem solving. Marketing should avoid self-promotion or creating an artificial demand for reports. Also, active listening of committee discussions and resulting personal contacts (not surveys) are the best methods of understanding what needs are to be met in evaluation reports. Roger Brooks suggested that a focus on "product quality" should be secondary to market research to discover evaluation product end-users' true desires. In Minnesota, a list of evaluation ideas is developed through discussions with committee chairs and inquiry letters. After the list of topics is pared somewhat by the audit commission, chosen topics are sent to end-users in a survey to measure interest. Six or seven topics are assembled for round table discussions (which are taped) with legislators invited specifically for their intimate involvement with the topics. Evaluation proposals are then made to this legislative audience. All of this is done -- and it's been done for ten years -- right after session, with per diem paid, and before vacations begin. To ensure that quality results are seen, packaging, promotion and placement are given attention. Emphasis on headline development of short reports, sidebars and bullets and repackaging some reports with better covers helps readability. Report summaries are sent to all legislators. A brochure called "Evaluating State Government" is sent to a huge mailing list each year. Reports are also placed widely, including libraries and the web site. Submitted by: Mark Bucherl Using the Internet for Research NLPES Members Other Government Sources Reference Sources News Media Discussion Groups National Legislative Program Evaluation Society National State Auditors Association (NSAA) |
© 2008 National Conference of State Legislatures, All Rights Reserved
Denver Office: Tel: 303-364-7700 | Fax: 303-364-7800 | 7700 East First Place | Denver, CO 80230 | Map
Washington Office: Tel: 202-624-5400 | Fax: 202-737-1069 | 444 North Capitol Street, N.W., Suite 515 | Washington, D.C. 20001
