With a stronger economy throughout the country, employers are having trouble attracting and retaining staff. Legislative audit offices are also experiencing these difficulties, and the presenters describe some of the ways their offices have addressed staff retention.

Under the Florida OPPAGA's old evaluation system, a supervisor did annual evaluations on employees' anniversary dates using seven rating categories and five rating levels. But the system wasn't very effective and the office decided to make changes. They simplified the job classes and rating system, identified key performance areas, broadened the sources of evaluation input, and now conduct all evaluations in June. They implemented the 360 and quality assurance feedback system and encourage employees to formulate individual development plans. This new system has helped in identifying the best and weakest performers, but some analysts are reluctant to provide negative 360s due to concerns about confidentiality.

The Arizona OAG has grown from a staff of 9 in 1978 to about 49 in the mid-1990s. Turnover in its performance audit division has grown from 7 full time equivalents in 1994 to a projection of 19 for 1998. The office has broadened its recruiting efforts from advertisements in local newspapers to recent listing on Internet sites, professor referrals, and on-campus presentations. Its recruiting efforts are impacted by a lower unemployment rate in Phoenix, high job growth rate, and better outside salaries. The Arizona OAG conducts an extensive five-step screening process that includes case and data analysis exercises before hiring new staff. To retain staff, the Arizona OAG has established a "buddy" system, conducts ongoing training, and allows flexible work schedules such as part time and telecommuting. However, this system is new and the office has not been able to test its success.

Some of the options for strategies to attract and retain staff that California's State Auditor is considering or planning to use include the following:

a.) "Sell, Sell, Sell" - The office can "sell" itself by encouraging its management to take the responsibility to convince its employees that the bureau is actually a good place to work. Management must be aware of and address issues that employees face.

b.) Invest In Assets - The office can invest in its most important assets--employees--by leading, guiding, mentoring, and challenging them. Management can share information faster with staff, and encourage them to develop personal development plans with a format that addresses their relationships and performance with all members of the office staff.

c.) Reward Performance - Management can be honest and direct with praise, providing good employees with cash payments, performance vouchers, and other kinds of rewards. The goal is to have employees who have been sold, invested in, and rewarded to stay.

Back to the top

Integrating Computer Tools into Program Evaluation

Moderator:

Craig Kinton, Audit Director, Texas State Auditor's Office

Presenters:

Gary Brown, Audit Division Administrator, Michigan Office of the Auditor General

Beth Larson, Assistant Auditor General, Michigan Office of the Auditor General

Charles Dormann, U.S. Department of Agriculture

In this session, the presenters shared examples of how they use the Internet, spreadsheets, and electronic working papers to enhance their audits. The Texas State Auditor's Office (TSA) uses all of these tools. TSA used the Internet and Microsoft Access in its Year 2000 Embedded System Survey. TSA needed survey information from 178 agencies. It developed an approach whereby agencies could either fill out a paper survey and return it to TSA or complete the survey on line on a web page TSA developed. Approximately half of the agencies chose to complete the survey on line. The benefits of the on-line survey include reduced data entry and data verification; automatic accumulation of results of the survey in Access; easier analysis through use of Access; and positive comments from the agencies. Because of its accessibility and capabilities, TSA also used the Internet to solicit information from agencies in other reviews.

TSA also uses spreadsheets and relational databases in sampling, combining, and comparing the huge amounts of data it reviews. For example, TSA used Microsoft Access to combine and compare voluminous records to discover overbilling by service providers of alcohol and drug abuse counseling. Using these computer tools, TSA developed suitable evidence for court proceedings from medical and billing records and time cards of counselors. TSA also identified 400 documents that were altered by five employees, billings when the patients were not in the hospital, and billings that exceeded the hours employees actually worked.

TSA has moved to electronic working papers using a system developed for them by Pricewaterhouse. TSA is using electronic working papers in its financial and compliance audit areas and will eventually also include the program evaluation area. TSA sees the benefits of electronic working papers to include increased efficiencies, easier handling of redundant tasks, more timely reviews of working papers, automatically generated exception reports, and information that helps determine audit progress.

Charles Dormann, from the USDA, introduced and discussed COBIT ( Control Objectives for Information and Related Technology), a concept and product developed by the Information Systems Audit and Control Foundation. COBIT is an IT tool that helps organizations meet their objectives by facilitating the understanding and management of information and IT risks. COBIT is touted as an authoritative governance tool that will bring attention to corporate governance; help hold management accountable for resources and meet the specific need for control of IT resources; provide a framework for risk assessment; and improve communication among management, auditors, and other users.

The Michigan Office of the Auditor General (OAG) has used various software applications and approaches to accomplish its audit objectives. In three recent audits, OAG used the mainframe to extract data and a statistical software, SPSS, to analyze and categorize a wide variety of data. Further, OAG used the IDEA software to sort, summarize, and join data into a single file.

The panel responded to the following questions:

Q. Which is a better analytical tool, ACL or IDEA?

A. ACL is more powerful, but IDEA is more user-friendly.

Q. Since computer technology changes rapidly, should we be concerned about the compatibility of equipment when archiving working papers?

A. To a certain extent, yes. However, because of improvements in the laser technology, archiving working papers using laser-reading equipment (CD-ROMS)is better than using the magnetic medium (floppy disks).

Q. How much does it cost to convert to an electronic working paper system?

A. Texas spent approximately $1 million.

Back to the top

The Yellow Book: Is It, or Can It Be, Relevant to Your Work?

Moderator:

Kurt Sjoberg, State Auditor, Office of the California State Auditor

Panelists:

Janice Mueller, State Auditor, Wisconsin Legislative Audit Bureau

Kent Hutchinson, Manager Quality Assurance and Publications Section, The Florida Office of Program Policy Analysis & Government Accountability (OPPAGA)

This discussion addressed the future evolution of Yellow Book standards. The discussion also highlighted varying opinions on the relevance and value of some of the Yellow Book auditing standards in the performance audit/program evaluation area.

The Future of Yellow Book Standards

Mr. Sjoberg, a member of the advisory council for auditing standards (Yellow Book committee), shared his insights on the future development of the Yellow Book as related to performance audits. He stated that the GAO is attempting to make the Yellow Book more relevant in this area, though it is still early in the process. The GAO recognizes that the current Yellow Book is too narrow and wants to expand its usefulness beyond the traditional auditing community.

On the other hand, auditing has many desirable attributes, such as rigorous standards. Therein lies a conundrum: the GAO must retain the standards' best attributes as much as possible, while broadening the use of the Yellow Book.

The future may include broadening the types of work that would come under the Yellow Book definition of performance audits such as evaluation, policy and program analysis, assurance services, and future-looking work. In addition, the committee may identify a basic core of standards for any work that falls under the Yellow Book, including qualities of judgment, rigor, independence, and due professional care. Branching out from this core, like spokes on a wheel, would be additional standards for specific types of audits. For example, financial audits would add fieldwork standards. In addition, the Yellow Book committee may also address other types of non-audit work such as policy analysis, analytical studies, and alternative analysis.

Different Views from Different States

The states of Wisconsin and Florida do not cite the Yellow Book standards in their performance audits, although they actually adhere to most of the standards, whereas the state of California cites the Yellow Book for all its audits, including performance, financial and compliance.

In fact, in California, it is a statutory requirement that the Office of the California State Auditor follow Yellow Book standards in all of its audits. California saw a need to have the auditing standards cited as a statutory mandate to prevent the politicization of its work. Any political attempts to influence the State Auditor's reports can be deflected on the strength of having the standards required by law.

Mr. Sjoberg discussed the challenge of producing relevant, timely audit reports while still following the sometimes demanding standards. However, states have the flexibility to use different approaches, delivering responsive work while still meeting standards. California does not find the Yellow Book standards burdensome, nor does following the standards slow down its work.

While agreeing that, overall, the Yellow Book provides important basic guidance, Ms. Mueller and Mr. Hutchinson had concerns about Yellow Book standards for performance auditors. They do not feel the rigid "checklist mentality" of Yellow Book peer reviews does anything to improve the research or quality of reports. They are concerned about adding time and cost to audits without adding value. Mr. Hutchinson prefers a peer review approach that emphasizes the impact of the reports, rather than the quality of the workpapers. Mr. Sjoberg added that other states shared the perception that the peer review is too focused on the process, with its narrow emphasis on documentation.

Ms. Mueller and Mr. Hutchinson also questioned the value of accounting-oriented continuing education requirements for program auditors, and felt that relevant training opportunities are few and costly.

There was also concern that Yellow Book standards would place limitations on flexibility and use of judgment. For example, Ms. Mueller wants to keep the flexibility of being able to determine what type of reports best serve the needs of her state.

Ms. Mueller and Mr. Hutchinson also expressed reservations about the ability of their respective offices to produce relevant work if encumbered by rigid Yellow Book standards. Mr. Sjoberg added that other states felt that standards should be more pragmatic, eclectic, and utilitarian. The Yellow Book is sometimes seen to be chauvinistic toward accounting and irrelevant for performance audits.

Mr. Sjoberg commented that the Yellow Book committee is mindful of these concerns and will consider them in the future development of the Yellow Book standards.

Back to the top

Building Teamwork from the Inside Out for Improving Organizational Performance

Presenter:

Lyn Wiltse, PDSA Consulting, Inc., Kirkland, Washington

Suppose you find yourself on a team -- not any team, but one designed by Lyn Wiltse to dramatically demonstrate and emphasize how teamwork really is different from working independently. You and four other team members are given several pieces of paper cut into geometric shapes. Your individual objective is to make a square in front of you and the team's objective is for all the team members to complete their own squares. But making a square isn't so easy, because initially you may not have all the pieces you need. Or, you may have too many. Ms. Wiltse tells you that if your pieces and the pieces of your teammates were combined and arranged properly, all the pieces would make five squares. You may give your pieces away to other team members, but you cannot take any pieces. You are not permitted to speak or gesture. The clock is running.

Your first feelings are frustrating because you don't feel in control. It's hard to resist the urge to break the rules. Quickly, you realize that you are expected to do more than merely reach your own objective. You become attentive to the needs of your teammates and give them some or all of your pieces, trusting that they may be able to make their squares even if you are unable to immediately make your own. You find that you can neither dominate the team, nor remain passive. Your teammates give you some of their pieces and rearranging them, you complete your square. You sit back, satisfied, waiting for the others to complete their squares. But they are unable to because they need some pieces from your completed square. You realize that if you give away some of your pieces, the entire team will be able to reach its objective. But you resist breaking up your square because you don't want to start over. It's hard to revise your thinking. Finally, you do give in and give up some key pieces, your teammates give you some of theirs and just before Ms. Wiltse says time is up, five squares appear on the table. Mission accomplished! We learn that trust, delegation, sharing a vision, not dominating, not tuning out, and flexibility are some characteristics effective team members possess.

If we wish to lead successful teams, Ms. Wiltse asks that we consider Maslow's hierarchy of five needs. The hierarchy can be thought of as a pyramid, with physiological needs, such as food, shelter and clothing, at the bottom. Once these needs are met, safety needs, such as job security, stable family life, and a safe working and living environment, can be addressed. Ms. Wiltse contends that these two broad needs, physiological and safety, are management's responsibility. Only after these lower two levels are fulfilled can real teamwork based on a feeling of belongingness, the third level, be achieved. When workers feel they belong, they can aspire to the next higher level, esteem. Esteem includes the desire for achievement, confidence, being useful in a company, praise and reputation. Finally, according to Maslow, when individuals have their need for esteem met, they can begin to maximize their potential, solve more complex problems and improve their capabilities at the highest level, self-actualization.

In addition to Maslow's hierarchy of needs, Ms. Wiltse wants us to understand that some forms of motivation work better than others. For example, intrinsic motivators, such as learning, a sense of achievement, and joy in work may be more effective than extrinsic motivators, such as a title, monetary reward, and recognition.

Applying what we know about motivation to teams, we may be able to get better results. For example, Ms. Wiltse says that about 80 percent of teams are project teams that are essentially management driven and only 20 percent are employee driven process improvement teams. If we could flip the ratio around, we might see boundaries removed and effectiveness increased. Consider the advantages:

Process Improvement Team	Project Team
Employee driven	Management driven
Collects new data	Uses existing data
Treats root causes derived from data	Begins with end (solution) in mind
Continuously works on draining the swamp	Kills the worst alligator(s), then disbands

Successful teams typically have 5 to 7 diverse members (yes, disagreement is an asset), with all members participating, clear mission and roles, and well-defined social norms. Such teams also obtain management support; they have developed a way to solve problems and they are aware of the group process.

Finally, Ms. Wiltse described the various stages of teamwork using Tuckman's model of team development: Forming, Storming, Norming, Performing and Mourning. The Forming stage is characterized by answering the question, "What are we supposed to do?" Tips for the team leader in this stage include setting the team mission and norms, clarifying expectations, and assuring management support. In the Storming stage, frustrated team members question each other in an attempt to determine how much power each has. The team leader will get through this stage more easily by acknowledging feelings, reviewing norms and sticking to the problem solving methodology. Norming is accomplished when members combine their resources and experience to accomplish the task. Some tips for Norming: acknowledge small wins, rely on facts and data and promote diversity. When members anticipate each other's needs and accomplish tremendous amounts in very little time, the team is in the Performing stage. Finally, the team is in Mourning when members feel a sense of loss after the team completes its mission. In this stage, it's time to identify what we have learned, carry them forward and celebrate!

In conclusion, effective team leaders know how to motivate people, place emphasis on improving processes instead of merely completing the project and they guide their teams through each stage of team development. Effective team leaders use the PDSA approach: they Plan their activities, Do what they have planned, Study the results and Act on what they have learned.

Back to the top

Award Winning Methodologies

Moderator:

Janice Mueller, State Auditor, Wisconsin Legislative Audit Bureau

Presenters:

R. Kirk Jonas, Deputy Director, Virginia Joint Legislative Audit and Review Commission (JLARC)

Lois Sayrs, Ph.D., Senior Methodologist, Arizona Auditor General's Office (Arizona OAG)

Paul Stuiber, Program Evaluation Director, Wisconsin Legislative Audit Bureau

The presenters described the reports issued by their offices that have recently received the Award of Excellence in Methodology from the National Conference of State Legislatures. In general, they believe that an audit methodology should be as rigorous as required for the purposes of the study and that the auditor should use methodologies that are sufficient, appropriate, direct, and simple, but should not push methodologies beyond their suitability or usefulness.

Kirk Jonas described three studies the JLARC has recently issued.

To evaluate the current system and develop alternatives for staffing standards for sheriffs' offices, JLARC conducted a comprehensive survey to identify full-time equivalent positions committed to work in different service areas. It had a 100 percent return rate, but in some cases, staff made personal visits to secure required data. It used correlation and regression analysis techniques to quantify relationships between number of staff and workload indicators and employed alternative interpretation and use of regression statistics.
To assess Virginia's revenue forecasting process, JLARC found that most of the substantial gaps in forecasted revenues and actual revenue collections were attributable to a declining economy. It used two separate sets of criteria to objectively assess the forecast process and models.
To review the solid waste facility management and its impact on minority communities, JLARC used a computer-based Geographic Information System to implement a more refined definition of community using census block levels as the unit of analysis, rather than a local jurisdiction or census tract.

Lois Sayrs described the Arizona OAG's several-year study of the impact of the at-risk pre-school program in 90 districts. The study covered 4700 children in public, private, and headstart classrooms. Barriers to the audit included uncooperative agency administration, on-going implementation of the program, opposition to testing the academic gains of pre-schoolers, and the varied population in the state.

The OAG mixed methods, including tests administered by the teachers, structured observation, teacher observation, and interviews to measure immediate effect, short-term effect, and four-year effect. These measures were quantitative as well as qualitative. The OAG also looked at program implementation including how districts get funds to private schools, program oversight, costs compared to other like programs, and statutory mandates. Keys to the success of the audit were the balanced methods, quasi-experimental methods, and clear results.

Paul Stuiber discussed the Wisconsin Legislative Audit Bureau's evaluation of the Learn Fair Program. This program is designed to improve school attendance and reduce the drop-out rate of students who are 13 to 19 years old and AFDC recipients. Over a period of several years, the study measured the difference between results in a control group (without the program) and a test group, determining whether those in the program were more likely to remain enrolled.

One of the major challenges was working with county social services agencies to randomly assign individuals into a control group or a test group. Although this five year study won a methodology award, one of the frustrations associated with it was that there was no impact because of a change in the welfare system.

Back to the top

Privatization of Correctional Facilities

Moderator:

James Barber, Deputy Director, Mississippi Joint Committee on Performance Evaluation and Expenditure Review

Presenters:

Professor Charles Thomas, Center for Studies in Criminology and Law, University of Florida

Ben DeGroot, Administrator, Community Correctional Facilities, California Department of Corrections

Many services traditionally provided by government are now delivered by private enterprise, including the construction and management of federal, state, and local facilities. The panel discussed the reasons why privatization of correctional facilities has expanded rapidly in recent years. One of the main benefits was cost savings to the public. Private companies can construct the facilities faster and operate these facilities at a lower cost than the government can.

Dr. Charles Thomas is a professor at the Center for Studies in Criminology and Law at the University of Florida who also does research on the privatization of government services. He has assisted the State of Florida and other states in making decisions regarding privatization of correctional facilities. He described the short history of privatizing correctional facilities, which only began in 1985 with the State of Kentucky awarding a contract for a minimal security facility to the Corrections Corporation of America. According to Dr. Thomas, this trend began because of an accelerated increase in prison populations across the country. Specifically, between 1961 and 1975 state and federal prison populations increased from 220,000 to 240,000 inmates, a modest 9 percent increase. Conversely, from 1976 to 1990, another 15 year period, the state and federal prison populations increased from 262,000 inmates to 739,000 inmates, a 181 percent increase. Because of this dramatic increase, the US, state, and local governments could not keep up with this expanding population. This environment allowed governments to seek alternatives to their slow procurement processes. Policy makers went to newly created private organizations that combined knowledge of the corrections industry and the financial backing. The trend of privatization has continued into the 1990's. Currently, 106,849 inmates are housed in facilities that are under contract with private firms.

Dr. Thomas then listed several reasons why this trend to privatization occurred:

Private firms have the ability to construct facilities faster than governments.
On average, privately constructed prisons are built for 15-25 percent less than an equivalent government facility.
The construction of prisons requires large capital outlays and private firms have several financial tools to acquire the funds.
Private firms are now large enough to create their infrastructure for construction. They are willing to take the risk and stay ahead of the demand.
Operational cost savings are between 10 and 20 percent, and these cost savings are often requirements in the contracts signed by these private firms.
The quality of service is as good or better in private facilities as in the prisons operated by government agencies.

Ben De Groot discussed his perspective on the privatization of correctional facilities and its effects in California. He stated that the prison population has increased at an alarming rate in California and the Legislature has not passed funding for a new correctional facility in the last five years. This caused the current prison system to operate at twice its intended capacity. To alleviate the surplus, California began contracting with private firms to build and operate low- and medium-level security housing facilities. In California, all contracted facilities have state correctional officers and parole officers assigned to them. This gives California assurance that the private firms comply with all applicable rules, laws, and contract requirements. In his opinion, private firms try to deliver a good product.

The panel then opened the discussion to questions.

Q: Why don't private firms operate higher security prisons?

A: In the past, private firms were restricted to lower level security positions because of concerns over their ability to handle some types of criminals. Recently, though, some states and the federal government have begun contracting out the higher security prisons.

Q: Are older prisons that are completely paid for cheaper than new prisons?

A: No, typically older prisons are labor intensive and require continual repair and maintenance. These costs are substantially lower in the newer prisons.

Q: What is a state's risk of litigation from inmates who are mistreated in private correctional facilities?

A: These private firms act as independent contractors to the state. Therefore, they shield the state from most types of litigation.

Back to the top

Collaboration, Coaching, and Mentoring

Moderator:

Joel Alter, Principal Program Evaluation Specialist, Minnesota Office of the Legislative Auditor

Presenters:

Ron Perry, Staff Coordinator, Washington Joint Legislative Audit and Review Committee

Jim Pelligrini, Deputy Legislative Auditor, Montana Office of the Legislative Auditor

This panel, which dealt with issues related to project staffing and staff management, consisted of two parts: discussions of how several legislative audit offices form and manage teams and of mentoring of team members. The three presenters responded to the following questions:

1. How often do staff work in teams to complete projects? How large is the team?

2. How are the teams selected? How is staff assigned to the team?

3. What is the role of the project manager on the team?

4. Does the office use a written workplan?

5. Is there staff specialization?

6. To what extent does the office use contractors and consultants?

7. How often are the meetings held?

The following summarizes the responses:

Washington Joint Legislative Audit and Review Committee

Washington JLARC has developed a team protocol and policies regarding team communication. The team protocol specifies roles of team members, methods to resolve disputes, and when and how the team should involve supervisors. Most teams consist of 2 to 4 auditors, selected primarily on the basis of availability. Every auditor is involved in at least one project, often more than two. A lead auditor functions as a project manager, a coach, and a mentor for other team members. The supervisor is less involved in the day-to-day operations of the audit. Generally, the lead develops the workplan. Team members usually meet once a week if there are issues requiring discussion. Each auditor submits a weekly progress report.

Montana Office of the Legislative Auditor

Montana has 11 performance auditors, in addition to financial auditors, information analysts, and data processors. A team generally consists of 2 to 4 auditors. Management decides on staffing assignments, although there is some specialization among staff, who have various academic backgrounds, including biology, statistics, economics, industrial engineering, and communications majors. Contractors were used only once, 18 years ago. Team meetings are informal and are not necessarily held on a regular basis. Progress reports are submitted twice monthly to supervisors and once a month to managers.

As a test case, Montana recently incorporated an agency's internal auditor and program staff as members of the audit team at the request of the agency, which also requested the audit. Before the audit started, the team discussed Yellow Book requirements and issues and procedures and roles of the team members. During the audit, the agency staff identified 14 issues and prepared related workpapers, which remained the property of the Office of the Legislative Auditor. The project worked well until the end, when the internal auditor and program staff recommended changes to the agency's operations. The agency's management rejected the recommendations and did not want numerous issues in the report. It has been two years since the audit started and it has not been completed. Problems with the project are that no one took control and that the agency management was not actively involved from the beginning.

Minnesota Office of the Legislative Auditor

Minnesota has 17 program evaluators. A typical team has 2 to 4 members, occasionally supplemented by contractors. Team members are assigned by management and do not have areas of specialization. The project manager functions as a researcher and liaison to upper management. The team submits a bi-weekly status report to management and meets once a month with management.

The presenters and audience members also discussed how to handle staff who are not team players. They emphasized the need to clarify expectations and discuss openly any problems that may arise. They also indicated that disagreement among team members does not necessarily mean a problem exists--thinking differently does not mean a person is not a team player. Often, however, auditors who are genuinely not comfortable working in a team leave the office. One audience member discussed an employee who had excellent skills, but who did not work traditional office hours. In this case, the office decided to allow her to work nontraditional hours, rather than lose her talents.

The presenters and audience members also discussed their offices' mentoring policies and practices. Washington tries to ensure that a new employee sees his or her first project from beginning to end because starting in the middle of a project can be difficult. The lead auditor on the job works closely with the new auditor. Montana and Minnesota have formal orientation procedures for new employees. Michigan has a formal mentoring program in which both new and experienced employees select mentors who are to provide guidance throughout their careers with the office. Several offices have "buddy" systems, the success of which depends on the chemistry between the new and more experienced employees.

Back to the top

Take Control of Audit Reports

Presenter:

Gaylan Nielson of Franklin Covey

According to Gaylon Nielson, to control audit reports we must manage the audit work and the audit report processes as well as develop document quality standards to manage messages. These actions will help reduce writing and revision time and produce useful reports.

To manage the audit and report writing processes, we should frontload the audit and use prototypes as management tools. Frequently, auditing and writing are treated as separate processes, yet the report is the only evidence of quality work. This results in a backend process sometimes characterized by extensive writing and rewriting. By reversing this process, the audit is geared towards the final report and writing time is reduced. Also, the time spent on issues which do not add value to the audit report is minimized.

Mr. Nielson suggested using the following techniques to frontload an audit:

Rebuild the planning process and discuss documentation requirements.
Develop an audit plan reflecting documentation needs.
Discuss and define document quality standards.
Provide writers with good report samples that can be used as models.
Discuss report users (readers, audience) in detail.
Discuss report length, amount of detail, contributors, data collection, research, etc.
Discuss style, organization, and layout.
Prototype the Executive Summary in the planning phase and use it to direct and plan the work and as a framework for the final report.
Discuss deadlines and incorporate milestones to avoid procrastination.
Develop a review plan to ensure opportunities to look at early drafts.
Promote opportunities for collaboration.

In addition to frontloading, the report should go through three prototype iterations.

Conceptual Prototype

Interim
Prototype

Mature
Prototype

Visualizes content categories and where they might appear on the page.

Helps identify how much information the report will contain and where gaps exist.

Looks like a rough draft.

May include:

-- headers and footers

-- sketches of graphics

-- indications of where major headings will appear

May include:

-- major headings

-- subheadings

-- captions

-- more detailed sketches of graphics

May include:

-- final versions of headings, subheadings, and key graphics

-- substantial text in some sections

Document Quality Standards

Document quality standards increase usability, accessibility, and readability. These are developed by determining what the user wants, creating standards that exceed user needs, and using these standards to write, review, and manage documents. Standards include:

Determine and state your purpose specifically and up front in the document.
State the required or requested action directly.
Use summaries.
Preview content in opening paragraphs, develop key points and details, and review major points in closing summaries.
Answer the question "So what?" by stating important ideas and benefits up front.
Design easy to read, attractive formats, create patterns, use white space, visuals, etc.
Use lists rather than paragraphs.
Keep language and logic simple.
Use the active voice- "I" and "we".

For more information about documentation quality standards, refer to the Franklin Covey Style Guide.

Back to the top

Foundations: Deliverers of Services or Devices for Circumvention?

Moderator:

Ann Campbell, Director of Investigations, Office of the California State Auditor

Presenters:

Ted Booth, General Counsel, Mississippi Joint Committee on Performance Evaluation and Expenditure Review

Andrew Young, Senior Auditor, South Carolina Legislative Audit Council

State agencies and officials offer many justifications for establishing relationships with foundations. For the purposes of this discussion, the panel defined foundations to include nonprofit organizations that raise funds to support state government functions and/or provide services to state entities for administrative fees. This panel discussed the validity of the justifications for establishing foundations and how they can be abused.

Justifications for the Use of Foundations

Panelists questioned whether foundations, as non-profit organizations used by state departments, even need to exist. Every benefit offered by foundations can be obtained without resorting to the use of foundations. Some state agencies admit that the primary reason for using foundations is to circumvent state rules, specifically regarding restrictions on the types of goods or services purchased and the process of purchasing. Since the foundations receive state money, the functions performed by the foundations should perhaps be subject to state restrictions to ensure that state resources are prudently spent. Alternatively, state law could be amended for valid reasons to give state agencies partial exemption from spending restrictions.

In order to circumvent the procurement procedures, some state agencies contract with foundations and specify the name of the subcontractor who will provide the services. For the administration of the contract, state agencies pay the foundations a fee of 15 percent or more of the amount awarded in the contract, even though the foundations have done nothing more than issue checks to the subcontractor.

Foundations may be exempt from state purchasing restrictions but they are not exempt from the Internal Revenue Code 501(c)(3) requirements. The private inurement rule under the IRC ensures that the organizations are not founded for the profit of the person running the foundations. Foundations are also restricted from lobbying activities, and conflict of interest laws apply if state dollars go through the foundation into the pocket of a public servant by way of cars or other amenities paid for by the foundation.

State agencies also argue that foundations receive more donations. Donors don't want their money mismanaged or diverted through a bureaucracy and are apprehensive about giving their money to state agencies. Because the donors don't understand how foundations work, they don't realize that their donations eventually end up at the state agency anyway.

Audits and investigations of university foundations reveal that some foundations are almost shell organizations. For example, foundations serving universities are housed in university property, and the employees who perform the work for the foundation are paid by the university. Generally a university vice president serves on the board of the foundation. In Mississippi it is standard procedure for foundations to supplement the salaries of athletic team coaches.

Some state auditors have continuing problems with access to records and documents. Agencies, including universities and foundations, attempt to limit auditor access to their records through legislation or other means because they want to be free of state oversight. However, most state statutes, if examined carefully, are generally broad enough to offer a means of access. For example, auditors have avenues of access through contracts awarding state funds or health benefits paid by the state for foundation employees. Moreover, some states are looking at the blurred lines between state agencies and foundations and determining that the foundations are, in fact, state agencies and must open their records to the public.

Access to records can be obtained in strange ways. After a newspaper in South Carolina ran a number of articles on the misuse of funds by a foundation, it was discovered that the press had obtained their information by digging through foundation garbage dumped in the city landfill. This information gave the state auditor's office access to the foundation's records and enabled it to perform an audit of the foundation.

Examples of the Misuse of Foundations

An example of the misuse of state funds involved the Student Aid Commission Foundation in California. The California Legislature approved the formation of the foundation for the purpose of hiring computer experts to fix and monitor the commission's computer system. The executive director of the Student Aid Commission was also paid by the foundation, and two thirds of the Commission's assets were transferred to the foundation. The foundation sent invoices for the computer services to the Commission which paid the invoices blindly. Eventually, the executive director was hired by the foundation and raised his own salary significantly in excess of state laws.

Another example of the misuse of a foundation was uncovered in an investigation of a manager in the California Department of Education. The investigation started during an unrelated audit of an economic program when the audit team became suspicious of payments from a community college to a student vocational club and its related foundation. The community college reimbursed thousands of dollars to the club and foundation without requiring adequate supporting documents indicating what the costs were for. One payment reimbursed the club for a $10,000 check to Disneyland. Further investigation revealed that a manager in the Department of Education administered the vocational program in California and monitored contracts awarding vocational state funds to the universities and community colleges. In addition, this manager was on the board of the club's foundation and cosigned checks for the foundation and the club. With his ability to influence the use of state funds awarded to the educational agencies, the manager was able to funnel over $95,500 into the foundation accounts, nearly half of which he spent on personal telephone bills, a trip to the Bahamas, purchases from Nordstrom's and other personal items. The manager also signed an unauthorized contract on behalf of the Department of Education and used employees in his division to fulfill the terms of the contract. The $25,000 check for the employees' services, made out to the Department of Education, was deposited into the foundation's bank account. Because the manager was in control of a foundation, he was able to divert state funds into his own pocket.

Investigators were able to examine bank accounts and documents revealing the manager's misuse of state funds because California has enacted statutes which give the state auditor's office access to records and property to the same extent that employees of the audited state agency or public entity have access. Because foundations typically have a university vice president, or other state employee, on their board of directors, the state auditor has access to foundation records.

While auditors can uncover fraud in audits involving foundations, the most common finding is a waste of state funds. Money obtained through university fund raisers may be used to wine and dine university employees. Foundation funds are used for perks for foundation employees. In one case, a university employee had two mistresses and used foundation money to take the women to lunch in alternate weeks.

To control the misuse of foundations, states can enact legislation which forces state agencies to apply for foundation grants in order to receive their services, enforce state audits of foundations which receive state funds, or eliminate the use of foundations by state agencies.

Back to the top

Electronic Commerce

Presenter:

Tom Carroll, Associate Partner, Andersen Consulting

Mr. Carroll believes that electronic commerce (eCommerce) is the most significant change in the way government operates in the last 30 years. Mainframe computing began the change in government operations, followed by the client/server computing technology which further changed the way organizations did their work. Because of these changes in technology, organizations had to re-organize to do their work, such as creating information technology shops within the organization. With eCommerce, which is another tool to help organizations work more efficiently and effectively, government agencies can now transact business electronically rather than in person or through paper.

In assessing whether eCommerce can help a governmental entity do a better job for its constituents, government administrators should ask themselves questions, such as the following:

Who are your stakeholders?
What does each value most?
What are some quick hit opportunities (Return on Investment)?
How can I transform my organization to get maximum value from eCommerce?

To get the most value from eCommerce, organizations need to publish, interact, transact, and transform using the technology. Publishing means telling your customers what they need to know. This can be accomplished through the use of a web page on the Internet. Interacting means telling your customers what they want to know by providing useful information on this web page and answering questions on-line through e-mail. Transacting means letting customers take care of transactions on-line through the same web page. Finally, transforming means working together, both within and outside of the organization, to get eCommerce to work.

For example, most governments now have on their own web sites. Some states have even established electronic purchasing for their agencies and some publish tax forms on-line. However, few have really taken full advantage of eCommerce to service their constituents. Government can use eCommerce in numerous ways. It can communicate rules and regulations on web sites. It can allow customers to request forms and ask questions on-line. It can allow customers to transact business on-line, such as filing tax forms and paying motor vehicle fees on-line.

Before beginning an eCommerce project, some basic keys to adding value must be understood, as follows:

1. You need to understand who are your customers and how many use the Internet. For example, students and universities use the Internet extensively. Also, many business people are now using the Internet to get information and transact their business. Projections indicate that, by the year 2000, eCommerce will be a $100 billion industry.

2. Next, you need to understand how business relates to government. Often, much of the same information goes back and forth between businesses and government agencies. For example, duplicate information is provided by businesses to multiple government agencies for permits and licenses, tax reports, and other information reports. With eCommerce, some of this information can be transmitted once for all purposes. To increase the chances for a successful eCommerce project, Mr. Carroll suggests considering some critical success factors. One factor is the ability to meet business requirements by providing easy access to the web site. Because not all businesses operate on a regular Monday through Friday, 8 a.m. to 5 p.m. schedule, access to the web site must be provided 24 hours a day, 7 day a week.

3. You need to obtain an understanding of what businesses want from government. For example, businesses basically want to be informed about the regulations that apply to them, they want less burdensome paper work, and they want to be in compliance with laws and regulations so that penalties are not assessed.

4. You need to understand how eCommerce could help government do a job better. For example, government could provide the compliance regulations on-line, it could consolidate registration and filing, and it could provide easy-to-use electronic forms and filing.

Where is all this headed and what are the trends ahead? According to Mr. Carroll, smart cities and governments will be taking advantage of eCommerce opportunities. There is a central emerging eCommerce technology such as net-centric computing and new hardware, factors that will make eCommerce more and more common. Some examples of emerging technology include brightware (purchasing software that will search hundreds of web sites on the Internet and locate the item you want at the best price) and rate technology (software that will automatically switch among service providers with the best rate at the time; i.e., long distance telephone companies.) Also, the next generation is growing up with internet technology in school and in the home. As a result, people will soon expect and demand virtualization in government.

Mr. Carroll concluded his presentation by providing a "Top 10 List" for making eCommerce successful:

10. Break down silos: Assign a single user identification code and reduce redundancy and duplication.

9. Build Transactional Trust: People won't use it if it is not trustworthy. Ways of establishing trust include providing good customer service, obtaining third-party endorsers, allowing personal interaction, providing education, and ensuring privacy.

8. The Personal Touch: Allow for alternatives that provide a personal touch for those people who fear depersonalization over the Internet.

7. Choice, Convenience, Control: Allow the user the ability to select what, when, and how.

6. Fix the Process Along With the Technology: Using a bad process with the Internet won't be worthwhile. Technology permits a better process.

5. Leverage Private Sector Solutions: Adopt private sector best practices. Most information is already kept in electronic form by users. Meet people in the middle rather than adopt a "my way or no way" philosophy. Intermediary firms, such as service bureaus and tax filing firms, will fill the market space.

4. Have a Change Management Plan: Be ready to adapt to a changing environment.

3. Understand and Get to Know the Customer: Government is in the business of serving its constituents.

2. Perform Cost/Benefit Analysis: Come up with an economic model that allows objective consideration of eCommerce. Understand the value as well as upfront costs to implement eCommerce.

1. Make It Easy: eCommerce must be fun to use, must save time, and can't achieve benefits without enough users.

Back to the top

Project Management How to Know When Trouble Lurks

Presenter:

Payson Hall, Consulting Systems Engineer, Catalysis Group, Inc.

A project is a unique and temporary undertaking to accomplish a defined goal (scope) within specified resource and schedule constraints. According to the Standish Group International, the Information Technology Project Success rate in 1994 was only 9%. Mr. Hall suggested that this statistic supports the fact that projects are not sure things and that project managers at all levels must recognize the risk of failure and avoid unrealistic expectations in order to be successful at managing projects.

"The Titanic Effect: The thought that disaster is unthinkable leads to unthinkable disaster." - G. Weinberg.

Mr. Hall identified the following 7 phrases that suggest projects may be in trouble, along with key factors you should recognize or consider at the beginning of each project to avoid these types of problems:

7. "This project is too important to fail..." - Project managers should have risk management plans and review them periodically. Make provisions for loss of key resources, delay of key tasks and failure of important tasks - before the problems arise! Consider what would happen if the project failed, then consider alternative approaches.

6. "We know the budget is unrealistic, but..." - Project schedules and budgets should be defensible. Always document assumptions made when planning the project, and monitor and explain all budget variances. Recognize that there will ALWAYS be variance.

5. "This will be a stretch, but with luck..." - Project plans should be credible and reviewable (others reading your plan should come to the same conclusions you do). Don't plan on 40 hour work weeks; there are only approximately 30 productive hours/work week. Task estimates should include effort estimates, and always check for over-allocated resources. (Projects often affect staff outside your immediate sphere of influence - recognize this in your preliminary planning.)

4. "Aaarg! The network is down again!" - Study the infrastructure of the organization(s) involved in the project and make sure that sufficient infrastructure exists to support the project. Make sure critical files are backed up, that you have sufficient administrative support and that you are not "saving the wrong dollar" (Don't save lower cost administrative salaries at the expense of higher cost professional salaries.)

3. "We can fix that during the next phase..." - Clearly define measurable project milestones and verify the completion of each milestone.

2. "Have they found a project manager yet?" - Every project team member should know "who is driving", and what the project manager's qualifications are. The project manager should have adequate organizational support and should have a back-up.

1. "Here's the spec, but it's not up to date..." - Unclear project definition is the #1 cause of failure. Uncontrolled change (in the budget, project scope, or specifications) is the #2 cause of failure.

Mr. Hall also spoke about the special challenges of public sector projects. Public sector barriers include the procurement process, politics and bureaucracy (including a lack of "profit" motive and communication barriers), staffing, and working with contractors.

Finally, Mr. Hall suggested several ways to improve the chances of project success, including education, recruiting and retaining good project managers, and encouraging open and honest communication. By defining the project early in the process, addressing risk realistically, and intervening early when problems arise with auditees, contractors, and others, you can reduce the chances of project failure.

Back to the top

Use of Focus Groups to Target Potential Issues

Moderator:

Dr. Timothy Plax, Director of Research, Ross-Campbell, Inc., Sacramento

Panelists:

Ed Kaempf, Founder and Director, NewPoint Group, Sacramento

Linda Triplitt, Evaluation Division Manager, Mississippi Joint Committee on Performance Evaluation and Expenditure Review

The panel discussed several audits for which they had conducted a focus group, facilitated group, or a version of one or the other. In discussing the alternatives, the panel found that the use of facilitated groups best met the needs of the audit industry. Specifically, in conducting facilitated groups, the panel stressed the following points:

1. Select a group facilitator with the proper qualifications

Remains neutral at all times.
Focuses on process, not content.
Positions the group as the expert.
Never does anything that the group can do for itself.

2. Properly plan facilitated sessions

Determine session opportunities and requirements.
Verify that the facilitated session's approach is appropriate.
Understand the organization, process, or work group.
Identify the sessions required.

3. Adequately prepare for facilitated sessions

Establish workshop session direction.
Identify and brief the facilitator.
Select and brief the user participants.
Develop the workshop approach and prepare the agenda.
Prepare workshop session materials.
Arrange the facilities, supplies, and equipment.

4. Professionally conduct facilitated sessions

Introduce the facilitated session.
Manage the agenda.
Ensure participant involvement.
Conclude the session.

5. Adequately analyze and document the facilitated sessions

Assess the workshop outputs.
Develop and distribute a facilitated session feedback package.
Verify facilitated session results.
Monitor open issues.

Back to the top

Healthcare: Evaluating Performance in Changing Markets

Presenters:

Dr. Scott Thomas, Senior Consultant; National Economic Research Associates; Los Angeles, California

Dr. Alyssa Lutz, Senior Consultant; National Economic Research Associates; Los Angeles, California

The presentation began with an overview of the types of performance measures commonly used in assessing healthcare providers and changing trends in the healthcare market, followed by examples of actual performance evaluations that Dr. Lutz and Dr. Thomas have participated in.

Performance evaluations of healthcare providers are used for many purposes, internal as well as external. External reasons include the need for information in Medicaid or similar negotiations, state antitrust investigations, mergers involving state assets, and certificate of public advantage audits that determine whether healthcare providers are following the terms and conditions of operation established in the certificates.

Commonly used performance measures are prices charged for individual transactions or overall prices, various types of costs to perform services, several measure of profitability, and quality of services. Individual transaction prices, such as the price of a chest x-ray, are relatively easy to compare to a standard or average price. Overall prices can be more complex and include all services offered and all payors, adjusted for case mixes and differences in local input costs, such as the severity of cases or type of facility providing the services. Types of costs used for evaluation and comparative purposes also may vary and include overall costs, costs per patient day, cost per case mix adjusted admission. Like the overall prices, these costs are adjusted for case mixes and for local conditions to ensure more accurate evaluations. Additional financial performance measures assess profitability by operating income, net income, and cash flow.

Performance evaluations using measures of the quality of services are the most difficult to complete. Mortality rates and surgery rates help measure the quality of services, but there are many other measures, including the appropriateness of services; technical excellence, or how well the services are performed; and the accessibility of services. Customer satisfaction, measured through surveys, is another commonly used measure of performance.

Mergers of hospitals, physician groups, and other healthcare providers and the growth of managed care are the most significant current trends in health care. Health maintenance organization costs have declined recently, but their profit margin is now so low, at 2 percent or less, as to make additional major reductions unlikely.

The presenters discussed three evaluations they have performed in the healthcare industry. One of these, an assessment of whether the Sacramento area market could absorb an additional provider of services to trauma victims, illustrated the need for using current, actual data in these kinds of evaluations. In this case, a previous outside consulting firm had determined that the market could not absorb an additional trauma center without a decrease in the quality of care. The consultants had concluded that the opening of the third center would reduce the visits to one of the original centers, in Roseville, to a point that it could not be profitable. However, this conclusion was based on clinical literature, rather than on what conditions actually existed in the state. When the presenters examined the range of visits to successful trauma centers throughout the state, they found that fewer visits were needed for the trauma center at Roseville to remain viable.

Back to the top

Poster Session: Showcase for Reports Issued and Techniques and Procedures Used by State Legislative Auditors

Participating States:

Arizona (Shan Hays)
California (Bill Shepherd)
Florida (Gena Wade)
Kentucky
Michigan (Gary Brown, Beth Larson)
Mississippi
Missouri (Jeanne Jarrett, Julie Miller)
Tennessee
Texas (Kyle Doerr)
Washington (Robert Krell)
West Virginia (Antonio Jones, Harvey Burke)

Arizona presented its training and career development programs. At the Office of the Auditor General, staff are encouraged to enhance their skills and abilities by attending a variety of in-house and off-site training classes. The office provides its staff with a class curriculum catalog that describes the different classes offered. The catalog provides a description of each class and indicates those staff levels that the class is recommended for. Classes include topics such as report writing, supervisory development, research methods, and project management. In addition, the office has developed a core curriculum for the various levels in its financial and performance audit divisions.

California presented a recent audit by the Office of the California State Auditor on the United States Border Patrol's policy to pay emergency care charges only for those unauthorized immigrants already in its custody at the time of treatment. The audit concluded that for 199 cases identified with Border Patrol involvement, health care providers in San Diego County incurred at least $2.9 million in unreimbursed charges. Because the Border Patrol denied the auditors access to sufficient detailed information, staff used other sources of data such as hospitals, ambulance companies, and the San Diego Emergency Medical System to identify incidents where emergency care services were provided that involved the Border Patrol.

In Florida, the Office of Program Policy Analysis and Government Accountability has developed the Florida Government Accountability Report (FGAR). The FGAR is an electronic encyclopedia that provides descriptions, analyses, and evaluations of major programs in Florida. Using the internet, one can search for current reports about policy issues and information about state programs. For each program, the FGAR provides profiles of the program's operations, amount of funding, and concerns and issues. The FGAR website is: www.oppaga.state.fl.us/government.

Kentucky featured several reports issued by the Auditor of Public Accounts. Recent reports include "Revenue Management at the Kentucky Horse Park", "The Commonwealth's Effort to Prepare for the Year 2000", and "Medicaid's Recipient Lock-In Program".

Michigan provided demonstrations of two software applications that staff of the Office of the Auditor General utilize during their audits and reviews. "WinIdea" is a software package that can be used to extract statistical samples from databases. The audit staff also use "SPSS", which is a statistical package that contains advanced features such as regression analysis. This computer based technology will continue to be relied on in future audits.

Mississippi's display summarized reporting objectives for legislative oversight. The Joint Committee on Performance Evaluation and Expenditure Review uses these objectives to provide its legislature with timely and accurate information on state agency programs and operations.

In Missouri, the Oversight Division of the Joint Committee on Legislative Research has the dual responsibility for preparing fiscal notes and performing program evaluations. Fiscal notes accompany bills before the Legislature and describe various aspects such as cost and fiscal impact, the impact on small businesses in the state, and whether any new state facilities would be required. During the rest of the year, the division focuses on program evaluations of state agencies and programs. The display highlighted many different performance and program reviews of topics such as health and social services and education.

Tennessee displayed reports produced by the Comptroller of the Treasury, Offices of Research and Education Accountability. One report that was highlighted was titled "The State of Corrections - Tennessee's Prisons Before and After Court Intervention". The report addresses whether the Tennessee Department of Corrections is at risk of additional federal intervention after recently being released from a federal district court order that mandated a federal receivership of the state's prison system. At the time of the review, the report concludes that a second federal receivership of Tennessee's prisons in the immediate future was unlikely, but that in spite of significant improvements, the state had not corrected all conditions cited in the original court order.

Several states, including Texas, are utilizing information technology to assist in their audits and program evaluations. Staff at the Texas State Auditor's Office provided a demonstration of an online survey used to obtain information from state and local governmental entities regarding internal controls. This format provides several advantages, including ease of completion and instant recording of responses. In addition, the survey is interactive and provides feedback on the elements of a solid internal control structure. Staff can use this information to focus their reviews on specific entities.

Washington's display provided an overall background of the Joint Legislative Audit and Review Committee. JLARC makes recommendations to the legislature and the executive branch, sponsors legislation, and reviews the status of implementation of its recommendations. Two of the audits highlighted were a privatization feasibility study of the Department of Corrections and a review of the State Investment Board. The display also included a picture of the JLARC offices, a converted apartment building that has been headquarters for the past 20 years.

West Virginia highlighted the Performance Evaluation and Research Division of the Legislative Auditor's Office. The division has conducted many performance evaluations of state agencies, boards, and commissions for the Joint Committee on Government Operations. The division also conducts research on special topics as requested by the Legislature or mandated by separate legislation. Some recent reports include reviews of the state's Child Protective Services program and the Workers' Compensation Appeals Board. As host of next year's NLPES conference in Charleston, the division also had a table with visitor information and assorted items such as pencils, pins, and the ever popular "Charleston Chew" candies (which promptly disappeared).

Back to the top

Juvenile Crime Prevention: What Works?

Moderator:	James Behunin, Audit Supervisor, Utah Office of the Legislative Auditor
Presenter:	Dr. Michael Schumacher, Chief Probation Officer, County of Orange, California

The session explored the trend toward prevention and early intervention in reducing juvenile crime. James Behunin has been working in Utah for over one year with the Juvenile Justice System. He indicated that this subject is drawing much interest around the country and he believes that auditors will be asked to focus more of their audit efforts towards juvenile programs over the ensuing years. For his review of Utah's Juvenile Justice System, he has used Michael Schumacher, Orange County's Chief Probation Officer, as an expert consultant and as an example of a practitioner "who is doing it right." His reports use recommendations as suggested by Dr. Schumacher and the juvenile program in Orange County.

The problem the nationwide increase in juvenile crime--currently, a "hot topic" for public officials. The response, in the past, has been to get "tough on crime" or to provide harsh sanctions such as "locking them up." The result is that many juveniles become more deeply involved in the system than is necessary. The system mixes juveniles who committed minor crimes ("at risk") with juveniles who committed felonies ("delinquent"). The system also misses the opportunity to intervene at an early age. The solution is to take a closer look at the problem--most offenses are committed by a few juveniles. Therefore, target the "at risk" youth for prevention and use graduated sanctions for the "delinquent" youth. The "prevention" to "graduated sanctions" approach includes the following:

At Home Programs: 1. Programs for all youth (night basketball games, etc.), 2. Programs for youth at greatest risk, 3. Intermediate intervention, 4. Intermediate sanctions

Outside Home Programs: 5. Community confinement (group home), 6. Secure facilities, and 7. After Care (After Care: A juvenile is on track in a group home and is released into his or her old environment. The juvenile goes back to the same delinquent behavior. After care is often needed to avoid recidivism.)

Most juvenile programs involve spending millions of dollars at the back end of juvenile programs, after crimes have occurred, rather than at the front end, for prevention, where fewer dollars are needed to address problems.

Dr. Schumacher explained that in the 1980s, the Orange County Probation Department (department) began an extensive study of juvenile offenders and developed the "8 Percent Problem Solution" through the following phases.

Phase 1: Problem definition (7 years)--profiled the 8 percent of juveniles most likely to have repeat offenses. These are juveniles who typically have three or more of the following profiles: poor school behavior or performance, chronic family problems, drug or alcohol use, and pre-delinquent factors such as a pattern of running away from home. (Orange County Probation Department publication: "The '8% Problem:' Chronic Juvenile Offender Recidivism")

Phase 2: Model development and field test (3 years--1993-1996)--used a control and test group for field test and found that the program reduced recidivism by approximately 50 percent (Orange County Probation Department publication: "8% Early Intervention Program--Program Design and Field Test Results")

Phase 3: Model Restructuring--1 year

Phase 4: California 7 county pilot--3 years through June 2000

Currently, the department is in the process of establishing Youth and Family Resources Centers. The department has opened three centers with future funding for five to seven centers. The key elements of these centers include: probation staff on-site; on-site school and tutoring; transportation provided daily to school and off-site activities; day treatment staff that coordinate recreation, community service, and life skill classes; drug and alcohol abuse counselors on-site; evening classes geared toward the whole family (parenting, anger management, ESL, etc.); intensive in-home family counseling (counselors spend time at juvenile's home); and mental health and health care services on-site.

The department is now requiring that youths released from juvenile hall, as part of After Care, participate in Youth and Family Resource Centers along with the 8 percent high-risk, first-time offenders.

Dr. Schumacher provided comparative costs of the incarceration and early intervention programs:

Juvenile Hall - $54,000/year/juvenile

Juvenile Camps - $41,000/year/juvenile

California Youth Authority - $34,000/year/juvenile

8 percent Early Intervention Program - $14,000/year/juvenile

Back to the top

Development and Use of Surveys and Questionnaires

Moderator

Jane Thesing, Senior Audit Manager, South Carolina Legislative Audit Council

Panelists:

Dr. Robert Oldendick, Associate Professor, Department of Government and International Studies and the Institute of Public Affairs, University of South Carolina

Joel Alter, Principal Program Evaluation Specialist, Minnesota Office of the Legislative Auditor

George McKee, Chief Analyst, Connecticut Legislative Program Review and Investigations Committee

The panel defined surveys, described the survey process, and provided practical advice on the preparation and analysis of surveys. Dr. Oldendick led the discussion, defining surveys as a way to gather information from a sample population. Surveys are valuable because they're fast - results can be developed in days or weeks. Also, with a survey you can afford to be thorough. With proper sampling you can contact a small group with very accurate results. Representativeness of the population from which the sample is selected is key to a good sample. Problems with the design and use of surveys include imprecision, extensive length, and misinterpretation or misrepresentation of data.

The purposes of a survey are:

to determine needs assessment
to evaluate
to determine level of client satisfaction
to describe population characteristics
to detect attitudes

There are seven steps in the survey process:

1. define your population of interest

2. choose mode of data collection

3. select a sample

4. design the questionnaire

5. collect data

6. code/process data

7. analyze the data

Dr. Oldendick discussed several of these steps listed in greater detail, as follows:

Modes of data collection: There are various ways to collect data. Face to face interviews were used predominantly until the 1960s. However, telephone and mail interviews are used more often now because they also increase coverage of the population and lower costs. Regardless of his or her mode of data collection, the surveyor needs to insure information is collected from a representative population.

Sampling: There are two broad categories of sampling: probability, or scientific, sampling, in which each element in the population has the same chance of being selected, and nonprobability sampling, in which the selection is subjective or arbitrary, such as one finds in street-corner interviews. There are also subcategories of probability sampling, as follows:

simple random sample (gold standard): every member in the population has a chance of being selected.
systematic sample: list population (i.e. 1 to xx) and take every Nth item after identifying a random start. This method is used frequently, it is not time consuming, and the results are generally representative.
stratified sample: if you know something about the population, it can be divided into groups. Independent random samples can be taken from the groups. This reduces the sampling error but you have to know something about the data in advance.
cluster sample: dividing the population to make samples manageable.

Questionnaire design: Dr. Oldendick had the following suggestions for questionnaire design:

1. Using both open- and closed-ended formats, design questions to answer the problem of interest: "What do we want this survey to do?"

Closed-ended question response formats are:

a. likert type - provide a range of potential survey responses such as "strongly agree" to "strongly disagree." Usually provide 5 to 7 selections.

b. forced choice - use when there are given policy decisions that are being considered.

c. rating scales - rate something along a dimension such as always/sometimes/never.

d. ranking scales - as in "favorite" to "least favorite;" limit number of items to 5 or less. Anything beyond 5 causes confusion. Also, it's hard to instruct people in this kind of response.

2. Rules for Questionnaire Construction.

Avoid loaded questions suggesting one alternative is preferred to another.
Avoid using slang and colloquialisms.
Avoid using technical terms and abbreviations.
Be precise and specific. For example, when asking questions with timeframes, consider saying 'last 12 months' vs. 'last year.')
Don't assume the respondent has knowledge of the topic.
Avoid asking two questions in one.
Avoid lengthy questions and keep wording simple.

Data collection and analysis: Before you collect data on a grand scale, pre-test the questionnaire and assess whether the responses are what you expected. Some standard approaches to analyzing survey data include:

frequency counts
cross tabulations
multivariate models
graphic presentations.

George McKee and Joel Alter described their experiences with using surveys and questionnaires to develop audit information and had the following suggestions for developing surveys:

Initially develop your survey questions as you would want to interview someone, then refine the questions.

Avoid asking factual questions if the information is available elsewhere.

When designing your survey, keep in mind the ease of data extraction.

Give the respondent a reason to complete and return the survey.

Consider coding population for follow-up contact if they do not respond to the initial survey.

To keep the total number of questions low and give the appearance of brevity, layer your survey questions. For example, ask a question with multiple parts (i.e. question 1 part a, part b, part c) as opposed to numbering each part of the question separately (i.e. 1., 2., 3., 4., etc.)

Keep in mind comparisons that can be built into survey data.

Back to the top

Views From the Other Side

Moderator:

Lois Benson, Audit Principal, Office of the California State Auditor

Panelists

Del Pierce, Undersecretary, California Business, Transportation and Housing Agency

Tom Roberson, Assistant Secretary, California Health and Welfare Agency

This panel provided an opportunity to find out how auditees view the work and practices of legislative auditors. Del Pierce and Tom Roberson both represent large agencies that have had frequent contact with the Office of the California State Auditor.

Previously an auditor and now in management, Tom Roberson shared some suggestions about critical elements of a successful audit, as follows:

1. Scope of audit: Auditor should develop scope of audit before going into the field. One result of undeveloped scope is that unnecessary work may be done. Audience participants pointed out that it takes time to develop the scope and not all information gathered results in findings; however, going into the field without a rigid scope may result in the uncovering of an important issue.

2. Staff skills and supervision: Staff assigned should have appropriate skills for the assignment. The audit should not be a training ground. Staff should be closely supervised and the work coordinated by the supervisor to avoid duplicate questions from different auditors. He noted that agency staff and managers fear auditors. Therefore, the auditor should be sensitive and non-confrontational.

3. Communication: Throughout the audit, management should be briefed on the progress and findings of the audit to reduce surprises at the end and potentially to reduce fieldwork. A good briefing of management also benefits the auditors in helping focus the report. Continued communication also helps build a trusting relationship between the auditor and management.

4. Report: In his view, audit reports should not be written for effect. The report should be truthful and not inflammatory. The tone of the report is very important and the language should be plain and direct.

Del Pierce added that the auditor should learn the department's perspective on the issues and that the department should also be helpful. An example of this was given by moderator, Lois Benson, who experienced an entrance where the department identified what it believed to be its problem areas and also good areas which were most important to meeting the department's mission.

Both panelists agreed that there has been an evolution in the purpose of the audits the Legislature requests. In the past the Legislature wanted to know what happened and the auditor told how the process could be done better. Now the Legislature wants something more like a consultant's report. For example, it wants to know the outcome of a particular policy or program and if its investment was worth the outcome.

Mr. Pierce stated that the auditor may be reluctant to share recommendations with the auditees, who would like the recommendations earlier so that changes could be implemented and problems fixed before the audit report is issued. Also, the auditor should consider how the department would implement the recommendations and if they are feasible by involving the operational side of the department.

Finally, Mr. Pierce acknowledged that audits of programs and processes help departments to go forward and do a better job.

Back to the top

An Approach to Evaluating Agency Staffing Levels

Moderator:

Julie Ferris, Lead Analyst, The Florida Office of Program Policy Analysis & Government Accountability (OPPAGA)

Panelists

Lois Sayrs, Ph.D., Senior Methodologist, Arizona Auditor General's Office (Arizona OAG)

Nancy Dufoe, Senior Policy Analyst, The Florida Office of Program Policy Analysis & Government Accountability (OPPAGA)

Julie Ferris and Nancy Dufoe from the Florida Office of Program Policy Analysis and Government Accountability presented their "three-tier" approach to evaluating agency staffing levels.

The Tier I analysis is a broad, state-to-state comparison using employee staffing levels obtained from the U.S. Census Bureau as the basis for comparison. This comparison should be used for limited purposes such as comparing employee data across states by government function and identifying areas where more in-depth analysis may be suggested.

The Tier II analysis addresses agency-to-agency comparisons and depends on statewide information about program performance, staffing, and cost. It can be used to establish staffing ratios and benchmarks for comparisons across programs and agencies and identifying potential areas of under-or over-staffing that should be studied with precise analysis.

The Tier III analysis uses more precise methods that are applied to targeted areas such as assessing the staffing needed to perform specific tasks at maximum efficiency levels.

Dr. Sayrs expanded on this approach by providing a case study designed to answer the question "Budget Reform on Staffing Levels, Is There a Trend?" The case study illustrated how the various approaches in the tiers could be used to answer the question. For example, Arizona found that Tier I would not be useful since the budget reform was specific to one state. However, Tier II was found to be useful in sampling state agencies to identify trends that warranted further analysis. Finally in Tier III, using the agencies identified in the Tier II analysis, Arizona was able to determine the probabilities of staff changes from one state agency to another, create a matrix of the staffing transition probabilities within the state agencies (i.e., increases, decreases, or remaining the same), and applying linear modeling to project the trend for the next four years.

In conclusion, the panelists provided the following key points to consider when evaluating staffing levels:

Since agency databases and job classifications are designed for purposes other than evaluating efficiency and effectiveness, this data should be used for limited analysis and supplemented with the development of your own data.
Since the U.S. Census Bureau is voluntarily reported, you will be unable to verify the accuracy of the information; however, it still provides a rough estimate.
The extent to which state agencies rely on local governments or contractors to provide services should be considered before reaching your conclusion.

Back to the top

Analysis of Educational Expenditures

Moderator:

David Summers, Senior Policy Analyst, The Florida Office of Program Policy Analysis & Government Accountability (OPPAGA)

Panelists

Robert McClelland, Senior Analyst, Hawaii Office of the State Auditor

Kim McDougal, Senior Policy Analyst, The Florida Office of Program Policy Analysis & Government Accountability (OPPAGA)

States are increasingly interested in knowing how school districts spend funds appropriated to K-12 education and whether districts are spending the money in the most efficient and effective manner. As a result, some states have put in place important mechanisms to provide information on how school districts spend their funds. This panel discussed two such mechanisms, in Hawaii and Florida.

Hawaii is unique in that it has the only state-administered, unified, public school system in the nation. The state spends about $1 billion each year to educate nearly 180,000 students. However, Hawaii has the same struggle as other states in determining how much it really costs to educate its students. The trend in Hawaii is towards decentralizing school management with a focus on individual districts and schools. Tough questions are being asked about costs and there are no clear answers to the question of how much money is actually getting spent at the classroom level.

Since 1994, the Hawaii Office of the State Auditor has performed several audits that have identified common themes. These themes are that current systems do not report useful or accurate educational expenditures and that the systems used to capture these expenditures are not useful and are geared towards answering compliance questions. The most recent audit has identified an easy-to-use software package that reports expenditures by location, function, and program. This information is useful for school level decision makers, lawmakers, administrators, and the general public. The school level information can be used to identify potential issues of concern or additional areas of analysis, to compare expenditures with those from other specific schools, and to track expenditures over time to identify spending patterns. The Hawaii State Auditor's report recommendations received a mixed reaction from the State Board of Education and the Department of Education, so it is uncertain whether the recommendations will be implemented.

Florida spent about $13 billion in fiscal year 1997-98 to educate its 2.2 million pre-kindergarten through 12th grade students. However, it too has the situation where the public lacks confidence that districts are using funds wisely, taxpayers are not in favor of raising taxes, parents are not satisfied with school and student performance, and legislators want to increase funds flowing to the classroom without raising appropriations.

The Florida Legislature created the Best Financial Management Practice Review for Florida School Districts to achieve these goals:

Improve school districts' management and use of resources
Increase public confidence and support for districts that demonstrate good stewardship of public resources
Encourage cost-savings

The program consists of a self-assessment prepared by the district and then a best practices review by the Office of Program Policy Analysis and Government Accountability. The documents used in the self-assessment and review were developed through an extensive literature review and discussions with education finance experts, professional organizations, education department staff in other states and Florida education stakeholders. The review framework consists of management or operational areas, goals, best practices, and indicators that are related to the best practices. The focus of the Best Practices is:

Using performance and cost-efficiency measures to evaluate programs
Using appropriate benchmarks to assess operations and performance
Identifying cost-savings
Linking financial planning and budgeting to district priorities, including student performance

While the results and report from the first district review are currently being finalized, Florida believes there are benefits from this program that can be useful to other states. These are:

What constitutes a best practice is transferable to other states' educational systems
It provides a blueprint for assessing a state's school district operations
The results can improve or identify efficient district practices, identify possible areas for cost savings, and target specific operational areas of concern to improve
It provides a useful tool for individual school districts to assess their operations independently

The documents that are available from the Florida Office of Program Policy Analysis and Government Accountability are:

Back to the top