NLPES 2001 Conference Notes: Evaluating Technology Projects

NLPES Conference Notes

Evaluating Technology Projects
Friday, September 7, 2001 1:30-2:45 PM

Moderator: Heather Moritz, Office of the State Auditor, Colorado

Ms. Moritz noted that the June-July 2001 NLPES survey asked “what are some good, timely topics that other states’ legislative program evaluation agencies may wish to consider?” 13 of the 25 respondents mentioned technology-related topics. She listed a handful of those survey comments, including the impact of technology on state government operations (Texas) and management and the use of Geographic Information Systems (GIS) (Montana). She concluded that this topic of evaluating technology projects is very timely.

Speaker: Melissa Schuiling, Office of the Auditor General, Michigan

Ms. Schuiling described her office as having 10 full time technology auditors on staff, 3 IT audits going on at all time, and 6-8 reports published each year.

She stated that the reason why evaluating project management and system development, is that typical time overruns, cost overruns, and few features actually delivered adds up to a lot of money. When auditing, you want to look at the controls or situations that will make the project unsuccessful in meeting the deadlines, staying within budget, complying with anticipated qulaity and meeting user needs.

She described 5 major technology projects evaluated in Michigan, 2 of which were successful and 3 of which were not.

The federally required computer system to track child support. The first contractor went bankrupt, so the state contracted with a second. At the time of the auidt, the state had spent $103 million and had implemented the project in 59 of 83 counties, which made up only 20% of the case load. Two of the biggest counties wouldn’t be on line anytime soon, because one had such a big case load that the project couldn’t handle it, and the other had already developed its own system and resisted the state’s. In the end, many child support functions had to be done manually.
A welfare eligibility and case management system. The system was estimated to cost $60 million, but $114 million was spent. At that time only 2 of 5 phases has been completed to provide only 10% functionality. Part of the problem was that the welfare regulations kept changing which changed the requirements of the system. In the end, 3 separate systems were needed to register a case, open a case, and determine eligibility.
An inventory of the state’s road and bridge infrastructure. The system was intended to inventory, budget and analyze data. The end product was just a data base that wasn’t useful. The system did not provide the information that users needed.
The Department of Natural Resources’ hunting and fishing licenses. This was a success. The system allowed purchasing licenses all over the state at retail locations. It improved accountability, increased revenue, and achieved objectives. The Department figured out what they wanted for the system and then hired someone to design it to meet their needs.
The Department of Natural Resources’s 1-800 system to make reservations at any campground. The reservations are made centrally and reservations a downloaded to all the campgrounds. The only problem was due to the high number of transactions, the Dept. didn’t trust it to work right and was also faxing the reservation information.

Weaknesses are generally found in project management, contract management and system development. Problems in project management include not planning or assessing feasibility, proceeding without executive-level support and lacking a strong project manager. Problems in contract management include poorly written contracts, unclear project deliverables, and failing to competitively bid contracts. Problems in system development included not understanding the needs of the users, not involving users at each phase of development, and not following a phased development approach.

Speaker: Kellie Monroe, Legislative Audit Bureau, Wisconsin

Ms. Monroe explained that the Wisconsin Legislature started to get concerned about IT projects because they heard that some work done by expensive consultants could be done by state employees, that consultants were being used for long term routine work, and that IT expenditures continued to grow as a percent of agency budgets. Her office’s audit looked at the performance of large IT projects, the use of consultants, and the best practices for managing IT projects.

They first quantified the issue. They found IT expenditures made up 37% of all state purchase orders in FY 1998-99, which was a 139.2% growth over 5 years. They found the state employed 1,383 IT FTE, not including the University of Wisconsin, amounting to Nearly $88 million.

Use of Consultants: State statutes provided guidance in when consultants should be used. They found that they were the statutes weren’t always followed, and usually the consultants cost more than hiring a state employee, in some cases significantly more. They also documented instances where former state employees returned as consultants at a great expense to the state. The audit recommended a reassessment of purchasing IT services from consultant and that agencies request more positions if they need to.

Performance of IT projects: The audit looked at whether 7 IT projects were completed on time and within budget, and whether it had the expected functions. Only one was deemed sucessful. Reasons for the success included: hiring an independent consulting firm to evluation bids and vendors, carefully developed project specifications tied to cost limits, hiring a project manager with experience, direct access to agency head to make critical decisions, and access to staff to test development.

Best practices: The audit developed a number of best practices regarding contract type, communicating desired functions, contract length, contract language, and project management. A copy of the audit “State Agency Use of Computer Consultants” from March 2001 is on the Wisconsin Legislative Audit Bureau’s webpage. You can also contact the Legislative Audit Bureau at 22 E. Mifflin Street, Suite 500, Madison, Wisconsin 53703, (608) 266-2818.

Ms. Monroe noted that the new budget created a Chief Information Officer, who is the head of the new Department of E-Government. The Department will approve new projects, develop standards, and assist smaller agencies with their IT needs.