NLPES 2001 Conference Notes: Research Methods

NLPES Site Map NLPES Conference Notes

Research Methods
Friday, September 7, 2001

Moderator: Joel Alter, Program Evaluation Coordinator, Office of Legislative Auditor, Minnesota

Joel Alter gave opening remarks about the use/misuse of statistics. He provided an interesting example of the misuse of statistics by quoting, "Every year since 1950 the number of American children gunned down has doubled." Using this example, by 1995 the figure of children affected by gun violence would have reached 35 trillion. We must be careful how we write and use caution in the way we think about the issues we have been asked to review.

Steve Harkreader, Florida Office of Program Policy Analysis

Florida recently experienced a $78 million shortfall in its appropriations for the Florida Medicaid program. The current shortfall is expected to be over $1.5 billion. As a result of watching the estimates for Medicaid spending fall considerably short of actual costs, a data driven policy analysis was conducted. Data driven policy analysis reviews a situation and provides options to correct the situation.

During the first phase of a policy analysis, the extent of the problem must be documented. Historical figures of the program are reviewed. Florida looked at the Medicaid program by service category to determine what was driving the deficits. They determined that the prescription drug program went from the fourth most expensive cost category to the second most expensive category.

Drug expenditures were looked at for 5 years. The increase in expenditures were reviewed and a determination was made of the factors responsible for the increases. Once this information was obtained, it allowed for the customization of recommendations and policy options addressing the particular causal factors affecting the increases in prescription drugs.

Research showed the cost components of prescriptions to be price and volume. Higher numbers of prescriptions and higher prices for those prescriptions were the primary factors resulting in the increase in pharmaceutical expenditures. However, the major factor was determined to be cost of the prescriptions. Policy options included changing eligibility requirements, limiting the number of prescriptions, price controls, and implementation of drug formularies. The Florida Office of Program Policy Analysis recommended implementing a drug formulary. Drug formulary legislation was subsequently approved by the Florida legislature.

Steve Hendrickson, California Bureau of State Audits

Mr. Hendrickson discussed the research methodology of the California State Auditor’s Office on an audit of California’s deregulated power companies, deregulation background information, and how deregulation led to California’s power companies buying power at "retail rates" but providing the power to consumers at "wholesale rates".

The scope of the audit focused on the design of the law implementing deregulation, examined the operation and interaction of the Independent System Operator and the California Power Exchange, and assessed the success of the state’s market surveillance at detecting and remedying problems in the electricity market.

Audit results indicated that the energy crisis in California resulted from poor design and implementation of the deregulation laws, the lack of use of long-term contracts, underscheduling needs by both buyers and sellers of power, and decision makers not responding to the findings of market surveillance groups. There were also the contributing factors of hot weather, a low water year in the Pacific Northwest, and increased natural gas prices. The State Auditor’s Office hired consultants who had expertise in the economics of the energy market and knowledge of regulated and deregulated energy markets to help them.

Mr. Hendrickson stressed the importance of selecting consultants carefully, making them disclose any potential conflicts of interest early in the review, and explaining to consultants the necessity of documenting their work in working papers. Audit teams need to build in extra time to review and edit consultants’ reports and handle difficulties associated with obtaining access to records and people.

Dr. Darrel Hobbs, University of Missouri - Columbia

In 1980, Dr. Hobbs involvement in a national research panel resulted in his involvement in the creation of the Office of Social and Economic Data Analysis (OSEDA) at the University of Missouri. The primary focus of OSEDA is to bring up-to-date information to the people of Missouri.

Dr. Hobbs quoted Ray Marshall, Economist and former Secretary of Labor during the Carter administration as saying, "Any organization tends to get back what it measures and rewards." If you want to bring about change in an organization, change what you are measuring and rewarding. Dr. Hobbs believes what we are measuring is becoming less important and what is becoming more important isn’t being measured. America tends to be stuck in the industrial era of the 19^th century in measuring performance. It is now necessary for us to determine the "whys"of something. For example, we aren’t measuring what affects student school performance. Unless facts are meaningful for the people using them, the facts are not useful. New information should be surprising.

Dr. Hobbs stated that Missouri has 115 counties, 942 municipalities, 540 school districts, and some 3,000 special districts that make up local government. Factors affecting the need or demand for local data include:

Devolution
Diversity
Rate and unpredictability
Change in traditional sectors, i.e. agriculture
Pressures on the environment, and
Change in the infrastructure to produce and transport value and information.