2002 NLPES-Fall Training Conference Notes
Plenary Session
Michael Quinn Patton
War Games - Thinking Strategically in Evaluation
Michael Quinn Patton began his talk by saying that legislative work
is among the best in the now global field of Evaluation. It sets a high
standard. Patton encouraged the audience to think critically about the
real mission of evaluation and to use strategic thinking to accomplish
it. He told several stories from military history to illustrate his points.
Strategy comes from the Greek word strategos- "to think like a general".
Generals build strategies through case studies, and case studies are considered
the preferred method for training senior people in any organization. Patton
used the Battle of Arbela, the ultimate case study at The War College,
to explain the importance of mission clarity.
In 331 BC, Alexander the Great led Macedonian troops in conquering the
(then) known world, which is now Persia/Iraq. (Patton noted that Alexander
is credited with the creation of "military intelligence," which, like anthropology,
requires learning about the culture and training of a group of individuals.
Alexander only fought two major battles; his other conflicts were settled
through negotiations, based on military intelligence.) Persians outnumbered
the Macedonians 5 to 1. Alexander told his troops that their mission was
not to defeat all Persians, but to drive King Darius from the battlefield.
To succeed against these overwhelming odds, Alexander devoted a good deal
of time to identifying the arrangement of the resources of his opponent;
he also demonstrated confidence in his troops, ensuring that the mission
was clearly described, and they had no distractions or diversions from
their goal.
The mission of evaluation (our King Darius) is to make sure that our
data and reports get used, and to teach others (particularly legislators)
to think evaluatively through legislative testimony, the press and our
reports. Our way of thinking is not normative, and we often encounter a
culture clash. Outsiders often want to learn more about the evaluative
process, but some of us mistakenly focus on creating a report. Reports
are a means to an end, not an end in and of themselves. We should follow
up on recommendations.
Patton interjected some (humorous) comments on the political nature
of our work, and how double-speak, prevalent in many parts of the public
sector, is not a part of evaluation, which delivers a reality check on
program performance. Patton identified these circumstances in which evaluation
is not political:
-
the program is unknown;
-
no one cares about the program;
-
there are no dollars in the program; and
-
no one making decisions about the program is sexually active.
On double-speak, Patton offered these examples:
-
wellness potential (a human characteristic; if it's at zero, the person
is dead)
-
pavement deficiencies (pot holes)
-
revenue enhancements (taxes)
-
acquiescent non-volunteers (persons subject to forced repatriation)
-
servicing targets (carrying out assassinations)
-
incomplete success (description of President Carter's failed hostage rescue)
-
pre-dawn vertical insertion (invasion of Grenada)
What gets measured, Patton advised, is what gets done. For instance, although
body counts were to be used solely for strategic purposes in the Viet Nam
conflict, body counts became the driving force for individual performance,
and led to tragic deaths of children, women, and elderly civilians. Once
the indicator was created, it was used. High stakes indicators such as
family reunification rates and reduction of hospitalization for the mentally
ill are indicators that are easy to manipulate, leading to dangerous situations
for vulnerable citizens. Evaluators are the safeguard against the corruption
of indicators. We must keep in mind the first Evaluation Standard - do
no harm.
General Robert E Lee was never taken by surprise because he knew the
importance of running multiple scenarios and forecasting problems and challenges.
We should do the same with our evaluation recommendations: what are the
possible consequences of our recommendations? Writing good recommendations
entails writing history, and then behaving as futurists. For example, if
we recommend increased efficiency for a welfare program, we have to be
mindful that, if followed, it could reduce the points of entry into the
program and consequently the number of people served. We should anticipate
change and consider contingency plans. Alice Rivlin recently theorized
that the U.S. financial system survived the terrorist attacks of September
11, 2001, because of the extensive preparation for Y2K, and the resulting
strengths of the interconnected data systems.
Patton closed by encouraging the group to seek knowledge, not just information.
He also left the group inspired with a group sing-along rendition of "The
Evaluators Hymn", the melody of which is borrowed from "Auld Lang Syne".
 2002
Fall Training Conference Notes
|
