Traits of an Outstanding Program Evaluator

September 2000

Beyond the skills acquired through formal education, what are the most important traits of an outstanding program evaluator (or Performance Auditor)?

 
 

From: Jane Thesing, South Carolina

Personality traits of a good auditor:

  • Curious
  • Analytical
  • Detail-oriented
  • Unflappable
  • Thick-skinned
  • Questioning
  • Judicious
  • Observant
  • Persistent
  • Practical
  • Considerate

From:

Ted Booth, Mississippi

Intellectual breadth and curiosity are critical to being a successful program evaluator. Often we must approach subject matter foreign to our formal training and experience. Generally, I have found that persons who are curious and value intellectual breadth have fewer problems working in new and different subject areas and are inclined to welcome the opportunity to try something different.

From:

Gary VanLandingham, Florida

A great program evaluator has, in addition to strong analytical skills, an innate curiosity about the world that leads him/her to always ask "why", strong intuitive skills that leads them to always try to organize information and understand what it means, strong writing ability (a talent as well as a skill), a sense of humor, common sense (a fairly uncommon attribute in the world), pride in always doing the best possible job (but not be a perfectionist that can waste a huge amount of time in the endless pursuit of the perfect sentence), and a willingness to put in the effort to get the job done.

In addition to being able to leap tall buildings with a single bound, we look for several things when recruiting staff. Four things stand out:

First, we look for people with a research orientation evidenced by taking a resource-oriented track in college (e.g., many research methods and policy analysis courses) and/or by having previous jobs in program evaluation and policy research. We aren't concerned with the specific academic degree that people get (it isn't important whether people have a public administration or business degree) so long as they are researchers. Virtually all of our hires have at least a master's degree.

Second, we look for people who are smart, intellectually curious, and problem solvers. This is harder to judge, but we consider college grades, standardized test scores, academic honors, the interview results, references, and a work exercise we administer as part of the process. This exercise is a case study asking people to look at background information on a fictional state program and asking them to identify potential issues, research methodologies, and potential recommendations.

Third, we look for people who are self-starters and hard working. This again is hard to judge, but we consider college success, prior jobs, references, the work exercise, and interview results.

Fourth, we look for people who can successfully work as part of a research team. Prior team experience in college and jobs, references, and interview results give us some insight into this.

Hiring smart, hard working researchers who are committed to doing a good job and who can work well with others (both within OPPAGA and the outside world) makes this job a lot easier. We don't try to hire people based on their writing skills. We have never found a good way to determine whether someone is a good writer during the selection process (prior work products are not a good evidence source because most are team projects that have been edited by others). We will note if someone seems to be a bad writer based on their work exercise and may decide not to hire them based on this result. However, we concentrate on training and developing writing skills once good people are on board.

From:

An ability to see more than one right way to do things.

From:

Creativity, specifically:

From:

Leslie Marks, Utah

One quality that I believe is very helpful is the ability to communicate well. We get so much of our information through personal contact with others: auditees, colleagues, consultants, etc. It is essential to be able to ask clearly for the needed information and also to pay attention to both the verbal and nonverbal messages received. The ability to communicate to our bosses, the legislators, is also very important, so formal communication skills via presentations or in a committee setting is also important.

The need for clear communication extends to the written realm. Our reports are our product, the proof of the work we've done. We need to be able to communicate our findings clearly and convincingly, in such a way to encourage implementation of the recommendations.

From:

Jeanne Jarrett, Missouri

The traits I appreciate in a program evaluator (beyond formal education) are good professional skepticism, enthusiasm for the work, a desire to learn more about state programs, good oral and written communication skills, and the ability to function well as a member of a team.

From:

John Turcotte, Florida

Outstanding program evaluators, either through previous experience or intensive study, have sufficient expertise to develop recommendations that solve problems. Armed with strong methodology skills, they are diligent and disciplined in their approach to assignments. When pursuing evidence and forming conclusions, they do so without creating undue hardship for research subjects or fellow evaluators. They communicate their work creatively through reports and presentations that are designed to inform and convince busy legislative readers.

From:

Lisa Kieffer, Georgia

We've recently put together the attached list of traits/attributes of an ideal auditor as one step in helping us identify potential candidates in our recruiting process:

  • Inquisitive: Ability to ask pertinent and probing questions regarding the position, duties, environment, etc.
  • Strong Communicator (Oral-Interpersonal and Written): Should understand the importance of effective listening, being able to articulate thoroughly through oral and written means
  • Professional: Appearance and demeanor
  • Autonomous: Ability to self-govern; work independently
  • Ethical
  • Mature
  • Analytical
  • Confident and Self-assured
  • Able to present and defend argument: Strong presentation skills
  • Assertive: Ability to confront when necessary (internally and externally)
  • An Initiator: Self-starter
  • Able to handle "highs" (inundated with deadlines and the like) and "lows" (revision period, waiting for response from agency); ideally, candidate should be able to remain level
  • Detail-oriented: Ability to understand how details affect end result
  • Able to see the "big picture" of projects and beyond the detail of individual audit tasks
  • Receptive to instruction
  • Able to respond productively to constructive criticism
  • Tenacious: P erseverance/Doggedness
  • Able to be a professional skeptic without becoming a cynic
  • Ambitious/Goal-Oriented: What results are they working towards achieving in their position?
  • Patient: Willingness to endure
  • Committed: Ability to follow goal through to the end

From:

Kate Wade, Wisconsin

Traits of an outstanding program evaluator:

  • detail-oriented
  • strives for accuracy in his/her work
  • thorough and persistent, following through on issues that seem to develop slowly or have some dead-ends
  • inquisitive, curious about how and why things work/operate the way they do
  • questioning and skeptical attitude
  • juggles multiple tasks/issues at once
  • organized mentally (but his/her desk could be cluttered!)
  • strong planning skills to guide analyses and to decide on tasks and priorities
  • quick learner of complex issues
  • quick on their feet in interviews and meetings
  • intuitive sense of which issues are important to explore
  • awareness of political reality and needs of legislators
  • sees the "big picture"
  • works well with people in a team setting
  • relates well with agency/program staff and gains their confidence
  • works well under pressure and/or with tight time deadlines
  • analytical approach to issues
  • creative (in considering analytical approaches and solutions to problems)
  • flexibility to adapt to changing situations
  • able to develop hypotheses, challenge hypotheses, and then let-go if the hypotheses are not supported by findings
  • clear, concise, persuasive writer

 

 

Ken Levine, Texas
  • A highly analytical mind
  • Diligence to dig, to look hard for problems and fully investigate potential solutions
  • A strong work ethic
John Norris
  • the ability to first visualize the elements of the report and then let the report guide the evaluation.
  • the ability to synthesize data into coherent report presentation
  • the ability to proceed when a straightforward solution is not available
Rachel Cohen, Texas
  • An ability to communicate easily--to figure out whether you're talking to the right people, to get people to understand why you're intruding on their terrain, to explain why what they're doing may not be working and why, and to explain to legislative staff what needs to be done.