NLPES Question of the Month

April 2001

What approaches has your office used to screen and hire the best candidates for employment as evaluators?

From: Rick Riggs, Kansas

The Kansas Legislative Division of Post Audit generally hires only at the entry level. Successful applicants must have a Masters degree or significant graduate-level coursework in a relevant field. We bring in the top candidates and put them through a 2-3 hour process consisting of an initial interview, a 1-hour writing exercise, and two in-depth interviews with teams of 3-4 senior staff.

The interviewers' impressions are quantified on an evaluation form, and the results tabulated. Then the interviewers meet and arrive at a consensus decision. If necessary, applicants may be brought back for a follow-up interview to address any interviewer concerns or lingering doubts.

We try, in the course of the interviews, to give the candidates a good sense of what the work involves. We've hired a couple of people who, within a relatively short time, decided that performance auditing wasn't their cup of tea. On the whole, however, we do pretty well at getting qualified, intelligent, critical thinkers with excellent writing skills.

From: Heather Moss, Washington

Despite the tight labor market here in the Pacific Northwest, JLARC has been successful in recruiting qualified, energetic staff. This is due to both the hiring process we use and the appeal of this office. Our hiring process includes many standard components, such as:

hosting informational discussions for graduate students,
requiring applicants to submit analytical writing samples, and
requesting and checking multiple references.

But the more innovative and insightful tools we use to screen candidates are our written and oral exercises. The exercises, which typically take place during the interview process, ask candidates to:

complete multiple question worksheets based on actual audit scenarios,
deliver impromptu "legislator briefings" based on a 2-page summary, and/or
lead short "team meetings" about scoping a study based on a legislative request. These exercises have allowed us to evaluate candidates in terms of how they respond to demanding situations and how they might perform specific work-related tasks.

We are also able to attract exceptional candidates due to the appeal of this office. Three of our last hires have come from other legislative committee staff offices, and one came directly from a graduate degree program. The aspects of this office that caught their attention include:

positive reputation of JLARC (with regard to both report content and impact),
informal protocols (we all have opportunities to meet and work directly with legislators and other high-ranking state officials),
extensive training opportunities (in and out of state),
flexible work schedules (alternate and/or reduced hours), and
competitive salaries (for the southern Puget Sound region).

Using these incentives and tools, we are able to attract good candidates and hire the best.

From: Glen Fowler, California

Our interviewing technique is the tool that has helped us effectively screen applicants. Using "Smart Hire" software, we developed behavior based questions which we use in our initial interview with candidates. The premise behind behavior based questions is that studies show that of those employees who are terminated, 80 percent are released for behavior reasons. Only 20 percent are terminated for being technically deficient or incompetent.

As such, the interview questions we use are custom built around the qualities we find that make an individual successful in our industry. After surveying our managers and executives, we identified these qualities to include the following:

Conscientious - refers to a person's degree of organizations, persistence, and commitment to work.
Undogmatic - involves openness to new experience from both the outer and inner world.
Emotionally stable - involves a person's degree of stress tolerance.

Without giving away our actual interview questions (in case one of you is thinking about applying for a position with us, as I strongly encourage you to do!!), each interview question asks a candidate to "provide us a specific example..." or "tell us about a time when..." The questions are designed to illicit an example that would demonstrate whether or not the candidate possesses the qualities/characteristics/traits that we value in an employee.

Only after passing this initial screening interview are the candidates invited to take our writing exam (a technical skill). If they pass our writing exam, we then conduct our second interview. This interview includes conceptual questions which again measure their technical competency.

We've been very pleased with the results of using behavior based interview questions in our initial screening interview.

From: Sylvia Hensley, California

At the California Bureau of State Audits, we use a fairly complex screening process to make sure we hire the best candidates. In addition to ensuring that candidates meet our education requirements, this process includes a series of 3 interviews and a written exam.

All candidates must complete an application certifying that they have met or will meet at graduation the following requirements:

Possess a JD; MPA; MPP; MSBA; MBA; MS Accountancy; BS, Business Administration--Accountancy; or BS Business Administration degree or A bachelors degree or equivalent with a minimum of 39 semester units and a variety of accounting, computer, communication, and quantitative courses. (Candidates to whom we extend offers of employment are required to provide transcripts.)

Candidates who certify that they meet the education requirements are scheduled for a Phase I interview. During the Phase I interview, which is generally conducted by a team of 2 auditors below the supervisory level, all candidates are asked the same 3 questions. The interviewers independently score the candidate’s response to each question. Candidates must score a minimum number of points from each interviewer to be asked to take the writing exam. (About 1 in 3 people interviewed are asked to take the writing exam.)

In the writing exam, we attempt to assess not only the candidates' writing skills but also their ability to think creatively and present a compelling argument. All candidates are given the same set of facts regarding information collected during a hypothetical audit and are asked to draft a 3-5 page essay that summarizes the auditor's key findings, draws conclusions about the information, and offers recommendations to correct the deficiencies the auditor noted. The candidates have 2 hours to write their essays. The essays are graded by an editor who assesses the candidates’ language use (spelling, grammar, tone, organization) and logic and analysis. Candidates who pass the writing exam are invited to the office for Phase II interviews. (About 1 in 2 candidates pass the written exam.)

During Phase II, we want the candidates to learn as much as they can about our work environment, and we want to ascertain whether the candidate will be a good fit for the office. Upon arrival, the candidate is given a 15-minute tour of the office. To facilitate easy conversation, these tours are generally conducted by our
lower-level staff. The candidate then meets with one of the deputy state auditors for about 45 minutes. The purpose of this interview is to make sure the candidates learn as much as possible about our work and our work environment. We encourage the candidates to do the questioning during this interview so we can also get a sense about their interviewing skills. Following this interview, the candidate is interviewed by a team composed of an audit supervisor and an audit principal. We call this our 'conceptual' interview because its purpose is to get a feel for how the candidates would approach various audit problems and how they would react in a variety of audit-related situations. Following the Phase II interviews, the deputy, principal, and supervisor decide whether to extend an offer to the candidate. (About 1 in 2 candidates who participate in the Phase II interview process receive job offers.)

Although the above is our actual screening and hiring process, it should be noted that we also exert considerable effort “marketing” the office by attending career fairs, speaking to campus organizations, etc.

From: Kate Wade, Wisconsin

In Wisconsin, we tend to hire at the entry level, and as a result our recruitments are timed to coincide with the academic calendar. With that in mind, here are some of the key steps in our screening and hiring process.

We strive to maintain contacts with the campuses from which we have hired recent graduates. For campuses in the Midwest, staff members visit the campuses in February and conduct informational meetings and interviews with students.
We keep the communication lines to the campus and students open after our visits by encouraging students to call or e-mail with follow-up questions as they develop their applications. It serves the Bureau well to ensure the students understand the skill set the Bureau is seeking and the nature of our work.
Applicants are asked to send a package of materials for screening: resume, transcripts, and a statement of interest. This package provides the management team with a well-rounded overview of each applicant.
Applicants with strong qualifications, as described in the application packages, are invited to visit our offices. The management team meets with each candidate to discuss their skills and interests; the candidate also completes a quantitative exercise. We also like to have newer staff meet with the candidates from their alma maters.
Hiring decisions are made by the management team. Prompt decisions have served us well in the competitive market of the past several years.

From: Ken Levine, Texas Sunset Advisory Commission

First, we try to write a non-bureaucratic notice that entices potential applicants that best meet our needs. Are you looking for creative thinkers? Then say so. If an opening leads to work on a team that will have extensive writing duties, then require significant writing experience (more than just school-related writing.) Re-think your notices and requirements regularly and target your needs.

Screening (beyond screening for minimum qualifications) is an art. We use people on staff that have extensive evaluation experience to do the screening. Each initial screening decision is reviewed. We also ask for a very short writing sample in addition to a resume and application to assist in the screening. Even though this allows an applicant to submit edited writing, you would be surprised at the varying quality of the submissions.

Interviewed applicants are given a short writing and analytical exercise based on a real world “problem” identified in our work. We basically ask how they would go about analyzing this problem. While most applicants handle the exercise well, some do not and others excel.

We use small panels to interview and focus questions on applicants’ experience and training. Questions relate directly to the functions of the job. We score applicants on a matrix of skills and characteristics important to successful performance of the job. Those with the highest scores are discussed by senior staff (including the director) and offers are made (after reference checks, of course).

Or we can just throw names in the hat......sometimes I wonder how that would work....

From: Wade Melton, Florida

In Florida, OPPAGA has developed a multifaceted approach to help us screen and hire the best candidates for analyst positions. Five components of our approach are discussed below:

Getting good applicants to apply: Our first challenge is finding good applicants. OPPAGA primarily uses internet listings including OPPAGA’s and NLPES’s website, word of mouth, university pipelines (liaisons with professors, internship programs, and visiting universities), and newspaper advertising.
Application review: Our next challenge is getting enough information on the application that would allow us to make sound decisions about whether to invite the applicant to our office for further consideration. In the past, the state employment application lacked enough information to enable us to know whether the applicant had ever really done the kind of work that we do. To remedy this, we created a supplemental application that asked the applicant to give us some detailed information on the kinds of research work and projects that they had previously done. We struggled to make the supplemental application detailed enough to be useful, yet short enough that it would not discourage good candidates from applying.
Informing the applicants about OPPAGA: In the past, we found that applicants often stumbled through the application/interview process because they did not really have a good idea what our office was all about. To remedy this, we now invite applicants who pass our initial screening to attend an on-site informational session in which we give them a detailed overview about our office. This overview is something like a sales presentation. We tell them why they should want to work for us, and we answer any questions that they might have about OPPAGA and what it is like to work here. Sometimes, this produces a self-selection process in which the applicant realizes that he/she doesn’t want to work for us. More often, the applicant comes away with a solid appreciation for our work and mission, and realizes the potential benefits of working here.
Having the applicants complete a simple 90-minute work exercise—a case example with five questions: Each question tests the applicant’s ability to perform one of the basic skills of the job. Performance on this exercise is a major factor in determining whether the applicant is invited for a panel interview. (We once tried a group exercise in which we watched six applicants work as a team to answer the case example questions. Although very interesting, we quickly discontinued this practice because we felt it was far too subjective to be useful.)
Our panel interview focuses on getting the applicant to talk about practical examples and experiences. We ask more of the “Describe for us a time when you had a conflict…” than the “If I had a conflict, I would…” type of question. Hearing examples of how the applicant has handled a variety of situations is much more informative than hearing theoretical platitudes. We also ask them to describe details of the research that they reported on their supplemental application. This provides us with better information to assess their interpersonal and technical skills.

Special thanks to Flora Caruthers and Byron Brown who have been instrumental in making OPPAGA’s recruitment and hiring practices successful and for their contributions in answering the Question of the Month.

From: Diane Herring, Texas State Auditor's Office

Prior to hiring, Human Resources begins with detailed job postings. These job postings outline our minimum requirements (i.e., degree, major, years and type of experience) as well as our preferred requirements (i.e., graduate degree, certification). Based upon these requirements, interested applicants submit resumes and/or State of Texas Employment Applications. Human Resources then screens these resumes and applications and recommends qualified individuals for an interview.

When an applicant comes to the State Auditor’s Office for an interview, they are first given an organizational overview and a realistic job preview. Following this, they are presented with a case study appropriate to the level of auditor for which they applied. The applicant must read a job-related scenario, prepare a written response, and give a 5-7 minute presentation to their interview team regarding the scenario. The point of the case study is to evaluate the technical content, the organization of information, writing mechanics, presentation and communication skills, level of analysis, and accuracy of information within both the writing sample and presentation. Following the presentation, the interview team conducts a structured interview of the applicant. The interview team consists of two experienced auditors that have been trained in interviewing technique. The interview is structured to provide an in-depth understanding of each applicant's technical knowledge, work history, communication skills, and interpersonal skills. Based upon the applicant's qualifications and interview score, the team will make a recommendation for hire or not-hire.

If the interview team recommends hiring the applicant, Human Resources will check references. The previous two supervisors are typically contacted for a performance reference. Additionally, Human Resources will verify dates of employment and ending salary. Assuming reference checks are positive, Human Resources will extend an offer of employment to the applicant.

From: Mitzi Ferguson, Arkansas

All employees of the Arkansas Division of Legislative Audit are hired as Field Auditors and must possess a bachelor’s degree in accounting with the required hours to take the Certified Public Accountant (CPA) examination. The Performance Section only has a few dedicated staff. These dedicated employees serve as management and supervisory staff for the Section. Field Auditors from our other Financial and Compliance Audit Sections (State Agencies, Education, Local Audits, and Information Systems) are selected for a project based upon their expertise in an area being reviewed. This gives the Performance Section the luxury of being able to choose the best personnel for each project from our 200 professional staff! It also keeps the field auditors happy by allowing them to become proficient in many aspects of the auditing profession. Any professional expertise not possessed by any of our staff may be contracted for.

From: Joel Alter, Minnesota

Our office’s recruiting efforts are pretty traditional—not many “innovative” bells and whistles. For instance, our job interviews have not included work-related exercises, like case studies or written “problem-solving” exercises. But we’ve generally had pretty good luck getting good applicants and hiring good staff, so I’ll mention of couple of practices that have worked well for us in staff recruitment.

First, we participate in a job fair at the University of Minnesota Humphrey Institute each spring. We have a booth with information about our office, and a couple of our office’s Humphrey Institute alumni are on hand to answer questions of students attending the job fair. It’s to our advantage to let graduate students talk with some “real life” evaluators who have received a public policy degree, and the fair is a good opportunity for us to collect resumes from people interested in temporary or permanent positions.

Second, most permanent staff hired by our office are interviewed by two separate teams of staff. No doubt, these teams ask candidates many of the same types of questions. But, by getting six to eight of our staff involved in interviews, we get more staff input into hiring decisions. (Also, there may be some advantage to letting prospective employees see more of our staff—so that they have a better idea of the types of people they would be working with.) Each interviewer rates the applicants on measures such as technical expertise, work experience, bearing, communication, motivation, and knowledge of policy analysis.

Finally, for years we have produced a “Guide to the Program Evaluation Division”—partly to help prospective employees better understand our office. It has information about how we get our topics, how we are organized, the backgrounds of our staff, etc.