NLPES Question of the Month
DO YOU EVER TAPE RECORD INTERVIEWS DURING YOUR PROGRAM EVALUATIONS/AUDITS--AND, IF SO, IN WHAT CIRCUMSTANCES?
Kathy Snider, South Carolina
We do not tape-record interviews. However, on limited occasions, when I knew we would be relying on the interview for primary documentation, we have asked the interviewee to review the write-up of the interview and sign a copy verifying that the information was correct. Also, we will often e-mail agency staff with questions or ask them to verify information obtained from other sources. That way we have it in writing.
Probably a good example of tape-recorded interviews would be the way attorneys take depositions and statements-don't they usually record them? Personally, I think tape-recording interviews is really a tool for criminal investigators and not auditors. I would imagine that it would make most people (on both sides of the table) very uncomfortable.
Rick Riggs, Kansas
We frequently reconsider the question of whether to tape interviews-almost every new hire brings it up sooner or later-but we always come to the same conclusion: no. The reasons are:
- The presence of a microphone would, we think, inhibit the interviewee.
- Taped comments still might be misinterpreted or incomplete, and the interviewee later could still say, "that's what I said, but it's not what I meant."
To minimize the chances that a subject later will disavow his interview comments, we use a signoff procedure. The auditor writes up her interview notes and adds language at the end something like, "The above, with corrections and additions as shown, is an accurate summary of my conversation with [auditor's name] of Legislative Post Audit." The summary goes to the interviewee for review. He can mark it up, add comments or documentation, delete things that are wrong (his mistake or ours), or that he doesn't like once he sees it in black and white. (If he's going to disavow something he said, better before the audit is issued than afterward.)
Interview summaries based on the auditor's notes can be relatively short, because they need only address the audit issues. The in-person interview may, depending on the interviewee and the auditor's interviewing skills, be a long, rambling affair with a high signal-to-noise ratio. The noise need not be transcribed.
Also, if the interviewee tends to talk around an issue rather than giving direct answers, an interview signoff allows the auditor to draw conclusions the interviewee may not have said in so many words. For example, the auditor may ask the question, "Have you instituted any new procedures since the incident to prevent similar mistakes in the future?" The interviewee might talk for 10 minutes about the agency's control procedures, but never mention any new ones. The signoff document's summary of the response might simply say, "No new control procedures have been instituted since the incident." The interviewee might disagree with that conclusion (maybe the auditor missed something, or maybe the interviewee thought of something later), and the signoff procedure allows him to correct the auditor's erroneous interpretation before it finds its way into the audit report.
Cortney Rowland and Chris Woolard, Ohio
For a recent study in Ohio, our evaluators tape-recorded numerous on-site interviews with a variety of stakeholders including school superintendents, treasurers, board members, and charter school officials. Towards the end of the data collection period, we concluded that the tape recorders may have been more of a hindrance than assistance and discontinued their use. Originally, every interview was taped to insure accurate and thorough notes. We planned to aggregate the data and had no intention of directly quoting individuals. However, two note-takers on laptops proved sufficient to create accurate notes without the aid of the audio recordings.
Our evaluators felt that the process of taping created an awkward opening. The equipment was set up and interviewees were twice asked for permission to record (once before taping and then again to provide an audible answer on tape). Additionally, they were reminded that the tapes would be destroyed after the notes were completed to alleviate any concerns about anonymity and confidentiality. Both of these factors contributed to the perceived awkwardness and the sessions felt much more formal than later interviews without the device. Subsequently, the researchers decided that recording was not necessary for accuracy and may have actually hampered informal conversation that would have allowed the subjects to be more forthcoming.
The protocol included one lead interviewer who was responsible for directing all questions and two note-takers (on laptops) that chimed in with follow-up responses, but were not directly responsible for questioning. They devised their own strategies of abbreviation and shorthand to insure/increase their capacity to listen, take extremely accurate notes, and follow the dialogue. Once back in the office, one person took primary responsibility for writing up the notes and then gave the document to the other note-taker in order to proofread, elaborate, and clarify any questions. A summary box was created to highlight four or five main points taken from the interview. The write-up was generally completed prior to the next interview to maintain the focus of the assignment.
Our office has never and will never electronically record primary interview subjects without their knowledge and consent.
Ken Levine, Texas
We do not and have never tape recorded interviews. Our feeling is that it can seriously inhibit the quality of the conversation. For most interviews, two staff members attend. One is the interview leader and the other is the primary note taker and will also ask follow up questions. We require the two participants to create a meeting summary that covers the main points of the interview and lists out any issues identified, follow up items needed, items requested etc. The summary is e-mailed to the team and management.
As a side note, we have prohibited an agency from taping an interview or other non-public meetings with our staff. Using some recorded question or comment out of context against us with their board/commission or with our legislative members would be very easy to do. And would immediately put us on the defensive. We simply think both sides are more free to ask and answer and discuss questions in an interview without fear of what the tape may sound like.
As you can tell from the above answers, we would not sanction secretly taping an interview under any circumstances. What a way to break down credibility with the agencies we have to work with. In general, it would go against our basic philosophy of being as open and honest as possible within the confines of an ongoing evaluation.
Paul Bernard, Georgia
We have never taped our interviews and I can't think of a situation in which the benefits would outweigh the disadvantages. Use of a tape recorder would seem to unnecessarily create an adversarial situation which is not conducive to a performance audit/program evaluation.
If we are concerned that someone may change their story at a latter date, we encourage one of two things: ensure that there are two analysts present at the interview or request that the agency provide us a written response. If there are serious concerns and the response is critical to our support for a finding, a written response is the best course.
To help ensure accuracy in your notes, it's best to be sure you ask clear questions and confirm the answer during the interview. After the interview, if there are any doubts about the accuracy of your notes, you can always go back to the agency to confirm what your notes say.
Mark Haldane, Arizona
I have not and would not tape record an interview. I believe people likely to be less candid and more reserved if they know they are being recorded. It makes many people uncomfortable to know their comments are being recorded and might be available to others after the interview.
Recording without the other party's approval is simply unethical and, in many states, illegal. If there are points in the interview that are unclear, ask the question again or ask for clarification. Make sure the interviewee knows how you understood complicated or controversial answers and give him/her an opportunity to clarify or retract. If your notes reflect this, it will be harder for the interviewee to successfully complain that you misunderstood or misconstrued the answer. In some cases you might want to give the interviewee a memo reiterating what was discussed at the interview. Obtaining existing documents that support the information obtained in the interview is always a good idea.
John Sylvia, West Virginia
We do not tape record interviews of auditee staff because I don't think auditees would feel comfortable with it and they would not response to questions openly. Agency heads and even lower-level staff would be concerned about innocent comments jeopardizing their jobs. I think it can be appropriate to use a tape recorder in certain circumstances such as complex issues or for issues that would have serious repercussions
if information was incorrect. I do not think it is ever appropriate to tape
record a conversation without the person's knowledge.
Some tips for conducting good interviews include having more than one auditor present, keeping the interview limited to a few topics, and recognizing when a response is important to your audit plan and ask the auditee to repeat his or her response for confirmation. Then of course confirm the interview in writing on a point-by-point basis, generally within two weeks. Confirming interviews in writing is a very important part of our auditing procedure.
One important advantage that West Virginia has in the way we confirm interview information is that the confirmation document has the interviewee's signature on the agency's letterhead. I know that not all states require their audit staff to have the auditee confirm conversations in writing, signed by the auditee on his or her letterhead. West Virginia, in nearly every case, follows the practice of summarizing the interview on a point-by-point basis, send it to the agency asking for it to respond whether it agrees with our summary or rewrite the summary in their own words to their satisfaction. We receive the agency's response in writing, signed on its letterhead. This process is a little time consuming, but it eliminates most confusion and it gives us the type of documentation that is most reliable.
Phil Durgin, Pennsylvania
We don't tape record meetings or interviews because it creates a legalistic/confrontational tone not only for that meeting, but for the entire audit. (If you've ever been the "tapee" in such a situation, you'll know what I mean.) If something comes up during the meeting that we believe we should document, we'll send the interviewee a letter summarizing the key points of the interview. If the interviewee sends a letter back that "corrects" his/her earlier statement, there isn't much we can do about it except to try to find other evidence.
Ethel Detch, Tennessee
We have occasionally tape recorded, but not often. We try to have two people in interviews to help with completeness and accuracy. We have also used a laptop sometimes. I don't think it's ever appropriate to record without the person's knowledge, although we refer potential criminal activity to another unit.
Sylvia Hensley, California
We do not tape interviews. Although taping provides a clear record of what is said, it may also result in reticence on the part of the person being interviewed. It is illegal in California to secretly tape a conversation without the permission of the participants.
Joel Alter, Minnesota
Taping interviews would add an element of tension to the interview setting. In many cases, interviews are more fruitful and frank when we can establish a rapport with the people we interview. I've never tape recorded an interview for a program evaluation, although our staff have occasionally discussed whether taped interviews might be useful in certain circumstances.
I occasionally think that a transcript of an important meeting would be useful-just as I appreciate having a tape of a legislative hearing so that I can hear the exact words someone used to characterize facts or express an opinion. For instance, when we interview top agency managers to get their "take" on an issue that has emerged in a study, it's sometimes important for our notes to record their comments verbatim, or close to it. But most program evaluators are not stenographers-and it's sometimes challenging to fully capture in our notes all the key information being conveyed in an interview. I think the best strategies are (1) having a second interviewer/note-taker at a meeting, and (2) restating the key comments of the interviewee during the interview, to assure ourselves that we did not misunderstand what was said.
Our office rarely, if ever, sends summaries of interview notes to auditees for confirmation. This would be time-consuming, and it would give the interviewee a chance to "sanitize" the interview notes-perhaps contrary to what the person actually said in the interview. However, we sometimes call the interviewee to confirm or more fully discuss some points made in the earlier discussion.