News and blurbs
Karl Spock (Texas)
Hello from Austin!
It’s hard to believe that my one-year term as chair is almost half over. Pondering that, I also realize that most NLPES events happen in the last half of the chair’s term, which ends in September. I’d like to spend a few moments bringing these events to your attention.
The last event of the NLPES year, our professional development seminar (PDS for short), is the crown jewel for our staff section, offering training and networking that consistently gets high marks for relevance. I’m delighted that Texas will host the PDS this year in my home town of Austin on Sept. 23–25. Host offices in Texas include the Sunset Commission (where I work), the Legislative Budget Board, and the State Auditor’s Office.
We are coordinating the PDS with another NCSL staff section, the National Legislative Services and Security Association (NLSSA), as we have successfully done several times in the past. Although training is primarily separate, the coordinated meeting helps with finances and broadens networking opportunities (and we also will feel very safe!).
We are lucky to have many resources for the PDS available to us through a strong NCSL connection in Texas. Patsy Spaw, NCSL’s current staff chair, is secretary of the Texas Senate. Rick DeLeon, our Senate sergeant-at-arms, currently chairs the NLSSA Executive Committee. We hope that many of you can come and experience a seminar that offers top-notch training. As important as the professional development component of the PDS is, I would be remiss if I didn’t also mention opportunities in the works to enjoy live music and barbeque after each day’s training ends.
The NCSL Legislative Summit, NCSL’s annual conference, offers another opportunity for training. This year, the Summit will be held in Atlanta, on Aug. 12–15. NLPES will be looking for opportunities to sponsor or co-sponsor training events targeted to our interests and needs at the Summit. Stay tuned for more on that topic later in the year.
I’d also like to call your attention to our NLPES awards, another major effort of our organization, with an application deadline of Friday, May 10, 2013. Executive Committee member Wayne Kidd will be spearheading the awards effort this year and has more to say on the topic later in this newsletter. I encourage you to show off your work and submit an application. You can submit based on a particular project through the Research Methods or Certificate of Impact categories, or you can focus on your office’s work as a whole through the Excellence in Evaluation category. Awards will be formally presented at the PDS in Austin in September.
By the time this newsletter is published, the NLPES Executive Committee will have completed its April 6 meeting in Austin, one of three meetings we hold throughout the year. At that meeting, we will have worked through details of items mentioned above, as well as other topics, such as possible future training webinars. In April, we will also be filling the four Executive Committee spots up for election, as explained in another part of this newsletter.
That covers much of what happens from now through the end of my term in September 2013. I hope to see you in Atlanta or Austin!
Angus Maciver (Montana)
With the spring thaw upon us, Report Radar has been buried in an avalanche of exciting new audit and evaluation reports from around the country. We have a lot of ground to cover in the areas of information technology, broadband internet access, retirement and investments, education, healthcare, state employees, and land use (revisited). Ready? Let’s go!
Information technology: No less than five reports addressing issues relating to information technology (IT) were issued recently. In October 2012, California released a report reviewing reliability of data in state systems used for audit purposes. For another multi-agency review of IT issues, see Kansas’ December 2012 report addressing IT security controls and protection of confidential information. Continuing with the IT security theme, Colorado’s Statewide Internet Portal Authority has a November 2012 report that includes a discussion of data security. Meanwhile, in February 2013 the Texas Sunset Commission released a report on the state’s Department of Information Resources, which also included review of IT procurement issues and the Texas Procurement and Support Services Division. Finally, the Michigan Office of the Auditor General released a report in November of 2012, which addressed the state’s IT backup and recovery services.
Broadband access: We also found three reports addressing state efforts to promote or expand broadband Internet access. In December 2012, both the Arkansas Legislative Audit Division and the Wisconsin Legislative Audit Bureau released reports on the efforts of nonprofit groups involved in expanding broadband Internet access. February 2013 saw West Virginia’s Performance Evaluation and Research Division release its report on the state’s use of federal funds under the Broadband Technology Opportunity Program. Such efforts are ongoing in many states, and these reports provide valuable insights from an audit and evaluation perspective.
Retirement and investments: State retirement systems continue to be front and center for a lot of legislatures. Florida performs a biennial evaluation of its State Board of Administration, which invests the state’s retirement fund assets. These reports are a good source for anyone beginning an audit or evaluation of retirement issues. You should also pick up Mississippi‘s and Virginia‘s December 2012 reports looking at their state retirement systems. Like Florida, Virginia’s report is also part of a series of regular, statutorily-mandated reviews and provides a good introduction to retirement systems oversight. Also, take a look at another Texas Sunset Commission report addressing the State Pension Review Board, which provides oversight of state and local retirement systems in Texas.
Education: Both New Mexico and Tennessee recently issued dual reports related to teaching. New Mexico published a November 2012 report addressing use of student test data to assess and improve teacher evaluations; and New Mexico followed up with a December 2012 report on teacher and school administrator preparation programs. Professional development for teachers was also the focus of a November 2012 report from the Tennessee Office of Research and Education Accountability; and Tennessee also released a January 2013 report discussing trends in teacher compensation, with a focus on alternative salary schedules.
Healthcare: Two states recently released reports addressing state trauma care networks. In December 2012 Georgia published a report on its Trauma Care Network Commission, followed soon after by a report by Mississippi on its Trauma Care Systems Fund. Both are valuable reads for anyone looking at audits or evaluations of similar efforts.
State employees: State employees and their pay and benefits continue to be a subject of intense interest. Idaho has a good report from January 2013 that took a broad-based look at state employee compensation and turnover. The Nevada Legislative Auditor released a report in December 2012 discussing the state’s employee benefits program, with a focus on healthcare cost information. And the New Hampshire Office of the Legislative Budget Assistant published a very interesting performance audit in March 2013 regarding how the state determines whether to contract or use state employees to provide services.
Land use issues: Finally, we revisit land use issues, some of which may be familiar from your in-depth reading of previous Report Radar content. Connecticut released a report in December 2012 that uses results-based analysis to review the state’s farmland preservation programs. In February 2013, Minnesota took an equally interesting look at conservation easements and made recommendations regarding strengthening oversight of these land conservation efforts. Lastly, North Carolina has an interesting report from January 2013 on state tracking of submerged lands under navigable rivers. Report Radar is sure the fieldwork for this project was especially challenging, and hopefully nobody drowned (or got their feet wet)!
You have your homework; now get reading! Farewell auditors and evaluators!
State Profile—Colorado Office of the State Auditor
Greg Fugate and Jenny Atchley (Colorado)
The Colorado State Auditor is a constitutional position appointed by the Colorado General Assembly. The State Auditor used to be an elected office until 1964 when, following a similar trend in other states at the time, Colorado voters passed a constitutional amendment to move the position to the Legislative Branch. The State Auditor’s authority to conduct performance audits was granted in 1969.
OSA’s mission is to improve government for the people of Colorado.
Our audits will identify efficiencies and cost-savings and improve effectiveness and transparency in government.
We will provide objective information, quality services, and solution-based recommendations.
We set the standard for good government.
Types of Audits Conducted
Reviews of annual audits of Colorado local governments
In addition to the state auditor and three deputy state auditors, the OSA currently employs 55 permanent staff, including:
23 performance auditors,
22 financial auditors,
4 IT auditors, and
6 administrative staff.
Our staff can find internal control deficiencies with the best of them, but we also enjoy a number of non-audit activities, including hiking; biking; skiing; snowboarding; photography; painting; cooking; baking; salsa and ballroom dancing; traveling; scuba diving; knitting/crocheting; snowshoeing; food, wine, and beer tasting; creative writing; playing music; and woodworking.
Longest Performance Audit Report
Division of Parks and Outdoor Recreation, Department of Natural Resources (June 2008) – 141 pages plus an appendix.
Shortest Performance Audit Report
Severance Tax Direct Distribution Payments, Department of Local Affairs (August 2007) – 13 pages plus appendices.
Annual Office Traditions
“Squat and Gobble” (Thanksgiving potluck), bocce ball tournament, holiday cookie exchange, chili cook-off, food and coat drive.
Time We Felt the Safest
During the 2008 Democratic National Convention, law enforcement officers and riot police used our building as their staging area.
Five Best Things About Living and Working in Colorado
The outdoors. Colorado has four national parks, 41 state parks, five national monuments, two national historic sites, one national recreation area, 15 national forests and grasslands, fout national historic trails, 25 scenic and historic byways, and 10 scenic and historic railways. Colorado also has 53 mountains with elevations higher than 14,000 feet. You can drive to the summit of two mountains (Pikes Peak and Mount Evans), and the road to Mount Evans is the highest paved road in North America.
Colorado is the “Napa Valley” of beer, with more than 160 established microbreweries. Denver also hosts the annual Great American Beer Festival where attendees are able to sample more than 1,800 different beers.
Six professional sports teams—Denver Broncos, Denver Nuggets, Colorado Rockies, Colorado Avalanche, Colorado Rapids and Colorado Mammoth.
Approximately 300 days of sunshine per year.
Colorado has fantastic theater and concert venues, including the famed Red Rocks Amphitheatre, as well as numerous outdoor festivals and fairs, such as the Telluride Bluegrass and Film Festivals, Colorado Shakespeare Festival, Palisade Peach Festival, National Western Stock Show, Taste of Colorado, Longs Peak Scottish/Irish Highlands Festival, and Ouray Ice Festival.
Jeff Grimes (North Carolina)
My path to a career in program evaluation with the state of North Carolina has been an indirect one. I was very interested in program evaluation in graduate school, but after graduating with a Master of Public Policy degree from Duke University, I took a job for a nonprofit organization in New Orleans. I worked on rebuilding and environmental policy issues following Hurricane Katrina and much of my time was spent reviewing federal and state recovery and rebuilding programs and analyzing whether programs were helping to reduce the future risk of flooding and storm damage. My next job was working for a private foundation where I reviewed grant proposals and evaluated the performance of nonprofit organizations that received grants. I didn’t know it at the time, but these jobs were good preparation for my career with the Program Evaluation Division (PED) of the North Carolina General Assembly.
Working in program evaluation for the state is a very different atmosphere from my previous work experiences, but the skills and type of analysis required are similar. Reading grant proposals and evaluating grantee performance have similarities with evaluating state agency effectiveness—both require objectivity and a need to look at outcomes. Clear and concise writing and the ability to work as part of a team have been important parts of all the jobs I have held.
I have enjoyed working at PED because I get to research and evaluate topics of significant interest to legislators. All our reports are presented to our legislative committee. Knowing our research has that type of audience is exciting. The other aspect of the work that I have appreciated is the emphasis on objective research and evaluation. I enjoy the inquiry process and the ability to follow data and evidence wherever it takes you. I think there is, unfortunately, a dearth of accurate and objective public policy research out there, which makes this type of work all the more important.
My other attraction to legislative program evaluation has been its interdisciplinary and collaborative nature. My first project at PED was working on an evaluation of the North Carolina Railroad Company, a state-owned corporation that owns 317 miles of rail corridor used for freight and passenger rail service. I have always had an interest in railroads—my grandfather spent his entire career working for a railroad—and the project allowed me to really dive into the topic. We reviewed financial data, capital spending, GIS data, properties data, and railroad leases. Synthesizing the relevant information into a report for the legislature was a challenge, but I found it rewarding to have a final report that tied everything together into a series of findings and recommendations.
I’ve been at PED for over a year now and really enjoy the interesting topics we are asked to explore. Back when I did nonprofit work after Hurricane Katrina, I was at times frustrated by government inefficiency and a seeming lack of accountability. Some agencies with which I interacted were not carrying out their responsibilities effectively or in a timely manner. Working at PED, I’m able to provide oversight of state agencies and identify ways to make programs more effective and efficient. From past experience, I know this work is very important, and I feel fortunate to have the opportunity to help make a difference.
Jeff Grimes is a Senior Program Evaluator with the Program Evaluation Division of the North Carolina General Assembly.
Research and Methodology Profile: When Less Is More
Mark West (Florida)
For several years, Florida Lottery officials requested that the Legislature increase funding for advertising to increase lottery ticket sales, 31 percent of which are transferred to public education. The Legislature asked our office to evaluate whether higher spending on advertising would result in higher lottery sales.
To answer this question, we used a statistical technique called time series regression. Using this method, we evaluated whether lottery sales fluctuated with advertising spending. Analyses in other states showed that major reductions in lottery advertising reduce sales; but it was unclear whether the opposite was true. Our approach allowed us to control for factors—in addition to advertising spending—that could affect lottery sales, including jackpot size, retailer penetration, economic conditions, and seasonality (sales tend to be higher in certain months, such as summer months).
We designed a time series regression model with monthly sales as a function of advertising spending. Lottery officials provided us with 33 months of ticket sales data for each of 10 market areas, which we converted to per capita sales and used as our dependent variable. We used per capita monthly advertising expenditures in each of the 10 market areas as an independent variable. We also controlled for retailer density, prize payout, economic conditions, month, and market area. Our final model explained 87 percent of the variation in lottery sales.
Our analysis showed there was no statistically significant relationship between advertising spending and lottery sales, as long as some advertising occurred. In other words, there was no clear evidence that increasing advertising spending would increase lottery sales, nor therefore the amount transferred to education. Instead, sales were more influenced by jackpot size and retailer penetration. However, the likelihood of higher lottery sales did increase when more advertising dollars were spent on additional airtime for existing commercials rather than producing new ones.
Based on our analysis, we did not recommend that the Legislature increase the Lottery’s advertising allowance. Instead, we noted that the Lottery could reduce the amount of its advertising budget spent on production costs by, for instance, increasing the number of times a commercial was aired once it was produced.
If this happens to you: Several issues to watch for if your office implements a similar research design are the effects of lagged advertising spending, serial correlation, and nonlinear prize payout patterns. A lottery commercial may affect sales both in the month it is aired and subsequently. For this reason, we tested for the effects of advertising in the months prior to when sales occurred. As a result, our model included controls for advertising spending in both the month in which sales occurred and the prior month. Literature on time series regression is also filled with cautions about the risk of serial correlation. We tested and corrected for serial correlation using a technique described by Gujarati (1988). Finally, we analyzed prize payout data to see if it had a normal distribution; it did not. As a result we included an independent variable in our model that was equal to the square of the prize payouts in each month.
Mark West is a Staff Director and methodologist for the Florida Legislature’s Office of Program Policy Analysis & Government Accountability (OPPAGA). Report Link
NEWS & BLURBS
Executive Committee News
Rachel Hibbard (Hawai‘i)
The NLPES executive committee (pictured left, hard at work!) met in Austin, Texas for an all-day meeting at the Texas Sunset Advisory Committee offices on Saturday, April 6. This was the first meeting for new EC member Katrin Osterhaus, who joined the committee in November to fill a vacancy.
NLPES' officers are the chair (Karl Spock), vice-chair (Lisa Kieffer) and secretary-treasurer (Wayne Kidd). The executive committee directs and coordinates NLPES’ efforts relating to its annual Professional Development Seminar as well as other NLPES programs and activities such as webinars and other training opportunities, the NLPES listserv, the NLPES newsletter, peer reviews of other NLPES member offices, and evaluating and issuing awards to NLPES member states for achievements in recognized categories. [Executive Committee members pictured (left to right): Rachel Hibbard (HI), Wayne Kidd (UT), Katrin Osterhaus (KS), Angus Maciver (MT), Lisa Kieffer (GA), Dale Carlson (CA), Karl Spock (TX), Nathalie Molliet-Ribet (VA), Charles Sallee (NM), Greg Fugate (CO). Not pictured: Marcia Lindsay (SC).]
Most of the committee members arrived in Austin on Friday afternoon in time to take a tour of the Capitol building, complete with a hair-raising climb to the top of the rotunda (for those who had the guts to do so—yours truly not included!). Friday evening was a Dutch treat dinner for the group at a famed local eatery, the Roaring Fork (yum!).
On Saturday, subcommittees met to discuss their respective areas—Professional Development, Communications, Peer Review, and Awards—and reported back to the full committee. The full committee spent most of its time preparing for the annual NLPES professional development seminar to be held in Austin in September. The committee discussed budget and logistics for the conference and toured meeting facilities at the state capitol, hotel, and two other facilities nearby; and upcoming locations for future PDSs.
After a long but fruitful day of discussion, the committee held another Dutch treat dinner at The Salt Lick, which (according to its website) is a world-renowned BBQ joint serving recipes that date back to the mid-nineteenth century wagon trains. Ever an auditor, Wayne (far left) took a moment to scrutinize the process; but we all enjoyed the results! Although possibly Angus regretted choosing the “all you can eat” option… it was a lot of meat!
The committee’s next meeting will be at the NCSL Summit in Atlanta in August.
Executive Committee Elections
Lisa Kieffer (Georgia)
It’s election time—but this time there are no ads and no robo calls! The NLPES Executive Committee elections occur each year in March and results are announced in April.
Executive Committee members each serve three year terms. Terms are staggered, so that four seats are up for election each year. There are currently no term limits. While retention of existing members helps to provide continuity and ensure continued progress on projects, new members bring fresh ideas and additional skill sets to the table.
The election cycle begins in February, with a membership-wide call for nominations. Those interested in running submit a short biographical statement of no more than 200 words. Statements usually address the individual’s background and his or her interests for the committee. For example, a candidate may be interested in creating more online training opportunities or finding new and effective ways to share information between offices. The statements are compiled and election ballots sent to staff in each NLPES member office. Ballots are returned to the NLPES liaison at NCSL, and votes are tallied. The four candidates with the most votes become members of the Executive Committee. Remaining candidates also serve an important role, however: If a current member of the Executive Committee cannot serve out his or her term, the candidate with the next highest number of votes is appointed to the Committee for the remainder of the departing member’s term—much like the Miss America pageant rules!
The scenario described above occurs when the number of candidates is greater than the number of vacancies, but NLPES bylaws also consider two additional scenarios:
If the number of nominees is equal to the number of members to be elected, the Executive Committee shall, by a vote of the majority, certify the nominees as duly elected members.
If the number of nominees is less than the number of members to be elected, the nominations and elections committee shall recommend and the Executive Committee, by a vote of the majority, nominate a person or people to appear on the ballot so as the number of nominees is equal to the number of Executive Committee members to be elected. The Executive Committee shall then, by a vote of the majority, certify the nominees as duly elected members.
This year we had four vacancies and four nominations, all from incumbent members. As a result, the Executive Committee was asked to certify those four as duly elected members. No membership-wide vote this year!
Please join me in congratulating the following on their re-election to the Executive Committee: Dale Carlson (CA); Greg Fugate (CO); Rachel Hibbard (HI); and Charles Sallee (NM). Next year, please consider running for an Executive Committee position! In addition to interacting with your peers in other states, you will have an opportunity to influence the direction of the profession you have chosen.
2013 NLPES Awards
Wayne Kidd (Utah)
It’s time to prepare your submissions for the 2013 NLPES awards! Please visit the NLPES website to review the descriptions and guidelines, and consider applying for the following awards:
Excellence in Evaluation Award: This award is presented to an office that is judged to have made significant contributions to the field of legislative program evaluation during a four-year period.
Excellence in Research Methods Award: This award is presented to the office or offices producing a report developed with the use of exemplary research methods. The Excellence in Research Methods Award may be awarded to a maximum of three offices in a given year.
Certificates of Impact: Certificates of Impact are presented to offices that released reports documenting public policy impact within their respective states. There is no limit on the number of offices that may receive an impact certificate. However, offices are limited to one submission per year.
The deadline to apply for an award is Friday, May 10, 2013. Each award has three judges; remember to provide your submission to all three judges, whose information is listed on the NLPES website. If you have any questions about the application process, please contact Wayne Kidd or call at (801) 326-1758.
Check out our…
Online training from the NLPES library: Rapid Response Assistance: As sessions heat up across the country, many of your offices receive urgent requests for research assistance. In response, you provide a range of products, from verbal briefings to memoranda to published reports. For a quick refresher on how to handle these time-sensitive assignments, visit our NLPES online training library. The narrated PowerPoint on rapid response assistance is a brief five minutes of good advice, put together by the Mississippi Joint Committee on Performance Evaluation and Expenditure Review (PEER). Check it out!
NLPES website: Learn more about NLPES and see what we do—spend a few moments touring our NLPES website. You’ll find general information about the NLPES including by-laws, executive committee membership and subcommittees, state contacts, awards, and information on peer review. We also have a training library and document resources including past meetings, minutes, awards, newsletters, and more. Come on by and visit for a while!
NLPES listserv—Join our NLPES listserv and:
Get our NLPES Newsletter delivered directly to your email box
Query other states about evaluation work similar to what you’ve just been assigned
Receive email announcements about performance evaluation reports from other states!
Simply send a blank email to email@example.com with the subject area blank and the word “SUBSCRIBE” in the body. You will receive a welcome message if successful (be patient; the response may take a little while). See the listserv link on the NLPES website for additional information such as how to post messages to the listserv and “netiquette” niceties, such as not hitting “Reply All” when answering a query posted to the listserv. You’ll be glad you joined!
Legislative careers website—Know a young professional thinking about pursuing a career with a state legislature? Point them to the opportunities posted on NCSL’s legislative careers website. Job seekers can explore the various types of work legislative staffers perform, including performance evaluation, budgeting, fiscal analysis, legal and policy research and opinions, bill drafting, public relations, librarians, building security, and information technology support. Opportunities are posted by states offering positions under Legislative Jobs.
Launched by NCSL in June 2012, this is a great website. According to NCSL, attracting young people to work as legislative staff will be increasingly important in the coming years. And even though baby boomers make up about a third of the national workforce, nearly half of legislative staff responding to a survey were 50 years old or older. Replacing those staffers will present challenges.
Check out the site’s welcome video, “A Day at the Capitol,” and learn more about the opportunities and rewards of working for state legislatures. Watch the videos under Career Paths to hear from staffers in different states describing their jobs and the satisfaction they gain from a public service career. Check it out!
NCSL Legislative Summit: Georgia will host the Legislative Summit in Atlanta from August 12–15. Registration is now open, and early bird rates are valid through May 31. Check out the NCSL website for details!
NLPES Professional Development Seminar: This year’s PDS will be held in Austin, Texas from September 23–25. Details are not yet available, but watch the NLPES homepage and NLPES News!
Look for the Summer 2013 edition of the NLPES Newsletter in August!
NLPES-NEWS is published by the National Legislative Program Evaluation Society, a staff section of the National Conference of State Legislatures. NLPES serves the professionals of state legislative agencies engaged in government program evaluation. The purposes of NLPES are to promote the art and science of legislative program evaluation; to enhance professionalism and training in legislative program evaluation; and to promote the exchange of ideas and information about legislative program evaluation.
2012-2013 NLPES Communications Subcommittee:
Dale Carlson (CA)
Charles Sallee (NM)
Rachel Hibbard, newsletter editor (HI)
NCSL Liaison to NLPES:
Brenda Erickson, (303) 856-1391
National Conference of State Legislatures • 7700 East First Place • Denver, Colorado 80230 • (303) 364-7700