John Anzalone is a partner at Anzalone Liszt Grove Research, a Democratic polling firm. Glen Bolger is a partner at the Republican polling firm Public Opinion Strategies.
If the 2012 cycle revealed anything about the current state of polling, it’s that, overall, it is still incredibly accurate.
Based on Pollster.com’s averages, public polls predicted the correct winner in 11 of the 12 presidential battleground states. And in the 12th—Florida—they were less than one point of, showing a tie in a state that President Obama won by nine-tenths of a point.
Not only were public polls extremely effective at picking the winners, they were remarkably good at predicting the margin of victory as well. Once again, in 11 of 12 states, they projected a margin of victory within three points of the actual result. The one exception in this case was Colorado, where their margin was four points of.
But while polling remains very accurate, because of the increasing difficulty of reaching and keeping people on the phone, getting it right is becoming harder and harder to do. In 2012, pollsters had to dial an average of 10 numbers to get one complete interview. In 1997, they only had to dial three. This plunging response rate makes it harder and more expensive for pollsters to get the representative sample they need.
So given the challenges posed by falling response rates, what does it take for a pollster to get it right these days? The industry must continue adapting to change.
According to the Center for Disease Control and Prevention, by the end of 2012, 36.5 percent of Americans lived in a cellphone-only household, with that number increasing by 2-3 percent every six months. That means by the time voters cast their ballots for the 2014 elections, over 40 percent of Americans won’t have access to a landline.
At this point, dialing cellphones should be a must for any pollster, especially when conducting polls in areas with a significant Latino population given that over 50 percent of Hispanic adults are cellphone-only. But just dialing a few isn’t enough. We believe that 20 percent should be the minimum share of cellphone interviews in any poll, and the goal should be higher, especially in areas with greater diversity or more young people.
As is typically the case, the main obstacle to doing cellphone interviews is cost. Because federal law requires that cellphone numbers be dialed by hand rather than by an auto-dialer, calling cellphones makes polling more expensive. So instead of incurring the higher costs of cellphone dialing, some pollsters have instead opted to cut corners by increasing the number of young people and minorities in their polls without dialing cellphones.
But weighting up these groups will not solve the problem of not conducting cellphone dialing, as those who live in cellphone-only households hold distinctly different attitudes from those who do not. For this reason, not dialing cellphones can cause a campaign to make serious strategic miscalculations.
In Florida, President Obama’s advantage over Romney among 18-34-year-olds who were polled on cellphones was 28 points higher than his lead among 18-34-year-olds who were polled on landlines. A pollster who did not include cellphones in the sample would therefore have identified this age group as a significant problem area for the president, while those who did would recognize it was one of his most important groups to turn out.
Though cellphone dialing gets more attention, using bilingual dialers to call Hispanic households is another essential component of the modern pollster’s toolkit, and one that is too often overlooked. According to the Census, 35 million Latinos now speak Spanish at home, up from 25 million in 2000. As a result, pollsters who fail to use a sufficient number of bilingual dialers to capture this group will get a very inaccurate picture of where the Hispanic electorate stands. That’s because, like cellphone and landline phone users, the political attitudes of Spanish-speaking Hispanics differ significantly from the views of their English speaking brethren.
For instance, in Nevada in 2012, about a quarter of Hispanic poll respondents opted to take surveys in Spanish and they were far more supportive of Obama than the Nevada Latinos who took polls in English. While Obama’s support among English-speaking Latinos still exceeded 60 percent, he did a remarkable 24 points better among those who preferred to speak in Spanish.
This divide, and the inconsistent use of bilingual dialers by pollsters, may help explain why the public polls in states with large Hispanic populations like Colorado and Nevada tended to be a little further of the actual results than polls in other states.
Just like cellphones, those pollsters who do use bilingual dialing often cut corners when doing it, such as calling back Spanish respondents with a Spanish-speaking interviewer rather than having a sufficient number of bilingual dialers on hand. Having to schedule callbacks with Spanish speakers is a good way to ensure that your poll will under represent these voters.
While getting the dialing end of polling right is critical, it won’t do you much good if you don’t have an accurate view of what the electorate will look like before dialing starts. Of course there is no way for a pollster to know exactly what a future electorate is going to look like, but the accurate ones base their projections on data from recent, comparable elections, and then make small adjustments based on census data and other outside factors.
Beware of any claims that the demographic or partisan makeup of a future electorate will differ significantly from historic trends. The truth is that while the candidates and issues may change drastically from cycle to cycle, changes among electorates are much more incremental.
For proof, just look at the similarities between the exit poll data in 2006 and 2010. Because one produced a 31-seat victory for Democrats, and the other delivered a 63-seat gain to Republicans, we often think of these two elections as cases of one party streaming to the polls while the other stayed dejectedly at home. Yet the data tell a very different story, as the partisan makeup of the two electorates was virtually identical. It differed by just two points. In 2006, Democrats held a two-point edge in party identification (38 percent to 36 percent). In 2010, the two parties were tied (35 percent to 35 percent).
The dramatic difference in results was due less to changes in who went to the polls and due more to the 38-point shift in the vote among independents. It shows that even in wave elections at opposite ends of the political spectrum, demographics and recent turnout trends are stubbornly persistent.
Take more time in the field
Polling is like fine wine—you have to give it time to produce the quality you expect. Campaigns and clients are always in a hurry, but polling should not be rushed. Pollsters now have the challenge of contending and competing with a food of other polls, paid phone programs, modeling, and marketing calls that make it harder for them to collect good data. Add on top of that caller ID and robo notifications from dentists, schools, and pharmacies, and it’s clear that people are being inundated with calls.
As a result, it becomes much more difficult to reach voters and get a representative sample. And that is where taking your time, doing proper callback procedures and lengthening the number of days you field a poll will help ensure quality results. At the end of the day you want the right interview not just any interview, and that may mean calling the prime person identified in a sample five times or more over the course of a poll before you move onto his or her neighbor.
Where are we headed?
Given all the challenges facing pollsters today, what lies in store for the future of polling? We are most likely headed towards a blended methodology approach, in which a single poll may be conducted, not only over landlines and cellphones, but also online and via text messaging. While calling voters on their cellphones may be critical to reaching them now, there is already data suggesting this method will be insufficient for reaching future generations of voters.
Last year, Pew Research found that just 39 percent of teenagers talk on their cellphone daily. That’s compared to 63 percent who text every day. Reaching voters on their cellphones is likely to become even more important, but pollsters may have to go about doing it in a completely different way.
Pollsters are increasingly turning to online polls as a cost-effective way to measure voter attitudes, and while online polls can be a very valuable supplement to telephone polling, they are not a substitute. This is because the same quality that makes online polls affordable their opt-in panels in which respondents receive incentives for taking surveys also prevents them from being truly random or representative.
Given the shortcomings of the sample, online polls are not ideal for gauging the state of the race or identifying key target groups. However, as long as care is taken to ensure that they represent the demographic and partisan makeup of the electorate, online polls can be an extremely cost-effective way for campaigns to test messages, mail pieces or ads.
In fact, some of the best message-testing methodologies are now only available online, making online polls arguably a better tool for testing messages than traditional polls, and at a fraction of the cost. This makes them an especially useful tool for campaign pollsters, as our role is much more about message development than prognostication.
There’s no doubt polling is becoming more difficult, but as the 2012 elections demonstrated, there is no evidence that it’s becoming less accurate. There will always be bad polls, but when taken as a whole, polls continue to be remarkably reliable at predicting election results. Whether that trend continues remains to be seen, but for now pollsters who do it right, get it right.
And as long as they continue to adapt to new ways of reaching voters and allow data to drive their decisions, we believe that pollsters will continue to be just as accurate in the years ahead.