This issue's shoptalkers: Glen Bolger, partner and co-founder of Public Opinion Strategies; Harrison Hickman, founder of Hickman Analytics; Alex Lundry, vice president and director of research at Target Point Consulting; and Matt Hogan, vice president at Anzalone Liszt Research.
C&E: Is the cellphone-only problem the biggest challenge for pollsters going forward?
Glen Bolger: You want the sober answer or the drunk answer?
Harrison Hickman: The cellphone problem gets worse over time and here’s why: someone living in Northern Virginia gets a cellphone and then they move to suburban Atlanta. So even if you’re sampling suburban Atlanta cellphones, you’re going to miss that person. The one thing I think we could all agree on if we were kings of the world is that telephone exchanges would all be contiguous.
Matt Hogan: It’s a real problem.Cellphone-only households are now a third of the country and we’re continuously shocked by how many pollsters are still not using cellphones in their samples. Maybe you could get away with it in 2010, but not next year. Thankfully it’s getting a little cheaper, but it’s still tough for campaigns to pony up. If you want an actual representative sample you need to spend the money on this.
Bolger: And that’s the real issue. Matt hit on it directly. It’s more expensive to interview cellphones. Doing it on the national level is obviously easy. But on the statewide level it’s tougher. Some statewide campaigns have enough money and realize they need to do it, but that’s not all campaigns. The other thing is that you can’t do cellphone interviewing below the statewide level on a fairly rigorous basis. If you call someone in Northern Virginia and reach them on their cellphone, you can’t ask what congressional district they live in. They’ll say, “Beats the hell out of me.”
Alex Lundry: The problem gets even worse when you look at cell-mostly households like mine. We have a landline in the house, but nobody knows the number and whenever anyone calls it we look at the caller ID to see if we want to pick it up. This is another 15 to 20 percent of the electorate.
Hickman: And you can’t make up for it by weighting. You used to be able to just adjust the number of people who are under 35 and weight them more. But the people who are landline only have a fundamentally different lifestyle and attitude, so you can’t make the traditional adjustment.
Lundry: The agency that actually tracks cellphone versus landline is the Center for Disease Control. They have an interest in tracking it because one of the first initial differences between the cellphone-only population and the landline population was on health issues. People with cellular service only were more likely to binge drink, use illicit drugs, have STDs…
Bolger: Vote Democrat. (laughter)
Lundry: Right. As that cellphone-only population has grown, though, we’re finding there are real partisan differences between the groups, too. We’ve been getting away with not doing this for a while. Now, I think it’s malpractice for a pollster to not offer a quote that includes cellphones.
Hickman: These public polls that rely exclusively on computer assisted interviewing are going to start making huge errors because they can’t reach the cellphone-only population.
C&E: How frustrating are public polls, especially when a bad poll gets attention in a race you’re working?
Bolger: They’re mostly background noise. The issue for me is when they’re automatically presumed above it all and the campaign pollsters are the ones presumed suspect. The case I make is the exact opposite. If our polls are wrong and we’re giving bad data to our campaigns, we don’t get hired anymore and we don’t get to feed our families. If the media poll is wrong they just say, “Well, this ad clearly changed things in the past two days.” The idea that media polls are somehow superior is the one thing that gnaws at me. I’m not going to give happy data to a client and then after they lose say, “Sorry, our data was totally wrong.”
Hickman: And media organizations are more budget constrained than campaigns are. It’s sort of difficult for people to believe sometimes, but a TV station may have $12,000 set aside for this kind of project. A campaign would be paying $25,000 for the same poll. So of course we can do it better.
Hogan: It’s just frustrating how many of them don’t release their demographics. They’ll release some poll that shows a big shift and really drives the narrative for a couple of days. In reality, there probably wasn’t any shift at all and their sample is just more Republican or Democrat than it was in the previous poll. Reporters are getting a little better about covering that and picking up what the sample is, but too few of these media pollsters release all their data.
C&E: How about the pollster as pundit? The ones who seem to spend more time on cable TV than on campaigns?
Lundry: I actually thought you were going somewhere different with the question, which is this new crop of pollsters—especially the IVR (interactive voice response) ones—who are clearly polling just for media hits. I think Public Policy Polling is probably the primary example of that. They do a good job from time to time, but some of the questions they ask are just clearly generated as link bait. They’re asking Barack Obama versus Charlie Sheen in their polls, which is just ridiculous. I think that’s more of an issue than the pollster as pundit.
Hickman: At least PPP has people who pay them for polls. Some of these people don’t even have paying clients. They just do it for publicity.
Lundry: And to their credit it has worked. They’ve got their name out there and built their brand very aggressively. But every once in a while PPP asks a question that really makes me cringe and wonder whether that’s making the role of pollster in the public domain better or worse.
C&E: How can you make use of social media for survey research? Have you been testing ways to incorporate it?
Lundry: Absolutely. We’re doing just that sort of thing with sentiment analysis of data that’s out there on the Internet. I think that we’re never going to get to a place where analysis of social network data or blogs or online news is going to replace polling. But it’s a useful indicator of when you should go in the field and what you should go in the field about. It’s a signaling mechanism.
Hickman: Is it leading or is it anomalous?
Lundry: It’s leading. In fact, all of the studies that we’ve done and others that have come out publicly by academics indicate that there is predictive power in these things. There was one simple study that took tweets with the word Obama and coded them for sentiment—either positive or negative—and used it to predict the Gallup daily track three days ahead of time. There was a correlation of about .75 between the two. Twitter has also been used to predict the stock market; it’s been used to accurately predict opening weekend box office returns for movies. It’s not perfect and it’s not a replacement, but I think it’s a pretty incredible complement that increases the efficiency when you go in the field.
Bolger: We’re still trying to figure out how to maximize analysis of blogs and Twitter and everything else. But people aren’t using social networking for politics as much as politicians think that they are or might want them to. These are more like early indicators or early warning systems as opposed to something that gives you a true sense of the state of the race.
Lundry: I also think the conversation on Twitter is somewhat limited. There’s a lot of great ability to look at the aggregate effect of communication—what news articles are out there, what’s on radio, what’s on TV. That’s where we find real value in it. It’s a leading indicator of what information people are being exposed to. And opinions don’t change without exposure to new information.
Hickman: You do see some things that are really fascinating about the number of mentions in social media. I’m sure that the number of mentions of Herman Cain has gone from some relatively small number to x plus a really big number.
Bolger: But he’s the exception and not the rule. I can’t tell you after 2008 how many candidates would say, “I want an Internet presence like Obama.” Well, ok, because that’s really easy.
Hickman: Sure, and I’d like to play in the NBA. (laughter)
Hogan: This is separate from social networking, but given how difficult it is to get young people on the phone I think we’ll see more use of online polling. It’s not representative, so you don’t want to use it for your basic polls. But for message testing I think there’s a lot of stuff you can do online that you can’t do over the phone.
Lundry: I actually think we’ve been having the wrong debate in the community about online research. We’ve been wondering how we can ask the same questions online and get the same answers we get on the phone. That’s not the right way to think about it. We need to figure out how to ask different questions that take advantage of the medium and come up with more interesting and maybe more insightful answers.
Hickman: And this is the hardest thing for me, but you do have to sort of suspend this commitment to having a perfectly random sample. You have to just accept that it’s a cross-section of some sort.
C&E: Same question when it comes to smartphones and tablets—how will those apply to survey research?
Lundry: We’ve never done anything with it, but I have seen some very interesting demos of ethnographic research—the idea of taking pictures of what you’re seeing in the shopping aisle. There are some companies that are experimenting with push surveys that are based on where you are at a given moment. So an application could know when you were in a supermarket and then push a text asking you to take a picture of what you’re looking at on the shelf.
Bolger: The challenge with all of this is that ultimately the bread and butter of what we do isn’t changing. It’s still about the status of the race, what messages are working and what we don’t want to put our money into. From that standpoint, all of this technology just isn’t there yet. We can do great things in terms of ad testing, but in a campaign we still need to know what groups are moving and what groups we’re lagging with. As an industry, we’re a long way from figuring out the new technology in terms of being able to reach these folks.
Hickman: I think the bigger use for most of us will be in the distribution of our findings to our clients—especially the tablets. That’s something clients will increasingly want from us so they can have their data in a form that they can manipulate on a tablet.
Lundry: I’m waiting for Facebook to open up its list for survey research. If they’re smart, I think they’ll realize it’s a way to monetize users very quickly and I’m sure that they’re having internal discussions about it. It would be the biggest survey panel in the world and it would be very interesting to see what sort of accuracy you can get out of it.
C&E: How well does the polling industry police itself and should individual pollsters be doing more?
Hickman: This is a business any fool can get into and a lot have. There isn’t a real barrier to entry and it’s a big problem for people who do it right. What happens is that you end up competing on price rather than quality. Now, in their defense, a lot of people who sell cheap polls don’t know how to do it right. And you can actually use some of these inferior methodologies and still get it right 75 percent of the time. What people really pay us for is that other 25 percent. You’re also right to call it an industry because it’s not a profession. It’s disappointing to me that we don’t do a better job of regulating people and don’t push people out of the business.
Hogan: Too many campaigns just don’t get the importance of cellphones or of bilingual calling for Latinos, for instance. So when you’re competing against pollsters who aren’t factoring that into costs, it’s tough to beat them on cost.
Bolger: A lot of candidates also think they’re experts on polling because they’ve seen a poll once or twice. They don’t know what goes into writing a good questionnaire or testing messages. It’s tough because you have an audience that thinks they’re experts, but they’re really not.
C&E: So how do you regulate the industry better?
Hickman: Well, AAPOR (American Association of Public Opinion Research) has a model. It’s pretty difficult to get somebody censured, though, and it’s hard because you are dealing with somebody’s professional livelihood. I can only think of four or five people who have been censured in a significant way and in a couple of instances it didn’t seem to have hurt them at all. But all professional organizations struggle with this. It’s a difficult thing, but I think we should require folks to have some level of training or at least have worked for someone with some training.
Bolger: I have a different viewpoint than Harrison, which is that it’s really hard to do this because campaigns are basically the Wild West. You just have to be patient and hope that quality wins out in the long run. There are lots of folks who have tried this and realized it’s a lot harder than they thought. I just think it’s very difficult to say, “Here’s what you need to have done to enter into this business.” If that was the case, why wouldn’t you do the same for media consultants?
Hickman: You probably should.
Bolger: I’ve thought about this over the years, but I just don’t think there’s any good way to do it.
Hickman: I don’t necessarily disagree with that. I think if there was a good way to do it, it would have been done. Bob Teeter and Peter Hart tried to form a super subset of pollsters at one time who would agree to certain sets of standards and it just failed.
C&E: Shifting to 2012, is Obama as weak as his numbers suggest?
Lundry: I think there’s no doubt Obama is very weak. The problem is that he’s probably the greatest campaigner we’ve seen in a generation of politics. He’s got an amazing organization and defeating him isn’t going to be easy.
Bolger: Demographically, Obama has huge advantages. Whoever tells you the African-American vote is at risk of not turning out, that’s bullshit. African-Americans are going to turn out and strongly support Obama. And the Hispanic population is growing. We need approximately 60 percent of the white vote to win nationally in the popular vote. John McCain won whites by 12 points. George W. Bush won whites by 12 points in 2000. The reason Bush was basically in a tie nationally and McCain got blown out is that there were more minority voters last time. Another advantage for Obama is the map. If you look at the Electoral College, states won by both Kerry and Obama equal 246 electoral votes. That’s not too far from 270. Bush-McCain states equal 179. This is going to be a very close race. I don’t think Obama can lose big and I don’t think he can win big.
Hogan: I think Obama will also benefit from the fact that this is a choice. The economy is struggling, but once the president is able to make a contrast with the Republican nominee on the issues—tax cuts for the wealthy or cuts to Medicare, swing voters are overwhelmingly on Obama’s side.
Hickman: If the election were today, up or down on Obama, it would be a landslide against the president of historic proportions. If the nominee were Romney or Huntsman or someone in the center, I think it would be overwhelming. I think he’d get to 240 electoral votes, but he wouldn’t win.
Hogan: I don’t agree with that at all. What makes you say that?
Hickman: Well, if you look in the swing states, Obama’s approval rating is about 40 percent. In the Gore campaign in 2000, we won every close state except one. We would argue we won that one, too. But we won Iowa, Wisconsin and New Mexico. We won every state that was within two or three points. It’s just hard to run the table when you start off with a baseline of 40 percent approval for an incumbent. You’ve run out of the ability to argue that it’s George Bush’s fault at this point.
Hogan: I disagree. I think if it was a referendum, you could make a case that he was in trouble. But it’s not. It’s a choice.
Hickman: But that’s why I say if it were today. If the election were today it would be too tough.
Bolger: I don’t agree with either of you. I don’t think it would be a blowout, but it is a referendum. Presidential elections are referendums. Obviously there are certain thresholds you have to meet, but voters are trying to decide whether they want to rehire the person for another four years.
C&E: How about the House? Will these Congressional races turn on national issues?
Lundry: Yes. I think we’re going to have yet another nationalized House election. We’re increasingly looking at parliamentary government here in that these House elections are just like list-based parliamentary elections.
Bolger: But I think it’s very difficult for the Democrats to take back control of the House. They can pick up some seats, but reelected presidents don’t have long coattails. Nixon, Reagan, Clinton—none of them had long coattails. And secondly, this is the first time ever that Republicans in the modern era have an advantage drawing district lines. Personally, I’m not a huge fan of gerrymandered redistricting just because I think more competitive elections are good for business...uh I mean good for the country. (laughter)
C&E: Can we look at unemployment and the right direction/wrong track number and predict Obama’s fate?
Hickman: We’d be out of business if it was that simple.
Hogan: We’ve actually stopped asking right direction/wrong track in our polls. You’re getting about 80 percent wrong track. At this point it’s not even useful anymore.
Bolger: I think you’re wrong. It’s very useful. You look at the times right direction has been below 25 percent—1980, 1992, 2006 and 2008. This is the fifth time and it’s the most sustained negative mood in modern American history. Look at 2000 and 2004—you had 45-50 right direction/wrong track. Those were 50-50 elections.
Lundry: You talk to some political scientists and they’ll tell you, “Here are the three variables you need to predict the outcome.” There’s some truth in that, but those models will only explain 80 percent of the variance. The other 20 percent is where campaigns win and lose. And that’s why you still need tools like ours to figure out that unexplained variance.g