We noticed that focus groups would reject a piece of mail because it looked like advertising. So we did two pieces: a traditional postcard and one that was just a sheet of paper lasered on both sides. No photographs, no colors, nothing. The letter basically presented a comparative statement, not a true negative, about the two candidates. The letter not only moved the needle 11 points over the control, but the recall on the letter was almost 80 percent higher than the recall on the two postcards. It was a very extraordinary result. Now, you can’t revamp all of persuasion mail on the basis of one test, but it is very suggestive that there is a window of opportunity in getting away from the graphics and the advertising look.
C&E: What’s happening with all of this on the digital side?
Beach: I think we’re looking at questions as basic as the percent of media spent online versus offline. In our world, this is still something being debated. Even the most aggressive campaigns probably aren’t going to spend a share of their paid media budget based on the ratio that people are actually getting most of their content online versus broadcast. But I think it’s a good test for experimentation because we do control so much of the process under one roof. We’re also doing these things every day—a fundraising experiment every day, a voter contact program every day. That’s one area where you could really see an influx of academic help.
Gimpel: You could do so much more online than you can do over the phone. The other thing that’s really important is not just getting people’s judgment about message, but seeing whether we can actually get people to behave in a certain way. And that’s what online does—it allows you to see if someone will actually click or behave in a certain way.
Green: I think this challenge of measuring the outcomes we most care about is a real one, and it’s not just real for experimental science. Too often, what people stop at is the immediate survey response when what we often care about is a behavior that is going to be lived out days or weeks later. Even though I’m proud of the Texans for Rick Perry experiment, at the end of the day it was still about survey responses. The next generation of experiments will take advantage of ready access to a wide array of different geographic media markets and maybe even individual level media markets with the ability to increasingly target messages to individuals. The ultimate experiment will measure voting choice—voting behavior at the precinct level or the individual level through a post-election survey.
Gimpel: There are other elements of behavior that we care a lot about. Getting people to sign up to volunteer or to make a small donation, for example. This is the kind of thing that we might be able to test online.
Green: Jim’s absolutely right. As resistant as campaign consultants are to GOTV experiments because the results won’t be known until the show is over, you would think they would be very eager to do organizational type experiments. Those results, in terms of how many volunteers you got or how many people actually showed up to a rally, are known in real time. If you had a far-flung campaign over a variety of battleground states, you’d be able to test fairly quickly to see whether your mode of getting the word out is actually generating bodies at events. C&E: From a practitioner perspective, is this a generational thing? Are there younger operatives now who are less averse to experimentation?
Beach: As somebody who came into the business after “Get Out the Vote,” I think that book really had a big influence on us. You saw it in the Bush-Cheney operation in 2004. The same people are now asking the question, and have been asking the question for a year now, what’s next? What are we doing that’s incorrect? It’s happening even with something as basic as measuring everything back to the marginal cost. That’s a basic thing in the business world, but it’s something we don’t do in politics. We send direct mail out to recruit a volunteer, but then we don’t go back to figure out how many volunteers we got from that $30,000 mailer. Every time I’ve ever done one of those things and then taken a nonscientific poll, asking, “How much do you think this going to cost,” everyone usually guesses about five percent of what the actual cost was.
Malchow: This was a barrier in getting people to test GOTV tactics at first because they would say, “I’m doing phone calls or sending mail, and it’s costing me $2 or $3 to get a vote.” When the returns came in, you would find out that it was actually costing $15 or $20 at best. Then you would go and tell people what worked best and tell them it only cost $20. A lot of the practitioners were horrified.
Green: I think of this as the trail-of-dollar-bills test. How much more effective is it than if I just leave a trail of dollar bills from the voter’s house to the polling place? Very often when you’re talking about $150 per vote, you probably would have been better off just leaving that trail of cash and making it a scavenger hunt