C&E: How much is that lessening on the Republican side?

Gimpel: In some sense I think the business orientation of the Republican side has helped. They want to know if their money is being spent wisely. A lot of the resistance is still there, but some donors are demanding this now. Because many of them are aware of experimentation, Republican donors are saying, “We’ll put up this money, but we want to know what we’re getting.” This is especially true in the 527 community.

Malchow: Alex Gage once told me a story that really illustrates this well. It was 2002, and he had just opened his firm to do modeling and use advanced statistics to determine which voters should be targets. He goes up to pitch Mitt Romney’s gubernatorial campaign, and his whole business concept is on the line here. He’s got to get a big client or this thing is not going to work. He goes into this impassioned speech about how all these techniques are so important and that it’s a new way of looking at things. The whole time his audience is just looking at him in this perplexed manner. He was sure he wasn’t going to be successful with the pitch. He finishes talking and one guy raises his hand and asks, “You mean to tell me they don’t do these things in politics?” (laughter)

Green: When we got going with Gov. Perry in 2005, it was a good example of what experimentation can do if the conditions are right. If you have a campaign manager like Dave Carney, who has a long term view, this can be very effective. He knows that he’s going to be back on the same political turf again and again and therefore wants to know what’s going to work for the long haul. Carney himself was interested in whether his subcontractors were spending his money wisely. So it was just a very nice set of conditions.

Dave was the one who asked, “Can we do something with respect to fundraising? Can we do something with robocalls and direct mail?” The best experiments are done with some practice runs. The nice thing about 2005 was that we did a lot of practice runs. We did two or three rounds of fundraising experiments and messaging experiments. That allowed us to get ready for the big dance, which was the radio and TV experiment.

C&E: Is that the best way to get campaigns to buy in? If you have the manager on board, is that enough?

Green: I think that’s the main thing. In that race, Dave was the one who deployed all the resources. That gave us the opportunity to run a very complicated yet informative mass media experiment. I’m not sure whether this has caught on, but this is the first TV experiment that I know of in a large market. The fact that we had Dave Carney saying to media buyers, “You’re going to roll out this media campaign in 18 experimental markets over a series of weeks”—that’s a unique opportunity. We were able to tell the campaign how many GRPs to spend, and then we could randomize when and where. On the back end, we had their survey operation to track results. Everybody was on the same page, and the only way you get that is for the manager to be directing the show.

Gimpel: You would need to ask Dave about this, but I think he had some salesmanship to do internally. There were some vendors who were perhaps a bit threatened and not so enthusiastic about being audited in this way.

Malchow: I remember pitching this stuff to Mark Penn back in 1996. He was advising Bill Clinton, and he saw the value in this. The problem was that the political wrestling that he would have had to do to make it work just wasn’t worth it for him.

Beach: When you’re fighting your political director, your finance director and all of your other consultants, you’ve got to really be a strong chief executive to push that across the whole board. That’s why you see it at the committee level more than you do in campaigns because they at least have long term leadership.

C&E: Given that unfettered ability that you had with the Perry campaign, what was the most interesting finding for you?

Green: For me, it was the impact of TV ads. The most interesting finding was that these ads had an immediate impact that was really quite substantial. They moved the dial noticeably in terms of public opinion, but then a week later the impact had subsided. People might have imagined that was true, but they didn’t know it. I also wonder whether it’s true for negative campaigns, or campaigns down the stretch run of the election. There are all kinds of things that people might want to know that this one experiment doesn’t answer. This is, so far, the only hard experimental data on the question. If it is true that in general the effects of TV advertising subside within a week or two, then that dramatically changes the calculation in terms of strategy.

Gimpel: We learned some things on the fundraising front, too. For small donor fundraising we learned that the type of message doesn’t matter much; it’s just that you contact a certain microtargeted group of donors. The fact that you make the contact is more important than what message you use.

Malchow: The whole issue of measuring and understanding persuasion is so much more complex than GOTV. At the Analyst Institute, I’m sure we have 300 experiments that have been done. Some of them are almost timeless. You can move them from one campaign to the other and they work. However, what we’re learning about persuasion, particularly persuasion mail, is that messaging can make a big difference.

I wrote a book in 2003, and the most fundamental assumption in this book was that undecided and independent voters were the primary targets of persuasion activity. In fact, we found that it varies widely from one campaign to another. This led to a new approach to persuasion mail where we mail in August. We might test one treatment group or we might test three treatment groups. We measure them against the control, and then you can come back to not only pick the creative treatment that works the best but also to model the movement so that you have a greater impact when you roll out.

Gimpel: I think we’re also going to see a wave of interesting experiments come out of this cycle on digital media. For instance, advertising against the login screens of people’s emails and some pre- and post-testing of persuasion. The Democrats have also done some very interesting things related to size. They have been sending out these really huge placard-sized mailers because that’s something that really grabs attention.

Malchow: The most interesting result we have gotten came from an experiment that I did in 2010 for Women’s Voices, Women Vote. They have actually made testing the foundation of their work. I can bring them something completely off the wall and they are willing to test it. We had some experience testing voter registration, and we learned that it made a huge difference if you just strip out all of the political rhetoric and create a piece that was black and white and looked like it came from the government.