This issue's shoptalkers: Michael Beach, partner at the Republican firm Targeted Victory; James Gimpel, professor of political science at the University of Maryland; Don Green, professor of political science at Columbia University; and Democratic strategist Hal Malchow. 

C&E: Let’s start with some history. Where does the story of experimentation and campaigns begin?

Hal Malchow: The history begins with many rejections of the idea of measuring what we do. I remember talking to the Democratic National Committee in 1992, and again in 1996, and then 2000, 2004, and in 2008, all about the importance of actually measuring what we’re doing. Almost every time the answer was no. They would say, “If we pull 10,000 voters out in a control group, it could cost us the election.”

But the real reason we were losing was that 80 percent of what we were doing was totally worthless. It’s almost a scandal that Don Green and Alan Gerber in the academic community had to come along and show us how this work can be accountable. Their work finally started to get attention, and that’s when the tide began to turn. I know the Republicans got there earlier. They were actually doing experiments and measuring in 2001 and 2002.

Michael Beach: That was obviously a huge part of the 2004 election. It was probably what helped drive the Democratic side to do a similar thing. I think it was the close call in 2000 in Florida that drove the need for experimentation, but it’s definitely been a wholesale change. The question now is can we continue to evolve and do it again?

Don Green: How did things change from 2000 to 2004?

Beach: The whole 72-hour victory program for the Republican National Committee was a big part of that. The weekend before the 2000 election you were getting ready for the election night party and not thinking about voter mobilization efforts. It seems crazy, but GOTV was basically about getting on television. Having a field program is no longer enough now. You need to look at new techniques and talk to voters in different ways.

James Gimpel: In 2002 Karl Rove and Ken Mehlman called in some Republican inclined academics to provide advice. That’s where academic involvement got started on the Republican side. It was really about two things. First, what academics have to say to consultants as far as voter file augmentation goes. The other part of it was this evolution of experimentation. Blaise Hazelwood was the political director going into 2004, and she did some experimentation on GOTV. I’m not quite sure there was all the strict balance and methodological perfection that we would demand on the academic side, but they were at least thinking in terms of doing the right comparisons.

Green: Had Karl Rove and others been doing randomized experiments prior to 2000?

Gimpel: I don’t know of anything prior to 2000. I think your early work certainly got their attention. I think there’s some suspicion when it comes to academics, so the Republican instinct is to kind of hold them at arm’s length and then try to replicate what they’re doing within the campaign. They wanted to do it on their own.

C&E: What’s your sense of the dynamic between academics and practitioners?

Green: There is this deep suspicion about two aspects. One is the one that Hal pointed out, which is the concern about having academics meddle in a campaign by withdrawing a control group. That is often a show-stopper for consultants. But then there’s the part that Jim alluded to, which is just the general concern about having findings disclosed to the public. People would be embarrassed if their tactics don’t work, so for any person who aspires to do an experiment it’s a delicate dance. One of the things that has changed in the wake of a lot of randomized experiments is the sense of how catastrophic a 10,000-person control group would be. People have seen other campaigns withdraw a control group and still live and learn to move forward with other techniques. Another change is that you now have people in this experimental enterprise who have been trained and are propagating this research method in other circles.

Gimpel: I know one of the concerns the consulting community has had is whether we can generate these results in time for them to be actionable. I think the experience with the Rick Perry gubernatorial campaign indicated that can happen. But with GOTV, if we have to wait until the voter file comes out with the completed vote history in December, how can that be useful to campaigns now?

Malchow: I think it’s worth pointing out that you still don’t see experiments inside the actual campaigns. Part of that is just the short term interest of the enterprise. But now you’re seeing an enormous number among the nonprofits and advocacy groups. Things have really come a long way.

Green: Hal, in your days as a consultant you actually conducted your own internal experiments despite the same short term pressures. What’s preventing others from doing what you did?

Malchow: We did a few experiments within campaigns, but generally we had to cut them some sort of deal. We’d give them more mail than they were going to get to make up for the control group, for example. But working with the nonprofits and getting experiments done was just the result of persistence. The initial reaction usually wasn’t, “Hey, this is a great idea.” One of the largest barriers is the consultants themselves, who in many cases are afraid of knowledge. It is an industry that has lived off hype and hyperbole for decades. All of a sudden people are measuring, and that makes folks nervous. On the Democratic side, I think that resistance has pretty much been worn down by now.

Beach: I think even when you get consultants to the point where they’re willing to make a change things tend to tighten up right when it comes to decision time. Even if you have someone willing to be proven wrong, they still end up going with what’s safe in their mind. We’ve seen that a lot on the consultant side. You actually succeed in convincing them it’s something they should do, everyone’s agreed and then right when the rubber meets the road, you go right back to the safe old way. That’s probably harming the operation more than helping.

Malchow: The organizations that are open to innovation are the ones that are highly accountable, where results are easily measured. Think of it along a big continuum. The most innovative is direct marketing because every single thing you do is measured in some way. No one ever goes into the conference room to argue about whether something worked or not. Political campaigns are at the other end of the spectrum. You could run the best campaign in the world and lose. You could run the worst campaign in the world and win. The manager is sitting there and he knows he’s going to be praised if they win and blamed if they lose. But if he does something different, then he’s really going to be blamed. I think that was always a barrier with campaigns. Less so today, but we still operate in an innovation-averse business.

C&E: How much is that lessening on the Republican side?

Gimpel: In some sense I think the business orientation of the Republican side has helped. They want to know if their money is being spent wisely. A lot of the resistance is still there, but some donors are demanding this now. Because many of them are aware of experimentation, Republican donors are saying, “We’ll put up this money, but we want to know what we’re getting.” This is especially true in the 527 community.

Malchow: Alex Gage once told me a story that really illustrates this well. It was 2002, and he had just opened his firm to do modeling and use advanced statistics to determine which voters should be targets. He goes up to pitch Mitt Romney’s gubernatorial campaign, and his whole business concept is on the line here. He’s got to get a big client or this thing is not going to work. He goes into this impassioned speech about how all these techniques are so important and that it’s a new way of looking at things. The whole time his audience is just looking at him in this perplexed manner. He was sure he wasn’t going to be successful with the pitch. He finishes talking and one guy raises his hand and asks, “You mean to tell me they don’t do these things in politics?” (laughter)

Green: When we got going with Gov. Perry in 2005, it was a good example of what experimentation can do if the conditions are right. If you have a campaign manager like Dave Carney, who has a long term view, this can be very effective. He knows that he’s going to be back on the same political turf again and again and therefore wants to know what’s going to work for the long haul. Carney himself was interested in whether his subcontractors were spending his money wisely. So it was just a very nice set of conditions.

Dave was the one who asked, “Can we do something with respect to fundraising? Can we do something with robocalls and direct mail?” The best experiments are done with some practice runs. The nice thing about 2005 was that we did a lot of practice runs. We did two or three rounds of fundraising experiments and messaging experiments. That allowed us to get ready for the big dance, which was the radio and TV experiment.

C&E: Is that the best way to get campaigns to buy in? If you have the manager on board, is that enough?

Green: I think that’s the main thing. In that race, Dave was the one who deployed all the resources. That gave us the opportunity to run a very complicated yet informative mass media experiment. I’m not sure whether this has caught on, but this is the first TV experiment that I know of in a large market. The fact that we had Dave Carney saying to media buyers, “You’re going to roll out this media campaign in 18 experimental markets over a series of weeks”—that’s a unique opportunity. We were able to tell the campaign how many GRPs to spend, and then we could randomize when and where. On the back end, we had their survey operation to track results. Everybody was on the same page, and the only way you get that is for the manager to be directing the show.

Gimpel: You would need to ask Dave about this, but I think he had some salesmanship to do internally. There were some vendors who were perhaps a bit threatened and not so enthusiastic about being audited in this way.

Malchow: I remember pitching this stuff to Mark Penn back in 1996. He was advising Bill Clinton, and he saw the value in this. The problem was that the political wrestling that he would have had to do to make it work just wasn’t worth it for him.

Beach: When you’re fighting your political director, your finance director and all of your other consultants, you’ve got to really be a strong chief executive to push that across the whole board. That’s why you see it at the committee level more than you do in campaigns because they at least have long term leadership.

C&E: Given that unfettered ability that you had with the Perry campaign, what was the most interesting finding for you?

Green: For me, it was the impact of TV ads. The most interesting finding was that these ads had an immediate impact that was really quite substantial. They moved the dial noticeably in terms of public opinion, but then a week later the impact had subsided. People might have imagined that was true, but they didn’t know it. I also wonder whether it’s true for negative campaigns, or campaigns down the stretch run of the election. There are all kinds of things that people might want to know that this one experiment doesn’t answer. This is, so far, the only hard experimental data on the question. If it is true that in general the effects of TV advertising subside within a week or two, then that dramatically changes the calculation in terms of strategy.

Gimpel: We learned some things on the fundraising front, too. For small donor fundraising we learned that the type of message doesn’t matter much; it’s just that you contact a certain microtargeted group of donors. The fact that you make the contact is more important than what message you use.

Malchow: The whole issue of measuring and understanding persuasion is so much more complex than GOTV. At the Analyst Institute, I’m sure we have 300 experiments that have been done. Some of them are almost timeless. You can move them from one campaign to the other and they work. However, what we’re learning about persuasion, particularly persuasion mail, is that messaging can make a big difference.

I wrote a book in 2003, and the most fundamental assumption in this book was that undecided and independent voters were the primary targets of persuasion activity. In fact, we found that it varies widely from one campaign to another. This led to a new approach to persuasion mail where we mail in August. We might test one treatment group or we might test three treatment groups. We measure them against the control, and then you can come back to not only pick the creative treatment that works the best but also to model the movement so that you have a greater impact when you roll out.

Gimpel: I think we’re also going to see a wave of interesting experiments come out of this cycle on digital media. For instance, advertising against the login screens of people’s emails and some pre- and post-testing of persuasion. The Democrats have also done some very interesting things related to size. They have been sending out these really huge placard-sized mailers because that’s something that really grabs attention.

Malchow: The most interesting result we have gotten came from an experiment that I did in 2010 for Women’s Voices, Women Vote. They have actually made testing the foundation of their work. I can bring them something completely off the wall and they are willing to test it. We had some experience testing voter registration, and we learned that it made a huge difference if you just strip out all of the political rhetoric and create a piece that was black and white and looked like it came from the government.

We noticed that focus groups would reject a piece of mail because it looked like advertising. So we did two pieces: a traditional postcard and one that was just a sheet of paper lasered on both sides. No photographs, no colors, nothing. The letter basically presented a comparative statement, not a true negative, about the two candidates. The letter not only moved the needle 11 points over the control, but the recall on the letter was almost 80 percent higher than the recall on the two postcards. It was a very extraordinary result. Now, you can’t revamp all of persuasion mail on the basis of one test, but it is very suggestive that there is a window of opportunity in getting away from the graphics and the advertising look.

C&E: What’s happening with all of this on the digital side?

Beach: I think we’re looking at questions as basic as the percent of media spent online versus offline. In our world, this is still something being debated. Even the most aggressive campaigns probably aren’t going to spend a share of their paid media budget based on the ratio that people are actually getting most of their content online versus broadcast. But I think it’s a good test for experimentation because we do control so much of the process under one roof. We’re also doing these things every day—a fundraising experiment every day, a voter contact program every day. That’s one area where you could really see an influx of academic help.

Gimpel: You could do so much more online than you can do over the phone. The other thing that’s really important is not just getting people’s judgment about message, but seeing whether we can actually get people to behave in a certain way. And that’s what online does—it allows you to see if someone will actually click or behave in a certain way.

Green: I think this challenge of measuring the outcomes we most care about is a real one, and it’s not just real for experimental science. Too often, what people stop at is the immediate survey response when what we often care about is a behavior that is going to be lived out days or weeks later. Even though I’m proud of the Texans for Rick Perry experiment, at the end of the day it was still about survey responses. The next generation of experiments will take advantage of ready access to a wide array of different geographic media markets and maybe even individual level media markets with the ability to increasingly target messages to individuals. The ultimate experiment will measure voting choice—voting behavior at the precinct level or the individual level through a post-election survey.

Gimpel: There are other elements of behavior that we care a lot about. Getting people to sign up to volunteer or to make a small donation, for example. This is the kind of thing that we might be able to test online.

Green: Jim’s absolutely right. As resistant as campaign consultants are to GOTV experiments because the results won’t be known until the show is over, you would think they would be very eager to do organizational type experiments. Those results, in terms of how many volunteers you got or how many people actually showed up to a rally, are known in real time. If you had a far-flung campaign over a variety of battleground states, you’d be able to test fairly quickly to see whether your mode of getting the word out is actually generating bodies at events. C&E: From a practitioner perspective, is this a generational thing? Are there younger operatives now who are less averse to experimentation?

Beach: As somebody who came into the business after “Get Out the Vote,” I think that book really had a big influence on us. You saw it in the Bush-Cheney operation in 2004. The same people are now asking the question, and have been asking the question for a year now, what’s next? What are we doing that’s incorrect? It’s happening even with something as basic as measuring everything back to the marginal cost. That’s a basic thing in the business world, but it’s something we don’t do in politics. We send direct mail out to recruit a volunteer, but then we don’t go back to figure out how many volunteers we got from that $30,000 mailer. Every time I’ve ever done one of those things and then taken a nonscientific poll, asking, “How much do you think this going to cost,” everyone usually guesses about five percent of what the actual cost was.

Malchow: This was a barrier in getting people to test GOTV tactics at first because they would say, “I’m doing phone calls or sending mail, and it’s costing me $2 or $3 to get a vote.” When the returns came in, you would find out that it was actually costing $15 or $20 at best. Then you would go and tell people what worked best and tell them it only cost $20. A lot of the practitioners were horrified.

Green: I think of this as the trail-of-dollar-bills test. How much more effective is it than if I just leave a trail of dollar bills from the voter’s house to the polling place? Very often when you’re talking about $150 per vote, you probably would have been better off just leaving that trail of cash and making it a scavenger hunt