Multivariable testing is a statistical tool that can gauge the impact of changes on processes from oil refining to retail sales, and it’s the most powerful tool for gaining knowledge I’ve ever known. But I can’t get most political consultants to believe that. In fact, most don’t even return my calls.
You can accuse me of hyperbole, but MVT works because it’s driven by data. I realize that scares some people, particularly those who have a vested interest in the sanctity of their expertise. Still, the numbers don’t lie.
We’ve done some 250,000 experiments with more than 1,000 organizations and have found a nearly inviolable pattern: fully half of suggested changes have no impact, 25 percent have a positive impact, and 25 percent have a negative impact. The constant: no one can consistently guess which changes are going to be in which group.
I founded my company—QualPro— after many years as a statistician and division quality manager at the National Nuclear Weapons Complex in Oak Ridge, Tennessee. As a statistician, I rely on complex mathematics to quickly and efficiently solve problems that appear intractable. The term multivariable testing was first coined by Forbes magazine, which has studied and reported on it.
MVT uses carefully designed statistical experiments to achieve results from small, quick tests involving many variables. We can then take apart the results analytically and isolate what variable or combination of variables caused the effects. The result is a powerful and efficient way to test potential improvements to complex processes and then learn which changes have the most impact.
Despite the potential widespread benefits of the process, the obstacle is that many experts don’t want to find out that what they thought they knew is actually wrong, nor do they want to think a mathematical process can replace their expertise. But we use all sorts of tools to hone our understanding and MVT is one that political consultants should embrace. Like scientists who come up with ideas, test them and then revise their theories, consultants should be more effectively testing their own ideas to achieve more consistent outcomes for their clients.
It’s why I jumped at the chance to work on a political campaign that appeared to be a lost cause: Tennessee Republican John Ragan’s 2010 bid for state House.
A Campaign Approach
In Ragan’s case, he found himself facing a well-funded Democratic incumbent in state Rep. Jim Hackworth. He was running uphill in a traditionally Democratic district and he was losing. Ragan was polling at just 32 percent, and he didn’t have the time or money to turn his campaign around. So Ragan decided to send out one direct mail piece to voters in his district, and it had to be good. He refined his message line by line based on the results of a QualPro-led experiment.
Though I have a strong interest in politics, my opinions and biases were not important in this case, nor are they in the application of our process in any other case. The MVT Process scientifically determines what content catalyzes a respondent’s reaction. It doesn’t say why something works or even whether it should, only that it does. Our tests generate information about how people respond to various words, images and layouts at a particular time.
In Ragan’s case, we were looking at two different mailer formats and testing the display of various pieces of information, including party affiliation and logos; descriptions of Ragan’s background, values, and legislative priorities; quotes and endorsements from other politicians, and selected facts about Tennessee state politics. The mailers weren’t flashy and they weren’t going to be. It meant that finding the optimal message would be key.
We decided to test 15 different variables: the type of card, the use of a follow-up phone call or visit, and 12 variations in the content and look of the mailing. A randomly selected group of 320 likely voters were shown the different variations of the mail pieces.
We lay out these variables in mathematically determined combinations to the right number of subjects. This allows our statisticians to reliably estimate the individual impact of each variable and the impact of different combinations of variables. Thirty-two different versions of the postcards were mailed to the 320 subjects. Telephone polls by a professional survey firm—before and after the mailings—measured the impact of the mailings on the likelihood that recipients would vote for Ragan.
The results on voter intentions as reported by the experiment’s subjects were dramatic even as some changes were barely noticeable to a casual observer. For instance, one variable compared a series of “Did You Know?” statements about illegal aliens to a list of Ragan’s attributes, including experience and responsibility. The use of that section seemed to focus voter anger and increase their inclination to vote for Ragan by a few percentage points.