Boost Your Marketing ROI with Experimental Design

By using mathematical mulas to select and test a subset of combinations of vari-ables, marketers can model hundreds or even thousandsfor-of marketing messages accurately and efficiently—a

Boost Your Marketing ROI with Experimental Design

tele-In this article, consultants Eric Almquist and GordonWyner explain that while marketing has always been acreative endeavor, adopting a scientific approach to itmay actually make it easier—and more cost effective—forcompanies to target the right customers “Experimentaldesign” techniques, which have long been applied inother fields, let people project the impact of many stimuli

143

by testing just a few of them By using mathematical mulas to select and test a subset of combinations of vari-ables, marketers can model hundreds or even thousands

for-of marketing messages accurately and efficiently—andthey can adjust their messages accordingly

The authors use a fictional company, Biz Ware, todescribe how companies can map out an a grid a com-

bination of the attributes (or variables) of a marketing message and the levels (or variations) of those attributes.

Marketers can test a few combinations of those attributesand levels and can apply logistic regression analysis toextrapolate the probable customer responses to all thepossible combinations The company can then analyzethe experiment’s implications for its resources, revenues,and profitability The authors also present the results oftheir work with Crayola, in which they used experimentaldesign techniques to test that company’s e-mail market-ing campaign

C   daily with hundreds,perhaps thousands, of marketing messages Deliveredthrough all manner of media, from television commer-cials to telephone solicitations to supermarket circulars

to Internet banner ads, these stimuli may elicit thedesired response: The consumer clips a coupon, clicks on

a link, or adds a product to a shopping cart But the vastmajority of marketing messages fail to hit their targets.Obviously, it would be valuable for companies to be able

to anticipate which stimuli would prompt a responsesince even a small improvement in the browse-to-buyconversion rate can have a big impact on profitability.But it has been very difficult to isolate what drives con-

sumer behavior, largely because there are so many ble combinations of stimuli.

possi-Now, however, marketers have easier access, at tively low cost, to experimental design techniques longapplied in other fields such as pharmaceutical research.Experimental design, which quantifies the effects ofindependent stimuli on behavioral responses, can helpmarketing executives analyze how the various compo-nents of a marketing campaign influence consumerbehavior This approach is much more precise and costeffective than traditional market testing And when youknow how customers will respond to what you have tooffer, you can target marketing programs directly to theirneeds—and boost the bottom line in the process

rela-Traditional Testing

The practice of testing various forms of a marketing oradvertising stimulus isn’t new Direct marketers, in par-ticular, have long used simple techniques such as splitmailings to compare how customers react to differentprices or promotional offers But if they try to evaluatemore than just a couple of campaign alternatives, tradi-tional testing techniques quickly grow prohibitivelyexpensive

Consider the “test and control cell” method, which isthe basis for almost all direct mail and e-commercetesting done today It starts with a control cell for, say,

a base price, then adds test cells for higher and lowerprices To test five price points, six promotions, fourbanner ad colors, and three ad placements, you’d need

a control cell and 360 test cells (5 × 6 × 4 × 3 = 360).And that’s a relatively simple case In credit card mar-keting, the possible combinations of brands, cobrands,

annual percentage rates, teaser rates, marketing sages, and mail packaging can quickly add up to hun-dreds of thousands of possible bundles of attributes.Clearly, you cannot test them all.

mes-There’s another problem with this brute forceapproach: It typically does not reveal which individualvariables are causing higher (or lower) responses fromcustomers, since most control-cell tests reflect the com-bined effect of more than two simple variables Is it thelower price that prompted the higher response? The pro-motional deal? The new advertising message? There’s noway to know

The problem has been magnified recently as nies have gained the ability to change their marketingstimuli much more quickly Just a few years ago, chang-ing prices and promotions on a few cans of food in thesupermarket, for example, required the time-consumingapplication of sticky labels and the distribution of papercoupons Today, a store can adjust prices and promo-tions electronically by simply reprogramming its check-out scanners The Internet has further heightened mar-keting complexity by reducing the physical constraints

compa-on pricing, packaging, and communicaticompa-ons In theextreme, an on-line retailer could change the prices andpromotion of every product it offers every minute of theday It can also change the color of banner ads, the tone

of promotional messages, and the content of outbounde-mails with relative ease

The increasing complexity of the stimulus-response

network, as we call it, means that marketers have more

communication alternatives than ever before—and thatthe portion of alternatives they actually test is growingeven smaller But this greater complexity can also mean

greater flexibility in your marketing programs—if youcan uncover which changes in the stimulus-responsenetwork actually drive customer behavior One way to dothis is through scientific experimentation.

A New Marketing Science

The science of experimental design lets people projectthe impact of many stimuli by testing just a few of them

By using mathematical formulas to select and test a set of combinations of variables that represent the com-plexity of all the original variables, marketers can modelhundreds or even thousands of stimuli accurately andefficiently

sub-This is not the same thing as an after-the-fact analysis

of consumer behavior, sometimes referred to as data ing Experimental design is distinguished by the fact thatyou define and control the independent variables beforeputting them into the marketplace, trying out differentkinds of stimuli on customers rather than observing them

min-as they have naturally occurred Because you control theintroduction of stimuli, you can establish that differences

in response can be attributed to the stimulus in question,such as packaging or color of a product, and not to otherfactors, such as limited availability of the product Inother words, experimental design reveals whether vari-

ables caused a certain behavior as opposed to simply being correlated with the behavior.

While experimental design itself isn’t new, few keting executives have used the technique—eitherbecause they haven’t understood it or because day-to-day marketing operations have gotten in the way Butnew technologies are making experimental design more

mar-accessible, more affordable, and easier to administer.(For more information on the genesis of this type oftesting, see “The Origins of Experimental Design” at theend of this article) Companies today can collect detailedcustomer information much more easily than ever beforeand can use those data to build models that predict cus-tomer response with greater speed and accuracy.

Today’s most popular experimental-design methodscan be adapted and customized using guidelines from

standard reference textbooks such as Statistics for

Experi-menters by George E P Box, J Stuart Hunter, and William

G Hunter; and from off-the-shelf software packages such

as the Statistical Analysis System, the primary product

of SAS Institute A handful of companies have alreadyapplied some form of experimental design to marketing.They include financial firms such as Chase, HouseholdFinance, and Capital One, telecommunications providerCable & Wireless, and Internet portal America Online.Applying experimental-design methods requires busi-ness judgment and a degree of mathematical and statis-tical sophistication—both of which are well within thereach of most large corporations and many smaller orga-nizations The experimental design technique is particu-larly useful for companies that have large numbers ofcustomers and that face rapid and constant change intheir markets and product offers Internet retailers, forinstance, benefit greatly from experimentation becauseon-line customers tend to be fickle Attracting browsers

to a Web site and then converting them into buyers hasproved very expensive and largely ineffective Getting itright the first time is nearly impossible, so experimenta-tion is critical The rigorous and robust nature of experi-mental design, combined with the increasing challenges

of marketing to oversaturated consumers, will make

widespread adoption of this new marketing science only

a matter of time in most industries

The ABCs of Experimental Design

To illustrate how experimental design works, let’s sider the following simple case A company, which we’llcall Biz Ware, is marketing a software product to othercompanies Before launching a national campaign, Biz

con-Ware wants to test three different variables, or attributes,

of a sales message for the product: price, message, andpromotion Each of the three attributes can have a num-

ber of variations, or levels Suppose the three attributes

and their various levels are as shown in “Attributes andLevels of a Sales Message.”

The total number of possible combinations can bedetermined by multiplying the number of levels of eachattribute The three attributes Biz Ware wants to testyield a total of 16 possible combinations since 4 × 2 × 2 =

16 All 16 combinations can be mapped in the cells of

a simple chart like “Biz Ware’s Universe of PossibleCombinations.”

It’s not necessary to test them all Instead, using

what’s called a fractional factorial design, Biz Ware

selects a subset of eight combinations to test

“Factorial” means Biz Ware “crosses” each attribute(price, promotion, and message) with each of the others

in gridlike fashion, as in the universe chart above tional” means Biz Ware then chooses a subset of thosecombinations in which the attributes are independent(either totally or partially) of each other The followingchart shows the resulting experimental design, with Xsmarking the cells to be tested Note that each level ofeach attribute is paired in at least one instance with each

“Frac-level of the other attributes So, for example, price at

$150 is matched at some point with each promotion andeach message This makes it possible to unambiguouslyseparate the influence of each variable on customerresponse (See “Biz Ware’s Experimental Design.”)The eight chosen combinations are now tested, usingone of several media: scripts at a call center, banner ads

on Biz Ware’s Web site, e-mail messages to prospectivecustomers, and direct mail solicitations (In general, youshould test using the medium you ultimately expect touse for your marketing campaign, although you can alsochoose multiple media and treat the choice of media as

an attribute in the experiment.)

Attributes and Levels of a Sales Message

PRICE Level

$150 $160 $170 $180 MESSAGE

How big should the sample size be to make the iment valid? The answer depends on several characteris-tics of the test and the target market These may includethe expected response rate, based on the results of pastmarketing efforts; the expected variation among sub-groups of the market; and the complexity of the design,including the number of attributes and levels In anyevent, the sample size should be large enough so thatmarketers can statistically detect the impact of theattributes on customer response Since increasing thecomplexity and size of an experiment generally addscost, marketers should determine the minimum samplesize necessary to achieve a degree of precision that is

exper-Biz Ware’s Universe of Possible Combinations

Promotion Message Price (1) Price (2) Price (3) Price (4)

(1) (1) (2) (2) (1) (2) (1) (2)

(1) (1) (2) (2) (1) (2) (1) (2)

useful for making business decisions (There are dard guidelines in statistics that can help marketersanswer the question of sample size.) We’ve conductedcomplex experiments by sending e-mail solicitations tolists of just 20,000 names, where 1,250 people eachreceive one of 16 stimuli.

stan-Within a few days or weeks, the experiment’s resultscome in Biz Ware’s marketers note the number andpercentage of positive responses to each of the eighttested offers (See “Biz Ware’s Design Results.”)

At a glance, you might intuitively understand thatprice has a significant impact on the response to the var-ious offers, since the lower price offers (Price 1 and Price2) generally drew much better response rates than thehigher price offers (Price 3 and Price 4) But statisticalmodeling, using standard software, makes it possible toassess the impact of each variable with far greater preci-

sion Indeed, by using a method known as logistic

regres-sion analysis, Biz Ware can extrapolate from the results

of the experiment the probable response rates for all 16cells (See “Biz Ware’s Modeled Responses.”)

Note that the percentages shown below don’t cisely match the original percentages from the test

pre-Biz Ware’s Design Results

Promotion Message Price (1) Price (2) Price (3) Price (4)

(1) (1) (2) (2) (1) (2) (1) (2)

9% 13%

6% 10%

That’s because Biz Ware used the original percentages tocreate a general equation for estimating the results in allthe cells When the new equation is then applied to thecells already tested, the results usually vary somewhatfrom the original numbers The important thing is thatthe tester ends up with a full set of consistent results forall possible permutations (For more about how thesecalculations were made, see “Estimating a ResponseModel” at the end of this article.)

With this complete picture, it becomes clear thatsome combinations are far more likely to be effectivethan others The combination of Price 1 ($150), Message

2 (Power), and Promotion 2 (Free Gift) is clearly themost attractive to consumers But is it the right combi-nation for Biz Ware? That’s where business judgmentcomes in: The company’s management will need to ana-lyze the experiment’s implications for its resources, rev-enue, and profitability

(1) (1) (2) (2) (1) (2) (1) (2) 14% 23% 28% 42%

Crayola, a division of Binney & Smith and Hallmark,launched a creative arts and activities portal on theInternet called Crayola.com The site’s target customersinclude parents and educators, and it sells art suppliesand offers arts-and-crafts project ideas and classroomlesson plans We conducted an experimental design tohelp Crayola attract people to the site and convertbrowsers into buyers.

Based on Crayola’s experience and market knowledge,

we identified a set of stimuli that could be varied to drivetraffic to Crayola.com and induce purchases One ofthese stimuli was an e-mail to parents and teachers Totest various components of the e-mail content and for-mat, we relied on the best judgment of Crayola’s market-ing staff about the messages that were most likely toappeal to the target markets The e-mail included fivekey attributes that seemed likely to affect the customerresponse rate, which would be measured by click-throughs to the Crayola Web site These attributes andtheir related levels were:

• Two subject lines: “Crayola.com Survey” and “Help

Us Help You.”

• Three salutations: “Hi [user name] :-),” “Greetings!”

and “[user name].”

• Two calls to action: “As Crayola.com grows, we’d like

to get your thoughts about the arts and how you useart materials” and “Because you as an educator have aspecial understanding of the arts and how art materi-als are used, we invite you to help build Crayola.com.”

• Three promotions: “a chance to participate in a

monthly drawing to win $100 worth of Crayola ucts; a monthly drawing for one of ten $25

prod-Amazon.com gift certificates; and no promotion