The essentials of AB and multivariate testing

AB testing software is then used to randomly splithe homepage visitors between the controland the new challenger.So,50,000 vitors are directed to the controland 50,000 vitors are directe

A B

THE DEFI NI TI VE GUI DE TO

www i nvespcro com

I nt r oduct i on

What s A/B t est i ng ( spl i est i ng) ?

What s Mul t vari at e Test i ng?

The Di f er ence Bet ween A/B Test i ng and Mul t vari at e Test i ng

Compari ng Mul t vari at e & AB Test i ng Tool s ( updat ed f or 2018)

The Pr ocess of Cr eat i ng a Successf ul A/B Test

What El ement s Shoul d

What El ement s Shoul d You I ncl ude i n an A/B Test ?

Mul t vari at e and A/B Test i ng Best Pract i ces

16 mi st akes t hat wi l ki l your A/B t est i ng and what you can do about hem)

01 03 11 18 23 29 34 54 61

I NTRODUCTI ON

Most websi t es cont i nue t o st ruggl e wi t h i ncr easi ng t hei r onl i ne conversi on rat es.

On average, an e-commer ce st or e convert s ar ound 3% of s t raf fic i nt o cust omers The r emai ni ng 97%

of he vi si t ors l eave t he websi t e wi t hout ever pl aci ng an or der

Ot her t ypes of websi t es st ruggl e wi t h conversi on as wel l On average, a l ead generat i on or subscri pt i on websi t es convert ar ound 15% of s t raf fic i nt o cust omers The r emai ni ng 85% l eave t he websi t e wi t hout ever pl aci ng an or der

I

I you ar e l ooki ng t o i ncr ease your websi t e conversi on rat e, you wi l have t o conduct AB and mul t vari at e

t est i ng The pr obl em i s t hat most est i ng pr ograms st art wi t hout any pl anni ng and t hus f ai l wi t h t he first

6 mont hs.

We know t hi s first hand because we have been hel pi ng onl i ne compani es conduct est i ng f or over 12 years Our goal r om t hi s gui de t o hel p you wi t h a good i nt r oduct i on t o mul t vari at e and AB t est i ng We

wi l cont i nue updat i ng t he di f er ent chapt ers i n t he book on r egul ar basi s, so i you have comment s or

f eedback, pl ease l et ’ s know ( @i nvesp)

01

THE DEFINITIVE GUIDE TOMULTIVARIATE AND AB TESTING

WHAT I S A/B TESTI NG ( SPLI T TESTI NG) ?

WRI TTEN BY: KHALI D SALEH

WHAT I S A/B TESTI NG ( SPLI T TESTI NG) ?

C H A P T E R 1

Ivisitors are notconverting on your website,hen obviously,here is a problem thats stopping them

You can go ahead and ask your design team to create new designs,buthe question remains:how do you know thathe new designs wilconvertmore vitors compared to the originaldesign?

Thats where AB testing comes in handy.A/B testing (sometimes referred to as splitesting)s the process ofesting multple new designs

ofa webpage againsthe originaldesign ofhatpage with the goalofdetermining which design generates more conversions

The originaldesign ofa page is usually

The originaldesign ofa page is usually referred to as the control.The new designs ofhe page are usually referred to as the “variations”,

“challengers” or “recipes.” The process ofesting which page design generates more conversions is typically referred to as a “test” or an

“experiment

A “conversion” wilvary based on your website and the page you are testing.For an e-commerce website,a conversion could be a visitor

placing an order.For a SaaS website,a conversion could be a vitor subscribing to the service.For a lead generation website,a conversion could be a vitor fillng outa contactorm

The homepage on an e-commerce website receives 100,000 visitors a month.To determine ithere is a way to increase

conversions,he design team creates one new design for the homepage

AB testing software is then used to randomly splithe homepage visitors between the controland the new challenger.So,50,000 vitors are directed to the controland 50,000 vitors are directed to the challenger.Since we are testing which design generates more orders (conversions),hen we use the AB testing software to track the number ofconversions each design generates.The A/B testing software wilthen determine the winning design based on the number ofconversions

1ST EXAMPLE

03

THE DEFINITIVE GUIDE TOMULTIVARIATE AND AB TESTING

The homepage for a blog receives 3,000 visitors a month.The primary conversion goalor the homepage is to geta visitor to

subscribe to the emailistofhe blog.The designer creates a new design for the blog homepage which highlights the

subscription box

The splitesting software is used to send 1,500 visitors to the originalpage design (control)and the testing software sends

1,500 visitors to the new design (challenger).The testing software tracks the number ofsubscribers (conversions)each design generates

A 2015 survey by E-consultancy showed that58% ofs respondents are conducting A/B testing:

Buthow successfuls AB testing in helping companies increase their conversion rates?

A 2017 survey by Optimizely shows thatonly 25% ofallA/B tests produce significantly positve results.VisualWebsite Optimizer reports

thatonly 12% ofallA/B testproduce significantly positve results.Finally,data fom Google shows thatonly 10% ofallA/B testproduce

significantly positve results

2ND EXAMPLE

04

THE DEFINITIVE GUIDE TOMULTIVARIATE AND AB TESTING

Background

46% ofsurveyed Netfx visitors complained thathe website does notallow them to view movie tles before signing up for the service.So,Netfx decided to run an A/B teston their registration process to see ia redesigned registration process wilhelp increase subscriptions

Creati ng an AB test

The new design displ

The new design displayed movie tles to visitors prior to registration.The Netfx team wanted to find out he new design with movie tles would generate more registrations compared to the originaldesign withouthe tles.This was analyzed by running an A/B testbetween

the new designs againsthe originaldesign

The testhypothesis was straightforward:Allowing visitors to view available movie tles before registering wilncrease the number ofnew

signups

In the splitest,he team introduced five diferentchallengers againsthe originaldesign.The team then ran the testo see the impact.Whatwere the results?

Resul ts of the AB test and anal ysi s

Resul ts of the AB test and anal ysi s

The originaldesign consistently beatallchallengers.The realanalysis happens after you conclude an A/B test.Why did the originaldesign beatallnew designs although 46% ofvisitors said thatseeing whattes Netfx carries wilpersuade them to sign up for the service?The team atNetfx gave three diferenteasons ofwhy the originaldesign beatallhe challengers:

1 Netfl i x i s al l about the experi ence:he more users interacted with the website,he more they love the experience.So,Netfx is more

than justbrowsing

2.

2 Si mpl i fy choi ce:he originaldesign (he control)showed users one option:sign up for the service.The new designs offered visitors multple options (multple movies).This complicated the choice which vitors had to make.More choices resulted in fewer conversions

3 Users do not al ways know what they want:The Netfx team argued thatestesults pointo the facthatusers do notalways know whathey want

Case Study: What Netfl i x l earned from AB testi ng?

05

THE DEFINITIVE GUIDE TOMULTIVARIATE AND AB TESTING

While these mightbe valid explanations,especially the second point,we would argue thathere is another reason altogether.

Could ibe thatvisitors finally see allhe movie options which Netfx offers and they do notfind the movie selection convincing,so they decide to walk away? Ithats the case,s the problem with the new designs or is ia problem in the movie selection which the site offers?

Ats core,AB testing software tracks the number ofvisitors coming to each design in an experimentand the number ofconversions each design generates.Sophisticated A/B testing software tracks much more data for each variation.As an example,FigPiiracks:

The splitesting software uses diferentstatisticalmodes to determine a winner in a test.The two popular methods for determining a

winner are Frequentistand Bayesian models

The splitesting software tracks conversion rates for each design.However,declaring a winner in a splitestequires more than generating

a smallncrease in conversion rates compared to the control

The Frequentistmodeluses two main factors to determine the winning design:

The conversion rate for each design:his number is determined by dividing the number ofconversions for a design by the unique

visitors for thatdesign

The confidence levelor each design:a statisticalerm indicating the certainty thatyour testhatwilproduce the same resultfhe same experiments conducted across many separate data sets in diferentexperiments

Think ofconfidence l

Think ofconfidence levelas the probabily ofhaving a resultSo,a challenger produces a 20% increase in conversions with a 95% confidence,hen you assume thatyou have an excellentprobabily ofgettng the same resultwhen selecting thatchallenger as your defaultdesign.also indicates thatyou have a 5% chance thatyour testesults were due to random chance,and a 5% possibily thatyou

found a wrong winner

The Bayesian modeluses two main factors to determine the winning design:

The conversion rate for each design:as defined above

Historicalperformance:he success rate ofpreviously ran A/B experiments ran on the webpage

Conversi ons

Page vi ews

Vi si tors Bounce rate Exi t

Revenue Revenue per vi si t

Leonid Pekelis,Optimizely’s firstn-house statistician,explains this by saying

We typically rely on multple metrics when determining a winning design for a test.Mostofour e-commerce clients use a combination ofconversion rates and revenue per visito determine a finalwinner in an experiment

Selecting which metrics wildepend on your specific situation.However, s crucialo choose metrics thathave an impacton your bottom

lne.Optimizing for a lower bounce or exitrates wilhave lte directand measurable dollar value to mostbusinesses

The team atBing was t

The team atBing was trying to find a way to increase the revenue which the site generates fom ads.To do so,hey introduced a new design thatemphasized how search ads are displayed.The team tested the new design vs.he old design.The splitestesults showed a 30% increase in revenue per vit

This,however,was due to a bug in their main search results algorithm in the new design.This bug showed vitors poor search results.And

as a resultvitors were frustrated and were clicking on ads

While the new design generated a higher revenue per vitthis was nota good long-term strategy.The team decided to stick to the old design instead

MostAB testing software automatically divides vitors equally between diferentvariations

There are however instances where you need to assign diferentweights to diferentvariations

For example,et’s take an experimenthathas an originaldesign and two challengers in iThe testing team mightwanto assign 50% of

the vitors to the originaldesign and splithe remaining 50% between variations one and two

Some Conversion optimization experts debate this question atgreatengths

Looking atyour analytics,you can typically notice thatdiferentvisitor segments interactdiferently with your website.Returning vitors (hose who visited site previously)generally are more engaged with the website compared to new vitors

Assi gni ng wei ghted traffic to di ff erent vari ati ons

Shoul d you run AB testi ng on 100% of your vi si tors?

07

THE DEFINITIVE GUIDE TOMULTIVARIATE AND AB TESTING

Bayesi an st at i st i cs t ake a more bot t om-up approach t o dat a anal ysi s Thi s

means t hat past knowl edge of si mi l ar experi ment s i s encoded i nt o a st at i st i cal

devi ce known as a pri or , and t hi s pri or i s combi ned wi t h current experi ment

dat a t o make a concl usi on on t he t est at hand.

“ Bayesi an st at i st i cs t ake a more bot t om-up approach t o dat a anal ysi s Thi s

means t hat past knowl edge of si mi l ar experi ment s i s encoded i nt o a st at i st i cal

devi ce known as a pri or , and t hi s pri or i s combi ned wi t h current experi ment

dat a t o make a concl usi on on t he t est at hand.

“

Bayesi an st at i st i cs t ake a more bot t om-up approach t o dat a anal ysi s Thi s

means t hat past knowl edge of si mi l ar experi ment s i s encoded i nt o a st at i st i cal

devi ce known as a pri or , and t hi s pri or i s combi ned wi t h current experi ment

dat a t o make a concl usi on on t he t est at hand.

When launching a new AB test,you wilnotice thatn many instances:

New vitors reactn a better way with your experimentchallengers

Returning vitors,who are used to your currentdesign,eactnegatively to your new designs

The facthatnew visitors convertathigher rates with new designs compared to returning visitors is attributed to the theory of momentum behavior

Iyou your website gets a large number ofvisitors,we recommend thatyou launch new tests for only new visitors and observing how they

reacto iAfter that,you can starthe testor returning visitors and compare their reactions to the new designs introduced in the

experiment

We typically recommend running holdback splitests for larger websites thateceive thousands ofconversions per month.n these types

ofests,you launch the tests to a smallpercentage ofyour site vitors.For example,you startwith launching the testo 10% ofyour vitors he results are encouraging,hen you expand the testo 25%,50%,and 100% ofyour website visitors

There are severaladvantages to running hold back A/B tests:

Discover any testing bugs:As you launch an AB test,your designs mighthave bugs in them.By running the teston a smallpercentage

ofyour visitors,only thatny segmentofhe visitors wilsee the errors in the new designs.Thatwilgive you the opportunity to fix these bugs before rollng outhe testo 100% ofyour visitors

Reduce revenue risk:by running the teston a smallpercentage ofvisitors,you reduce the risk ofhaving one ofyours testvariation causing a significantdrop in revenue

Iyou choose to run hold back A/B tests,make sure thatyou starta new testeach tme you change the traffic allocation going through the experimento avoid any statisticalproblems with results

Hol dback spl i t tests

5%

Unti l wi nner i s decl ared

3 3

There is a lotofmath thatgoes into determining how many variations should be included in an A/B test.The following are general

guidelines you can apply,however,more details wilbe covered in a later section:

Calculate the monthly number ofconversions generated by the particular page you plan to test:

on the conservative side,divide the totalmonthly conversions generated by the page by 500 and subtractone

on the aggressive side,divide the totalmonthly conversions generated by the page by 200 and subtractone

Iyou have less than 200 conversions a month,your website is noteady for A/B testing.Focus on driving more visitors to your website

Your websi t e generat es 1, 000 conversi ons per mont h:

On t he conservat i ve si de, an A/B t est can i ncl ude one chal l enger agai nst he

ori gi nal 1000/ 500 – 1)

On t he aggressi ve si de, an A/B t est can i ncl ude f our chal l engers agai nst he

ori gi nal 1000/ 200 – 1)

Agai n, hi s i s a si mpl i ficat i on of he cal cul at i on, but wi l gi

Agai n, hi s i s a si mpl i ficat i on of he cal cul at i on, but wi l gi ve you a good

st art i ng poi nt

“ Your websi t e generat es 1, 000 conversi ons per mont h:

On t he conservat i ve si de, an A/B t est can i ncl ude one chal l enger agai nst he

WHAT I S MULTI VARI ATE TESTI NG?

WRI TTEN BY: KHALI D SALEH

WHAT I S MULTI VARI ATE TESTI NG?

C H A P T E R 2

Edi tor note:

Please note thatwe use button colors,onts,headlines as a

way to clarify the conceptofesting.SuccesfulAB and

multvariate tests wilnclude more sophisticated changes to

your page

Mult

Multvariate Testing or MVT testing is the process ofesting

multple variations ofmultple elements on a webpage with

the goalofdetermining the bestcombination ofdiferent

elements on the page to increase conversions

By using MVT testing softwa

By using MVT testing software,you can testdiferent

variations ofany elementon your page (headlines,mages,

buttons,etc.)o measure their impacton your conversion

rates.The following image displays an example ofhow MVT

testing software works

In this example,he software tests diferentvariations ofhe page headline,mage,and the “callo action” button:

The originalheadline is tested againsthree other possible headlines,or a totalofour possible headlines on the page

The originalmage is tested againstwo other possible photos,or a totalofhree possible pictures on the page

Three diferentbuttons are tested againsthe originalbutton on the page,or a totalofour possible buttons on the page

As a vitor arrives ata page,he software picks one ofhe four headlines,one ofhe three images,and one ofhe four buttons to display

11

THE DEFINITIVE GUIDE TOMULTIVARIATE AND AB TESTING

Your team does nothave to create allofhe 48 designs;he software wilswap the diferentvariations and create the designs automatically and create all48 possible variations.The following image shows four ofhe 48 possible designs the testing software can generate.

The totalnumber ofesting variations (also called challengers)depends on the number ofelements you wilteston a page (headline,

image,buttons,etc.)and the number ofvariations you wilbe testing for each ofhese elements

You can calculate the totalnumber ofchallengers in a multvariate testmultplying the number ofdiferentvariations ofeach ofhe

elements

For a webpage in which we wilbe testing (N)number ofelements,we calculate:

Totalnumber ofpage variations = Number ofvariations of1stelementx Number ofvariations ofhe 2nd elementx Number ofvariations

ofhe 3rd elementx …x Number ofvariations ofhe Nth element

The number ofpage

The number ofpage variations can grow very fast.Some testing software allows you to tens ofhousands (sometimes milons)of

variations ofa single page

12

ORI GI NAL

Ori gi nal Headl i ne Ori gi nal mage Ori gi nal But t on

Ori gi nal Headl i ne 2nd I mage

Ori gi nal But t on

Ori gi nal Headl i ne 3nd I mage

Ori gi nal But t on

2nd Headl i ne 2nd I mage Ori gi nal But t on

DESI GN 1

DESI GN 2 DESI GN 3

THE DEFINITIVE GUIDE TOMULTIVARIATE AND AB TESTING

Let’s take the product page from apple.com as an example:

On this page,you can test:

diferent variations ofthe headline

displaying two MacBook models per lne

( curr ent l y, each model akes a l ne)

diferent product images

Let’s take an example from salesforce.com:

On this page,you can test:

diferent variations ofthe headline

displaying the side navigation or not showing i

diferent hero imagesCTA colors

CTA text

How do to create a successful mul ti vari ate test?

Multvariate testing software allows marketers to create and startsimple tests in a few hours.Buthats the easy part!Many companies

ultmately failwhen designing successfulestscenarios,assessing results,and creating meaningfulollow-up tests

Poorly designed experiments can take years to conclude.Even worse,hey mightnotprovide accurate insights into whatelements convertmore visitors into customers

Imagine a case whe

Imagine a case where you plan to testdiferentheadlines on a page.You startby coming up with ten diferentpossible variations to the headlines.Which ofhese ten possible headlines should you testagainstyour originalheadline? Why notestallofhem? Why notestvariations ofmages,buttons,and layouts?

You wilmostkely find yourselfelying on guesswork to determine which versions to include in the test.The same logic,ofcourse,applies

to allelements you wanto teston a page

Withoutbeing judicious with testscenarios,you mightend up attempting to testmilons ofcombinations

14

THE DEFINITIVE GUIDE TOMULTIVARIATE AND AB TESTING

Testing is an essentialcomponentofany conversion optimization project.However, should notbe the only component.Testing should only take place after the conclusion ofother equally critcaloptimization stages,such as persona development,voice ofcustomer research (ncluding polls and surveys),heuristic evaluation,usabily testing,site analysis,and design and copy creation.Each ofhese elements

provides a building block towards a highly optimized website thatconverts visitors into customers

To create a successfulest,you mustgo through the following steps:

Evaluate the page,ooking for possible problems in i

Prioritze the issues identified on the page in terms ofheir impacton your conversion rate

Create a hypothesis ofhow to fix some ofhe top issues on the page and the effectyour fix wilhave on your conversion rate

Asserthe validity ofyour hypothesis through multvariate or AB testing

Analyze the results ofhe testo determine the correctness ofhe testhypothesis

Create a new testbased on the testesult

The resul ts from runni ng mul ti vari ate testi ng

While MVT testing is powerfuln helping online business increase conversions rates,he results you wilachieve fom running a single testmay vary

You can choose diferentapproaches to design and create your multvariate test:

ELEMENT LEVEL TESTI NG:n this type ofesting,you testdiferentvariations ofan elementon the page.For example,you test

diferentheadline variations or severalmages.The goalofan “elementevelest” is to measure the impactofhatelementon your conversion rate

Elementevelesting is considered the easiestype ofesting equires the leastamountofeffort.And in mostcases,

elementevelesting has minimum impacton your website conversion rates

P

PAGE LEVEL TESTI NG:n this type ofesting,you testmultple page elements athe same tme.As an example,you can test

diferentpage layouts,and/or a diferentcombination ofelements and so on.Page levelesting requires more effortom the developmenteam to implementand igenerates a higher impacton your conversion rates compared to elementevelesting.Carefully designed page-levelesting can produce anywhere fom 10% to 20% increase in conversion rates

VI S

VI SI TOR FLOW TESTI NG:n this type ofesting,you testseveralnavigation paths for vitors within your website.As an example,an e-commerce website mightestsingle step vs.mult-step checkout.Another example is to testdiferentways

visitors can navigate fom category pages to productpages

Visitor fow testing can getcomplicated quickly ypically requires a higher levelofeffortom your developmenteam to

implement.Done correctly,his type ofesting wilhave a higher impacton your conversion rates compared to page level

The dangers of mul ti vari ate testi ng

Iyou are notcarefulwith planning your multvariate tests,you wilend up with weak qualiy tests thatake too long to implementand

produce neither results nor insights

You mustalways remember thatesting (AB or multvariate)s only one componentofa conversion rate optimization work

We have seen many companies thatentiely relied on testing software withoutdoing an in-depth analysis ofwhathey were testing.Our

2007 article on the case againstmultvariate testing points outhis example:

When creati ng an MVT test, keep these possi bl e probl ems i n mi nd:

Be aware ofcreating the testwithoutpaying close attention to the hypothesis behind i

Be mindfulofhe number ofvariables you are testing and their dependency on one another

Be aware ofhe length ofme iwilake to complete the testo a statisticalsignificance

Let’ s do some si mpl e math.

Say you want o t est si x di f erent el ement s on a page ( headers, benefit s l st , hero

shot s, cal l o act i on, et c)

For each el ement , you wi l choose f our di f erent opt i ons Thi s means you wi l have a

t ot al of 4^6 = 4, 096 possi bl e scenari os t hat you wi l have t o t est

As a general rul e of humb [ bei ng mo

As a general rul e of humb [ bei ng more aggressi ve] , you wi l need around 200

conversi ons per scenari o t o ensure t he dat a you are col l ect i ng i s st at i st i cal l y

si gni ficant Thi s t ransl at es i nt o 4, 096 * 200= 819, 200 conversi ons.

I your websi t e convert s around 1%, you wi l need 819, 200 * 100=81, 920, 000 vi si t ors

bef ore you st art gai ni ng some confidence i n your t est resul t s.

I est i ng 4, 096

I est i ng 4, 096 vari at i ons sound di f ficul t i magi ne how compl i cat ed mat t ers wi l get

by addi ng vari at i on i n campai gns, of f ers, product s, and keywords Yes, runni ng t hat

many t est vari at i ns i s not unheard of or many l arger websi t es.

“ Let’ s do some si mpl e math.

Say you want o t est si x di f erent el ement s on a page ( headers, benefit s l st , hero

shot s, cal l o act i on, et c)

For each el ement , you wi l choose f our di f erent opt i ons Thi s means you wi l have a

t ot al of 4^6 = 4, 096 possi bl e scenari os t hat you wi l have t o t est

As a general rul e of humb [ bei ng mo “

I est i ng 4, 096 vari at i ons sound di f ficul t i magi ne how compl i cat ed mat t ers wi l get

by addi ng vari at i on i n campai gns, of f ers, product s, and keywords Yes, runni ng t hat

many t est vari at i ns i s not unheard of or many l arger websi t es.

16

THE DEFINITIVE GUIDE TOMULTIVARIATE AND AB TESTING

THE DI FFERENCE BETWEEN A/B TESTI NG

AND MULTI VARI ATE TESTI NG

WRI TTEN BY: KHALI D SALEH

THE DI FFERENCE BETWEEN A/B TESTI NG AND MULTI VARI ATE TESTI NG

C H A P T E R 3

Both A/B testing and multvariate testing share the same concept:spliing vitors between severaldesigns ofyour website or webpage to determine which ofhese designs generates more conversions

So,which ofhese two methods is bestsuited to help you increase your website conversion rate?

AB Testing (or Splitesting)allows you to testone or

more variations for each page on your website

againsteach other

Let’s take few examples:

On an e-commerce website,you can create a new

design ofyour productpage to testwhether the

productmage should be placed on the leftor the

Edi tor note:

Please note thatwe use button colors,onts,headlines as a way to clarify the conceptofesting.SuccesfulAB and multvariate tests should include more sophisticated changes to your page

There are two generalapproaches to creating AB tests:

1.Testing new,radicaldesigns ofan entie page or process.As an example,esting three entiely diferentand distinctdesigns ofa homepage

The testwilrequire more tme to reach statisticalsignificance (conclude)

Your results are more lkely to suffer fom statisticalerrors

In an idealworld,each challenger to the cont

In an idealworld,each challenger to the controlwilnclude one change.However,his approach does notwork in realiy.Doing so wil

require both a long tme and a significantbudget.Testing more radicalchanges in each variation is a better approach.Athe same tme,

is importanto remember that:

A single and coherenthypothesis drives every change you make on a page.The hypothesis can drive severalchanges to several

elements on a page

Radicalchange to a page does NOT mean thatyou are randomly changing elements on the page.You should always te any change to your testhypothesis

Multvariate testing allows you to testseveralvariations of

multple elements on a webpage allatonce

For example,you can testmultple options ofhe productname

on an e-commerce productpage,along with various

alternatives ofhe productmage and diferentypes and colors

ofadd-to-cartbuttons

Y

You can also use multvariate testing to evaluate various

headlines and severaldesigns ofhe CTA on a lead generation

landing page

MVT testing allows you to zoom in and focus on the changes

you are making atan elementevel

19

THE DEFINITIVE GUIDE TOMULTIVARIATE AND AB TESTING

While multvariate testing is greatn theory to analyze elementevelmpacton conversion rates,has three drawbacks:

Itequires significantevels ofraffic or conversions to conclude a test

Ifyou are notcarefulwhen designing your multvariate test,you can end up testing thousands ofdesigns againsthe original

A single multvariate testncludes severalsimultaneous changes to multple elements;hus iis hard to isolate the exacteason a particular design performs a certain way s also difficulto assess the impactofelements interaction on your originaldesign

MVT testing is

MVT testing is very powerful.However,you mustbe carefulwhen using ito optimize your conversion rates.Mostcompanies forgetabout

the large volume ofvisitors and conversions required to conclude a test.As a resultthey find themselves running tests for severalweeks

withoutbringing them to a conclusion

Additonally,since testing software allows them to do so,many companies tend to alternate elements randomly withouthinking abouthe rationale behind the change.This mistake alone is enough to reduce the impactor even kilhe benefits ofany testing program

Should you startwith an A/B or multvariate testor a particular page?

There is no correctanswer to this question.A/B tests are usefulor testing alternate designs ofan entie page or a process.We usually

recommend using them while deciding high-levelor radicalchanges to the optimized page (or area).MVT testing,on the other hand,allows

for fine-grained testing on a particular page.They are helpfuln determining the mostmpactfulelements on visitors

We highly recommend using AB testing in the following scenarios:

For mostcompanies,we recommend starting with an A/B experimento assess significantdesign changes to the website

However, you already have an existing page,you can startwith a smallmultvariate testess than 12 or so diferentscenarios).The goal

ofhis initalests to determine which ofhe elements (headline,mage,benefitst,etc.)esonates mostwith vitors.Analysis ofhe first

testesults wilhelp guide the need for further MVT or A/B tests

We hi ghl y recommend usi ng mul ti vari ate testi ng i n the fol l owi ng scenari os:

Iyou have been conducting AB tests for ateast8-12 months

Iyou have a large number ofmonthly vitors

Iyou have a large number ofmonthly conversions (more than

10,000 conversions per month)

Iyou are looking to fine-tune existing designs

We hi ghl y recommend usi ng AB testi ng i n the fol l owi ng

scenari os:

Iyou are starting outwith the testing process

Iyou have lmited number ofwebsite vitors

Iyou have lmited number ofconversions

Iyou are looking for a radicaldeparture fom your existing

Fi ve steps when testi ng your websi te

1 Det ermi ne whi ch of your websi t e pages ar e good candi dat es f or opt i mi zat i on by findi ng whi ch of

your pages ar e l eaki ng t he most vi si t ors Cal cul at e t he pot ent i al evenue i mpact you can gai n f om fixi ng t he l eak.

2 Check t he number of vi si t ors t o your webpage The number of vi si t ors who wi l go t hr ough t he act ual

web page or pr ocess i mpact s your t est i ng vel oci t y Al t hough your si t e may have 500, 000 vi si t ors a mont h, a part i cul ar page t hat you want o t est may r ecei ve f ewer vi si t ors i n a si ngl e mont h

3 Do not run your t est s f or l ess t han t wo weeks nor l onger t han f our weeks

4 Pi ck t he ri ght conversi on goal Not every t est shoul d have t he goal of ncr easi ng t he macr o

conversi ons Many successf ul est s hel p i n i ncr easi ng mi cr o-conversi on rat es Thi s i s part i cul arl y

i mport ant your websi t e or l andi ng page does not ecei ve enough conversi ons St art i ng out wi t h

mi cr o conversi on t est s al l ows you t o conduct experi ment s on a smal l er scal e.

5 Exami ne what el ement s you shoul d t est Not al l el ement s on a page wi l have t he same i mpact on

your conversi on rat e Det ermi ne whi ch el ement s wi l have t he most mpact on your bot t om l ne based on t he voi ce of cust omer dat a, persona devel opment , and anal yt i cs anal ysi s.

21

THE DEFINITIVE GUIDE TOMULTIVARIATE AND AB TESTING

COMPARI NG MULTI VARI ATE & AB TESTI NG TOOLS

( UPDATED FOR 2018)

WRI TTEN BY: KHALI D SALEH

C H A P T E R 4 COMPARI NG MULTI VARI ATE 4& AB TESTI NG TOOLS ( UPDATED FOR 2018)

COMPARI NG MULTI VARI ATE & AB TESTI NG TOOLS ( UPDATED FOR 2018)

C H A P T E R 4

Iis tme now to selectan AB or multvariate testing platform to startrunning your tests.There are over thirty diferentools available in the marketplace for you to choose fom.So,you mightask:

Whats the rightsplitesting toolor my website?

Wil affecthe success ofmy conversion optimization program by choosing one particular tool?

The markets fullofesting tools thatrange fom simple fee tools to sophisticated enterprise splitesting software.This article wilwalk you through the pros and cons ofdiferentsplitesting tools available

W

We wilalso suggestcriteria to help you selecthe software thatbestfits your needs,considering ease ofuse,he agily ofesults,costofrunning,esting support,operationalssues,and testing analytics

2018 data fom datanyze shows the following top AB testing tools:

We classify A/B testing tools based on diferentcriteria:

Based on toolprice

Based on how the splitests are implemented on a

websit

Based on the scale ofsoftware (enterprise vs.smaller

scale software)

Befo

Before we discuss diferentways to classify diferent

splitesting software,et’s startwith a briefdiscussion

on in-house AB testing tools

The di ff erent types of AB or mul ti vari ate testi ng tool s avai l abl e

23

THE DEFINITIVE GUIDE TOMULTIVARIATE AND AB TESTING

I n-house AB testi ng tool s

Few companies build their in-house testing tools to conductAB or multvariate testing.There are two main reasons for using an in-house

AB testing tool:

legacy builtesting tools;

the complexity ofwebsite architecture

The

There is one more reason to consider for developing an internalesting tool.Many AB tests thatgenerate significantesults require

considerable technicalmplementation.And while A/B testing tools are greatn randomly spliing website visitors between the diferentvariations,hey are nothatgreatn doing a post-testanalysis to determine which vitor segments reactpositvely to a particular design

within the test.This is another reason thatsome large e-commerce and SaaS companies opted to build their own testing tools that

integrate directly with their platform

In-house testing tools accountor less than 2% ofhe

In-house testing tools accountor less than 2% ofhe overallmarket.We expecthis segmento remain athe same level

Cl assi fyi ng A/B testi ng tool s based on pri ce

Splitesting tools vary in price fom the fee ofcharge to tools thatcan costens ofhousands ofdollars

Cl assi fyi ng AB testi ng tool s based on testi ng technol ogy (cl i ent si de vs server si de)

There are two diferentypes ofways testing tools are used on a website:

TOOLS PRI CED BASED ON SI TE METRI CS

These tools are priced based specific website metric.Diferentools use diferentmetrics to sethe price including:

1.The number ofested visitors in a tme period (ypically a month)

2.Number ofhe website monthly visitors

3.Number ofmonthly page views

4.The number ofcalls to the testing tool

These tools wil

These tools wilvary in price and range anywhere fom $50/month to tens ofhousands ofdollars

FREE AB TESTI NG TOOLS

These tools provide the customer the abily to conductsplitesting fee ofcharge,buthey offer lmited sets ofeatures,

including a smaller setofsegmentation,argeting and reporting functions.The leading player in this space is Google Optimize which continues to add more features on a regular basis

1

CLI ENT-SI DE TESTI NG TOOLS

In these tools,he testing scripts are executed by the browser ofhe website visitor.When a visitor comes to a webpage,he

testing software javascripts executed,and the particulars ofhe testare loaded on the page

Client-side testing tools typically come with a pointand click visualeditor thatallows users to manipulate a web page

In theory,hese tools require lmited involvementom the user’s technicaleam.Creating new designs in these tools is much

faster compared to their counterparts

Client-side testing tools are greator creating new designs for one page,buthey are less suited to conductmore complex

testing such as visitor fow tests.The cornerstone “visualeditor” ofhese platforms breaks when modifying websites thately heavily on javascriptor dynamic content

These tools come with a low-costmonthly subscription,starting at$50 per month.The subscription costwilvary based on

the number ofvitors a website receives in a month

Amongstclient-side testing tools,you can find:

Optimizely VisualWebsite Optimizer Convert Omniconvert

SERVER-SI DE TESTI NG TOOL

In these tools,he testing scriptruns on the testing platform as opposed to the vitors’browser.The vitor’s browser gets the finalversion to be displayed

There are severalways testing is implemented using server-side tools.Adobe testand targe,or example,elies on the marking made by users on specific areas ofhe tested webpage.Variations ofhe marked areas are then created in the testing software

by the optimization or developmenteam.Each new testwilrequire involvementom the developmenteam

Se

Server-side testing tools fequently involve custom configuration to retrieve meaningfulvisitor tracking data.They also require

organizations to have a full-time dedicated resource(s)o manage the software and to analyze the results

While many ofhese tools do notcome with an easy to use “visualeditor” to make quick changes on a webpage,hey do

provide detailed reporting and targeting features

Monthly plans for these tools startaround $1,000 and vary based on the number ofpage views a customer uses in a month.n additon to the software cost,we estimate a $100,000 annualn-house costofrunning tests on enterprise testing packages.Se

Server-side testing platforms are usefuln conducting sophisticated testing such as mult-page testing,mobile application

testing or visitor fow testing.However,hey are an overkilwhen doing quick A/B testing for a webpage design due to the substantialcostassociated with the deploymentofeach test.The leading player in this field is Adobe Testand Target

Sel ecti ng the ri ght testi ng tool for your websi te

With many tools available,he following criteria can help you choose the bestsplitesting software to meetyour requirements

2

EASE OF USE: HOW FAST CAN YOU DEPLOY A TEST?

We evaluate the abily ofa testing toolo deploy tests quickly as a way ofmeasuring is ease ofuse

When you are conducting a conversion optimization program,your goals to implementests efficiently and noto have

technology as a barrier to your efforts ofncreasing conversion rates

Testing tools vary temendously when icomes to ease ofuse.Some ofhe more sophisticated tools require complicated testsetup,which can take days.Client-side testing tools outperform any ofheir counterparts in this area

1

25

THE DEFINITIVE GUIDE TOMULTIVARIATE AND AB TESTING

WHAT I S THE COST OF RUNNI NG THE AB TESTI NG SOFTWARE?

The costofrunning testing software varies fom zero (Google Optimize)o thousands ofdollars per month

In additon to the costofrunning the software iselfthere mightbe additonalequired investments.Some splitesting

software is too complicated and wilrequire having full-time staffo manage in additon to involvementom your development

team.That,ofcourse,wilncrease the costofrunning your testing program

2

TESTI NG SUPPORT

Sooner or later you wilneed some help fom the testing software company.Many smaller companies ignore this one area when they are doing the initalassessmentofhe testing packages.We highly recommend evaluating the diferentalternatives available

3

OPERATI ONAL I SSUES (SCALE, PERFORMANCE, AND HI GH AVAI LABI LI TY)

How long does itake the testing software to load up a particular design? Does the testing software use CDN (contentdelivery network)o deliver designs to visitors who lve in diferentparts ofhe world?

We highly recommend evaluating the response tme for each ofhe diferentsoftware packages.Avoid any testing package

thatakes longer than 500 milseconds to deliver a design

4

TESTI NG ANALYTI CS

In additon to reporting the conversion rate for each variation,some splitesting platforms wilalso reporthe following metrics

for each ofhe variations:

This data is essentialo understand the impactofhe new designs on allaspects ofhe vitor experience on your website.We

recommend evaluating whatmetrics the testing software reports in a testas thatcan impactyour finalanalysis

5

Googl e Opti mi ze

Opti mi zel y VWO

Starts from $125/month

TOOL I NI TI AL SETUP COST MONTHLY COST

26

THE DEFINITIVE GUIDE TOMULTIVARIATE AND AB TESTING

WHAT MULTI VARI ATE APPROACH DOES THE SOFTWARE USE?

This points particular in evaluating multvariate testing software.When launching a multvariate test,he software wildisplay

the originaland new variations to diferentvitors to determine the winning design.To selecthe winning design,he software

wiluse either fullactorialor fractionalactorialesting:

Ful l Factori al Testi ng

The testing softwa

The testing software wiltestallofhe diferentcombinations ofelements and their alternatives.So,a testhas four elements and each elementhas three diferentcombinations in ihe testing software wilestallpossible 3^4= 81 designs

Fracti onal Factori al testi ng (Taguchi method)

The testing softwa

The testing software wilselecta subsetofallpossible combinations ofhe diferentelements and their alternatives.So,n our example above,he testing software wilully testa smaller setess than 30)o determine the winner.Fractionalactorial

testing allows tests to run faster since they do notully testallpossible combinations.Critcs ofhis method pointouthat s

less accurate compared to fullactorialesting

The debate between fullactorialvs.ractionalactorialhas been going on for years

MostMVT testing softwa

MostMVT testing software uses fullactorialn determining the winner ofan experimentor a mix between the two diferentmethods.While this mightbe a sticking pointor some conversion experts,or 90% ofonline businesses, s nota critcalpointn selecting which software to use

6

27

THE DEFINITIVE GUIDE TOMULTIVARIATE AND AB TESTING

THE PROCESS OF CREATI NG A

SUCCESSFUL A/B TEST

WRI TTEN BY: KHALI D SALEH

THE PROCESS OF CREATI NG A SUCCESSFUL A/B TEST

C H A P T E R 5

You need people who can design proper testscenarios,analyze results accurately,and create meaningfulollow-up experiments

Poorly designed experiments mightnotprovide concrete insights into why visitors behave a particular way on your website.You need criteria to determine,or example,which elements on a page you should test,which externaland internalactors could affecthe results,and in which ways to create the designs for new phases ofyour testing program

As much as testing is essentialo any con

As much as testing is essentialo any conversion optimization project, should only be conducted after the completion ofequally essential

stages ofoptimization work such as persona development,site analysis,design and copy creation.Each ofhese elements provides a building block towards a highly optimized website thatconverts vitors into clients

Find below four steps to follow in creating a successfulsplitest.Please refer to this article to geta more detailed guide on how we come

up and conductour conversion optimization projects

Before thinking aboutelements on the page

to test,startby analyzing diferentproblem

areas on your website

How do you do that? Severalconversion

optimization methodologies can help you

Invesp uses the Conversion Framework for

page analysis

Probl em i denti ficati on

29

THE DEFINITIVE GUIDE TOMULTIVARIATE AND AB TESTING

The Conversion Framework analyzes seven diferentareas on the page:

Elements ofhese seven areas affectwhether visitors stay on your website or leave.You mustkeep in mind thatdiferentelements have

diverse impacts based on the type ofpage you are evaluating

Using the Conversion Framework,a conversion optimization expertcan easily pinpoint50 to 150 problems on a webpage

We do NOT believe you should attempto fix allofhese atonce.Prioritze and focus on the top three to seven problems to getstarted

A hypothesis is a predictive statementabouthe impactofemoving or fixing one ofhe problems identified on a webpage

The image below shows the originaldesign ofa shopping cartor one ofour clients which sells nursing uniforms.When our team

examined the analytics data for the client,we noticed the high checkoutabandonmentrates:

Abandonmentrates for un-optimized checkoutusually range fom 65% to 75%

This clienteported checkoutabandonmentrates close to 82%.Nothing in the checkoutpage explained this high rate

Our team,hen,conducted a usabily test.Nurses were invited to place an order on the site while the optimization team observed and conducted exitnterviews to gather information fom participants.The nurses revealed thathe biggestproblem was the fear ofpaying too much for a product.As nurses are price conscious,hey are aware thathey can buy the same iem fom other competing website or brick and mortar stores

Test hypothesi s

Personas development

Trust and confidence

Our clientwas aware ofhe price sensitvity issue,and thatprice played a significantole in deciding where vitors purchased a uniform or not.The client’s website already offered money-back guarantees and 100% price match.The problem is thathese assurances were only

displayed on the main homepage ofhe site,while mostofhe vitors landed on category and productpages.Visitors did notknow about

these assurances

The hypothesi s for thi s parti cul ar test:usabily study revealed thatvitors are sensitve to price,hus adding assurances can reduce the vitor price concerns and wilreduce the cartabandonmentby 20%

The image below shows the new design ofhe shopping ca

The image below shows the new design ofhe shopping cart

The team added an “assurance center” on the left-hand navigation ofhe cartpage reminding visitors ofhe 100% price match,and the money back guarantee

The new version ofhe page resulted in a 30% reduction in shopping cartabandonment

A hypothesis thatworks for one website may notsucceed or,even worse,deliver negative results,or another site

After the results ofhe previous client’s testwere published in the InternetRetailer magazine,another clientapproached us to testan assurance center on their site.This clientwas also looking for a way to reduce their cartabandonmentrate

The new design ofthe cart page with the assurance center

added to the left navigationThe originaldesign ofthe cart page

31

THE DEFINITIVE GUIDE TOMULTIVARIATE AND AB TESTING

This testhad the same hypothesis as the lastone,hatmostonline visitors did notconverton the site due to the price FUD and thatadding assurances on the cartpage would ease the shoppers’concerns.

When we tested the new version with the assurance center againsthe control,he results pointed outo an entiely diferentoutcome.The new assurance center caused the website conversion rate to drop by 4%.So,while the assurance helped one client, produced a negative

impactwith another

Can we say with absolute certainty thatadding an assurance center for the second clientwould always produce negative results? No.S

Severalelements could have influenced this particular design and caused the drop in conversion rates.The assurance center design,copy

or location could have been the realeason for the drop in conversions

Validating the hypothesis through testing and creating a follow-up hypothesis is athe heartofconversion optimization.n this case,we needed to testmany diferentelements around the assurance center before we can decide is impacton conversions

Tests thatproduce increases in conversion rates are excellentn validating initalassumptions aboutvisitors and our hypothesis

W

We do notmind tests thatesultn reducing conversion rates because we can learn something aboutour hypothesis fom these tests

We do worry aboutests thatdo notproduce any increases or decreases in conversion rates

Once you have the hypothesis,he nextstep is to startcreating new page designs thatwilvalidate i

You mustbe carefulwhen you are creating new designs.Do notgo overboard with creating new variations.Mostsplitesting software

allows you to create thousands inotmilons ofvariations for a single page.You mustkeep in mind thatvalidating each new variation

requires a certain number ofconversions

For high converting websites,we lke to lmitpage variations to less than seven.For smaller sites,we lmitpage variations to two or three new variations

Create vari ati on based on test hypothesi s

How do you judge the qualiy ofhe new designs you introduced to testyour hypothesis? You letyour visitors be the judge through AB or multvariate testing

Remember the following procedures when conducting your tests:

Selecthe rightAB testing software to speed up the process ofmplementing the test.Technology should help you implementhe test

faster and should NOT slow you down

Do notrun your testor less than two weeks.Severalactors could affectyour testesults,so allow the testing software to collectdata

long enough before concluding the test

Do notrun

Do notrun your testor longer than four weeks.Severalexternalactors could pollute your testesults,so try to lmithe impactofhese

factors by lmitng the testength

Let vi si tors be the j udge: test the new desi gns

32

THE DEFINITIVE GUIDE TOMULTIVARIATE AND AB TESTING

WHAT ELEMENTS SHOULD YOU

I NCLUDE I N AN A/B TEST?

WRI TTEN BY: KHALI D SALEH

WHAT ELEMENTS SHOULD YOU I NCLUDE I N AN A/B TEST?

C H A P T E R 6

Edi tor note:

We highly recommend thatyou implementhe diferentdeas in this blog posthrough AB testing.Use the guide to conductAB testing and figure outwhich ofhese ideas in the article works for your website visitors and which don’tDownload Invesp’s “The Essentials of

Multvariate & AB Testing” now to startyour testing program on the rightoot

As you starthe process ofmultvariate or AB testing,one ofhe critcalquestions you wilface is:Whatelements ofhe page should change and test?

This a

This article wilsuggestmany elements thatyou should consider when creating a test.However,before you jump into selecting elements and creating variations for them,ask yourselfWhy are you picking a particular iem to test?

The process ofselecting elements to teston a page is where realconversion optimization happens

Iyou are looking for a long-term strategy,something thatwilproduce a meaningfulmpacton your bottom lne,hen you mustollow a

structured conversion optimization process

A good conversi on speci al i st shoul d be abl e t o suggest many di f erent

el ement s t o t est on a part i cul ar webpage The real chal l enge, hough, es i n

creat i ng a l ong-t erm st rat egy; creat i ng a t est i ng program t hat quadrupl es your

onl i ne sal es; creat i ng a conversi on opt i mi zat i on program t hat spans t hree,

f our or five years.

“ A good conversi on speci al i st shoul d be abl e t o suggest many di f erent

el ement s t o t est on a part i cul ar webpage The real chal l enge, hough, es i n

creat i ng a l ong-t erm st rat egy; creat i ng a t est i ng program t hat quadrupl es your

onl i ne sal es; creat i ng a conversi on opt i mi zat i on program t hat spans t hree,

f our or five years.

“

A good conversi on speci al i st shoul d be abl e t o suggest many di f erent

el ement s t o t est on a part i cul ar webpage The real chal l enge, hough, es i n

creat i ng a l ong-t erm st rat egy; creat i ng a t est i ng program t hat quadrupl es your

onl i ne sal es; creat i ng a conversi on opt i mi zat i on program t hat spans t hree,

f our or five years.

34

THE DEFINITIVE GUIDE TOMULTIVARIATE AND AB TESTING

The truth about conversi on opti mi zati on

We understand thatyou mightbe looking for shortcuts.You mightbe looking for ways to capture the low-hanging fruits first.Mostofhe quick wins wilbe a resultoffixing usabily problems on your website

Butfixing usabily problems merely makes your website “usable.” Whatyou should aim for is a high converting website.There is a

significantdiference between a usable website and a highly converting website.The mindsetofvisitors and how they interactwith each of

these websites is diferent:

With a usable website:visitors can easily shop on your sit

With a highly converting website:visitors wanto shop on your websit

The following sections wilp

The following sections wilprovide you with diferentsuggestions on elements you should consider when creating your test

Val ue proposi ti on

We have seen many companies thathink ofhe value propositon as a single statementon a website (such as the tagline).A value

propositon is beyond that.tmustbe conveyed in the copy,he images,he diferentelements ofhe web page,he overalldesign ofhe website,anding pages,and campaign

How shoul d you test the val ue proposi ti on?

1.Testdiferentstatements thatexpress the value propositon

2.Testocation/placementofhe value propositon statementon a web page

3.Testhow diferentelements on the page can convey your business value propositon

Trust el ements

We covered trustn a chapter ofour book,Conversion Optimization:The Artand Science ofConverting Prospects to Customers

Whatsymbols,designs,copy do you have to supportrust? You should startby:

Listing allelements thatncrease the confidence of

vitors in your business

The sequence ofems regarding their perceived value

to vitors

Notice how we use three diferentareas on our “contact

us” page to increase visitor trust:

Since our goals to gi

Since our goals to give vitors the confidence thatwe

are a trustworthy business thatdelivers results,we use

our clients’names and logos,as wellas the number of

successfulA/B tests we have conducted

35

THE DEFINITIVE GUIDE TOMULTIVARIATE AND AB TESTING

The image increases vitor trustby showing a large number ofattendees.

The textbelow the image focuses on the number ofspeakers and sessions (rustndicator and an offer feature)

Here is a 2018 image fom the IRCE website:

Again notice the following trustelements:

The image increases vitor trustby showing a large number ofattendees

The textbelow the image focuses on the number ofspeakers and sessions (rustndicator and an offer feature)

How shoul d you test trust factors?

1.Testdiferentstatements and elements thatncrease visitor trust

2.Testocation and placementofrustelements