VN IPP - Week 1 - Session 04 - Experiment Design Basics & The Real Book

VN IPP - Week 1 - Session 04 - Experiment Design Basics & The Real Book

Experiment Design Basics

&

The Real Book

Learning goal:

Which value prop is better? Faster

download time or obfuscated IP address.

Hypothesis:

Using VP of Faster download time instead

of obfuscated IP address will increase

conversion rate on home page above 15%.

Learning goal:

Which value prop is better? Faster

download time or obfuscated IP address.

Hypothesis:

Using VP of Faster download time instead

of obfuscated IP address will increase

conversion rate on home page above 15%.

Retrospectives

G: Do customers want our VP?

H: 50% of people we interview

will say they want it!

M: % of people who want it.

F: <50%

P: Go interview people on Market

street

T: 1 day

R: Success, 7 people said yes.

N: Do a landing page test.

s! No one will talk to us.

The Unofficial Startup

Real Book

Quantitative

• Who is our customer?

• Do people use it?

• Which is better?

• Did we do the right thing?

• Are these results good?

• Are we making progress?

CLOSED-ENDED QUESTIONS

Why?

Qualitative

• How should I design it

• Do people use it?

• What’s wrong with this?

• How could I make this better?

• Why do people do that?

• What do people love about it?

• How can I improve conversion?

• Why is there drop off?

• What do they think?

OPEN-ENDED QUESTIONS

Problem Market

• Who is our customer?

• What are their pains?

• What job needs to be done?

• How do we find them?

• How much will they pay?

• How do we convince them to buy?

• How much will it cost to sell?

• Can we scale marketing?

Solution Product

• How can we solve this problem?

• What form should this take?

• How important is the design?

• What’s the quickest hack?

• What’s the minimum feature set?

• How should we prioritize?

• Is this solution working?

• Are people using it?

• Which solution is better?

• How should we optimize this?

• What do people like / dislike?

• Why do they do that?

Generate Ideas &

Identify Assumptions

Research

• Who is our customer?

• What are their pains?

• What job needs to be done?

• Is our customer segment too broad?

• How do we find them?

• How can we solve this problem?

• What form should this take?

• How important is the design?

• What’s the quickest hack?

• What’s the minimum feature set?

• How should we prioritize?

Test Hypotheses

Experiments

• How much will they pay?

• How do we convince them to buy?

• How much will it cost to sell?

• Can we scale marketing?

• Is this solution working?

• Are people using it?

• Which solution is better?

• How should we optimize this?

• What do people like / dislike?

• Why do they do that?

• Who is our customer?

• What are their pains?

• What job needs to be done?

• Is our customer segment too broad?

• How do we find them?

• Customer Discovery Interviews

• Contextual inquiry / ethnography

• Data mining

• Focus groups *

• Surveys * (open ended)

• Are they really willing to pay?

• How much will they pay?

• How do we convince them to buy?

• How much will it cost to sell?

• Can we scale marketing?

Click Me

• How can we solve this problem?

• What form should this take?

• How important is the design?

• What’s the quickest hack?

• What is the minimum feature set?

• How should we prioritize?

2

Our First Version (The Concierge Model)

• Is this solution working?

• Are people using it?

• Which solution is better?

• How should we optimize this?

• What do people like / dislike?

• Why do they do that?

References & Contributors