Cookbook Modeling Data for Marketing_3 pptx

The following code uses PROC TABULATE to create the decile analysis, a table that calculates the number of observations records in each decile, the average predicted probability per deci

The following code uses PROC TABULATE to create the decile analysis, a table that calculates the number of observations (records) in each decile, the average predicted probability per decile, the percent active per responders (target of Method 2, model 2), the response rate (target of Method 2, model 1), and the active rate (target of Method 1).

title1 "Decile Analysis - Activation Model - One Step";

title2 "Model Data - Score Selection";

proc tabulate data=acqmod.mod_dec;

weight smp_wgt;

class mod_dec;

var respond active pred records activ_r;

table mod_dec='Decile' all='Total',

In Figure 5.5, the decile analysis shows the model's ability to rank order the prospects by their active behavior To clarify, each

prospect's probability of becoming active is considered its rank The goal of the model is to rank order the prospects so as to bring the true actives to the lowest decile At first glance we can see that the best decile (0) has 17.5 as many actives as the worst decile

Figure 5.5 Decile analysis using model data

Page 117(9) And as we go from decile 0 to decile 9, the percent active value is monotonically decreasing with a strong decrease

in the first three deciles The only exception is the two deciles in the middle with the same percent active rate This is not unusual since the model is most powerful in deciles 0 and 9, where it gets the best separation Overall, the model score does a good job of targeting active accounts Because the model was built on the data used in Figure 5.5, a better test will be on the validation data set

Another consideration is how closely the ''Percent Active" matches the "Predicted Probability." The values in these columns for each decile are not as close as they could be If my sample had been larger, they would probably be more equal I will look for similar behavior in the decile analysis for the validation data set

Preliminary Evaluation

Because I carried the validation data through the model using the missing weights, each time the model is processed, the validation data set is scored along with the model data By creating a decile analysis on the validation data set we can evaluate how well the model will transfer the results to similar data As mentioned earlier, a model that works well on

alternate data is said to be robust In chapter 6, I will discuss additional methods for validation that go beyond simple

decile analysis

The next code listing creates the same table for the validation data set This provides our first analysis of the ability of the model to rank order data other than the model development data It is a good test of the robustness of the model or its ability to perform on other prospect data The code is the same as for the model data decile analysis except for the

(where=( splitwgt = )) option This accesses the "hold out" sample or validation data set.

proc univariate data=acqmod.out_act1

(where=( splitwgt = )) noprint;

weight smp_wgt;

var pred active;

output out=preddata sumwgt=sumwgt;

run;

data acqmod.val_dec;

set acqmod.out_act1(where=( splitwgt = )) ;

if (_n_ eq 1) then set preddata;

retain sumwgt;

number+smp_wgt;

if number < 1*sumwgt then val_dec = 0; else

if number < 2*sumwgt then val_dec = 1; else

if number < 3*sumwgt then val_dec = 2; else

if number < 4*sumwgt then val_dec = 3; else

if number < 5*sumwgt then val_dec = 4; else

if number < 6*sumwgt then val_dec = 5; else

if number < 7*sumwgt then val_dec = 6; else

if number < 8*sumwgt then val_dec = 7; else

if number < 9*sumwgt then val_dec = 8; else

val_dec = 9;

activ_r = (activate = '1');

run;

title1 "Decile Analysis - Activation Model - One Step";

title2 "Validation Data - Score Selection";

PROC tabulate data=acqmod.val_dec;

weight smp_wgt;

class val_dec;

var respond active pred records activ_r;

table val_dec='Decile' all='Total',

Page 119with the best decile attracting almost seven times as many actives as the worst decile We see the same degree of

difference between the "Predicted Probability" and the actual "Percent Active" as we saw in the decile analysis of the model data in Figure 5.5 Decile 0 shows the most dramatic difference, but the other deciles follow a similar pattern to the model data There is also a little flipflop going on in deciles 5 and 6, but the degree is minor and probably reflects nuances in the data In chapter 6, I will perform some more general types of validation, which will determine if this is a real problem

Method 2:

Two Models — Response

The process for two models is similar to the process for the single model The only difference is that the response and activation models are processed separately through the stepwise, backward, and Score selection methods The code differences are highlighted here:

proc logistic data=acqmod.model2

(keep=variables) descending;

weight splitwgt;

model respond = variables

/selection = stepwise sle=.3 sls=.3;

model respond = variables

/selection = score best=2;

run;

The output from the Method 2 response models is similar to the Method 1 approach Figure 5.7 shows the decile analysis for the response model calculated on the validation data set It shows strong rank ordering for response The rank ordering for activation is a little weaker, which is to be expected There are different drivers for response and activation Because activation is strongly driven by response, the ranking for activation is strong

Method 2:

Two Models — Activation

As I process the model for predicting activation given response (active|response), recall that I can use the value activate because it has a value of missing for nonresponders This means that the nonresponders will be eliminated from the model processing The following code processes the model:

Figure 5.7 Method 2 response model decile analysis.

proc logistic data=acqmod.model2

(keep=variables) descending;

weight splitwgt;

model activate = variables

/selection = stepwise sle=.3 sls=.3;

model activate = variables

/selection = score best=2;

run;

The output from the Method 2 activation models is similar to the Method 1 approach Figure 5.8 shows the decile analysis for the activation model calculated on the validation data set It shows strong rank ordering for activation given response As expected, it is weak when predicting activation for the entire file Our next step is to compare the results of the two methods

Figure 5.8 Method 2 activation model decile analysis.

Comparing Method 1 and Method 2

At this point, I have several options for the final model I have a single model that predicts the probability of an active account that was created using Method 1, the single -model approach And I have two models from Method 2, one that predicts the probability of response and the other that predicts the probability of an active account, given response (active|response)

To compare the performance between the two methods, I must combine the models developed in Method 2 To do this, I use a simplified form of Bayes' Theorem Let's say:

P(R) = the probability of response (model 1 in Method 2)

P(A|R) = the probability of becoming active given response (model 2 in Method 2)

P(A and R) = the probability of responding and becoming active

Then:

P(A and R) = P(R)* P(A|R)

Therefore, to get the probability of responding and becoming active, I multiply the probabilities created in model 1 and model 2

Page 122Following the processing of the score selection for each of the two models in Method 2, I reran the models with the final

variables and created two output data sets that contained the predicted scores, acqmod.out_rsp2 and acqmod.out_act2

The following code takes the output data sets from the Method 2 models built using the score option The where= ( splitwgt = ) option designates both probabilities are taken from the validation data set Because the same sample was

used to build both models in Method 2, when merged together by pros_id the names should match up exactly The

rename=(pred=predrsp) creates different names for the predictors for each model.

proc sort data=acqmod.out_rsp2 out=acqmod.validrsp

(where=( splitwgt = ) rename=(pred=predrsp)) ;

by pros_id;

run;

proc sort data=acqmod.out_act2 out=acqmod.validact

(where=( splitwgt = ) rename=(pred=predact)) ;

To compare the models, I create a decile analysis for the probability of becoming active derived using Method 2

(predact2 ) with the following code:

proc sort data=acqmod.blend;

Page 123

if number < 2*sumwgt then act2dec = 1; else

if number < 3*sumwgt then act2dec = 2; else

if number < 4*sumwgt then act2dec = 3; else

if number < 5*sumwgt then act2dec = 4; else

if number < 6*sumwgt then act2dec = 5; else

if number < 7*sumwgt then act2dec = 6; else

if number < 8*sumwgt then act2dec = 7; else

if number < 9*sumwgt then act2dec = 8; else

act2dec = 9;

run;

title1 "Decile Analysis - Model Comparison";

title2 "Validation Data - Two Step Model";

PROC tabulate data=acqmod.blend;

weight smp_wgt;

class act2dec;

var active predact2 records;

table act2dec='Decile' all='Total',

Figure 5.9 Combined model decile analysis

Page 124

Summary

This chapter allowed us to enjoy the fruits of our labor I built several models with strong power to rank order the prospects by their propensity to become active We saw that many of the segmented and transformed variables

dominated the models And we explored several methods for finding the best-fitting model using two distinct

methodologies In the next chapter, I will measure the robustness of our models and select a winner

Page 125

Chapter 6—

Validating the Model

The masterpiece is out of the oven! Now we want to ensure that it was cooked to perfection It's time for the taste test!Validating the model is a critical step in the process It allows us to determine if we've successfully performed all the prior steps If a model does not validate well, it can be due to data problems, poorly fitting variables, or problematic techniques There are several methods for validating models In this chapter, I begin with the basic tools for validating the model, gains tables and gains charts Marketers and managers love them because they take the modeling results right

to the bottom line Next, I test the results of the model algorithm on an alternate data set A major section of the chapter focuses on the steps for creating confidence intervals around the model estimates using resampling This is gaining popularity as an excellent method for determining the robustness of a model In the final section I discuss ways to validate the model by measuring its effect on key market drivers

Gains Tables and Charts

A gains table is an excellent tool for evaluating the performance of a model It contains actionable information that can

be easily understood and used by

non-Page 126technical marketers and managers Using our case study, I will demonstrate the power of a gains table for Method 1.

Method 1:

One Model

Recall that in Method 1 I developed a score for targeting actives using the one-model approach To further validate the results of the model, I put the information from the decile analysis in Figure 5.6 into a spreadsheet; now I can create the gains table in Figure 6.1

Column A mirrors the decile analysis in Figure 5.6 It shows each decile containing 10% of the total mail file.

Column B is cumulative percent of file for validation portion of the prospect data set.

Column C, as in Figure 5.6, is the average Probability of Activation for each decile as defined by the model.

Column D, as in Figure 5.6, is the average Percent Actives in the validation data set for each decile This represents the

number of true actives in the validation data set divided by the total prospects for each decile

Column E is a cumulative calculation of column D or Cumulative Percent Actives At each decile you can estimate the

number of true actives for a given "Depth of File" or from decile 0 to a designated decile If we consider the value for decile 4, we can say that the average Percent Active for deciles 0 through 4 is 0.214%

Column F is the number of true actives in the decile (Col A * Col D).

Figure 6.1 Enhanced gains table on validation data.alidating the Model

Page 127

Column G is column F divided by the sum of column F This represents the percent Actives of the Total Actives that are

contained in each decile If we consider the value in Decile 1, we can say that 27.71% of all Actives in the validation

data set were put into decile 1 by the model

Column H is a cumulative of column F This represents the total Number of Actives for a given ''Depth of File" or from

decile 0 to a designated decile

Column I is column H divided by the number of total actives At each decile you can determine the Cumulative Percent

of Total Actives that are contained in decile 0 through a given decile If we consider the value in decile 1, we can say that 42.79% of all Actives are in deciles 0 and 1.

Column J is the "lift" of each decile The lift is calculated by dividing the decile percent active by the overall percent

active in column D The value of 277 in decile 0 means that the prospects in decile 0 are 277% more likely to activate than the overall average

Column K is the cumulative Lift through a specific decile It is calculated by dividing a decile value in column E by the

overall percent active

The lift measurement is a favorite among marketers for evaluating and comparing models At each decile it demonstrates

the model's power to beat the random approach or average performance In Figure 6.1, we see that decile 0 is 2.77 times

the average Up to and including decile 3, the model performs better than average Another way to use lift is in

cumulative lift This means to a given "Depth of File": the model performs better than random If we go through decile 3,

we can say that for the first four deciles, the model performs 64% better than average Figure 6.2, a validation gains chart, provides a visual interpretation of the concept of Lift.

The gains chart is an excellent visual tool for understanding the power of a model It is designed to compare the percent

of the target group to the percent of the prospect file through the rank ordered data set At 50% of the file we can capture

74% of the active accounts— a 47% increase in Actives (This will be discussed in more detail in chapter 7.)

Method 2:

Two Models

Recall that in Method 2, I built a model to target Response and a second model to target Actives given response.

In Figure 6.3 we see the ability of the Method 2 model to rank order the data When comparing the active rate for the deciles, the power of the model is slightly better than the Method 1 model in distinguishing actives from nonresponders

and nonactive responders The lift measures imitate the similarity.

Page 128

Figure 6.2 Validation gains chart

Recall that the response model in the two-model approach did a good job of predicting actives overall The gains chart in Figure 6.4 compares the activation model from the one -model approach to the response and combination models

developed using the two-model approach The response model seems to do almost as well as the active models Again, this supports our theory that the probability of being active is strongly driven by response

In conclusion, Method 1 and Method 2 produce very similar results when modeling the probability of an account being

active This may not always be the case It is worth exploring both methods to determine the best method for each

situation In the next sections, I will compare the stability of the models through further validation In chapter 7, I will discuss alternate ways to use the Method 2 models in the implementation process

Page 129

Figure 6.3 Enhanced gains table on validation data

Figure 6.4 Gains chart comparing models from Method 1 and Method 2