## Predicting DVD purchase

In this report we'll estimate several different models on the `dvd` data. Specifically, we'll use Logistic Regression, Neural networks, and Decision trees to model the probability of purchasing a DVD after exposure to an "instant coupon" online. The models can all be estimated in Radiant.

### Data description

The `dvd` has an `attribute` that contains a description of the dataset:

`r gsub("## DVD\\n\\n### Description", "", attr(dvd, "description"))`

To keep the `dvd` data loaded in Radiant through the _Data > Manage_ tab clean and ensure the analysis is reproducible we'll start by making a working copy in the _Data > View_ tab. We'll call the new dataset `dvd_wrk`.

```{r}
### filter and sort the dataset
dvd_wrk <- dvd %>%
  select(buy:last)
register("dvd_wrk", "dvd")
```

To compare model performance we need a training and a validation (or test) dataset. In Radiant we can create a `training` variable in the _Data > Transform_ tab that is 1 if an observation is assigned to the training sample and 0 otherwise. Because the `dvd` dataset is very _clean_ we'll use a training sample of only 20% of the data. This will increase the chance that we'll overfit the training data, particularly for the Neural Network model and (un-pruned) Decision Trees. 

```{r}
## created variable to select training sample
dvd_wrk <- mutate(dvd_wrk, training = make_train(0.2, n(), seed = 1234))
```

To make things even a bit more interesting lets add a few random generated variables as well.
 
```{r}
## create new variable(s)
dvd_wrk <- mutate(
  dvd_wrk, 
  rnd1 = rnorm(20000), 
  rnd2 = rnorm(20000), 
  rnd3 = rnorm(20000), 
  rnd4 = rnorm(20000), 
  rnd5 = rnorm(20000)
)
```

### Estimate a Logistic Regression

In a first step lets estimate a Logistic Regression with all available explanatory variables and generate a prediction of the response probability of buying a DVD (i.e., `pred_logit`). 

```{r fig.width=7, fig.height=4.43, dpi = 200}
result <- logistic(
  dvd_wrk, 
  rvar = "buy", 
  evar = c(
    "coupon", "purch", "last", "rnd1", "rnd2", "rnd3", 
    "rnd4", "rnd5"
  ), 
  lev = "yes", 
  data_filter = "training == 1"
)
summary(result)
plot(result, plots = "coef")
pred <- predict(result, pred_data = dvd_wrk)
plot(pred, xvar = "coupon")
dvd_wrk <- store(dvd_wrk, pred, name = "pred_logit")
```

It is not straightforward to evaluate the relative importance of the variables in the model based on the plot above because the variables are on different scales. Lets re-estimate to get standardized odds-ratios.

```{r fig.width = 7, fig.height = 4.43}
result <- logistic(
  dvd_wrk, 
  rvar = "buy", 
  evar = c(
    "coupon", "purch", "last", "rnd1", "rnd2", 
    "rnd3", "rnd4", "rnd5"
  ), 
  lev = "yes", 
  check = "standardize", 
  data_filter = "training == 1"
)
plot(result, plots = "coef")
```

### Estimate a Neural Network 

Next, we'll estimate a Neural Network with 8 nodes in the hidden layer and parameter decay of 0.5. The optimal values for these parameters can be established using a grid-search in, e.g., the `caret` package in R. The network plot below illustrates how inputs are connected to the nodes in the hidden layer and to the output node. Predicted probabilities are stored in `pred_nn`.

```{r fig.width = 7, fig.height = 6.5}
result <- nn(
  dvd_wrk, 
  rvar = "buy", 
  evar = c(
    "coupon", "purch", "last", "rnd1", "rnd2", "rnd3", 
    "rnd4", "rnd5"
  ), 
  lev = "yes", 
  size = 8, 
  decay = 0.05, 
  seed = 1234, 
  data_filter = "training == 1"
)
summary(result)
plot(result, plots = "net")
pred <- predict(result, pred_data = dvd_wrk)
dvd_wrk <- store(dvd_wrk, pred, name = "pred_nn")
```

### Estimate a Decision Tree

Finally, we'll estimate a Decision Tree. The optimal size of the tree can be established using cross-validation. We'll generate predicted probabilities for an un-pruned tree (i.e., `pred_crtree_large`) and for a pruned tree (i.e., `pred_crtree`). The pruned tree has four nodes.

```{r}
result <- crtree(
  dvd_wrk, 
  rvar = "buy", 
  evar = c(
    "coupon", "purch", "last", "rnd1", "rnd2", "rnd3", 
    "rnd4", "rnd5"
  ), 
  type = "classification", 
  lev = "yes", 
  prior = 0.5, 
  data_filter = "training == 1"
)
summary(result, prn = TRUE)
pred <- predict(result, pred_data = dvd_wrk)
print(pred, n = 10)
dvd_wrk <- store(dvd_wrk, pred, name = "pred_crtree_large")
plot(result, plots = "prune", custom = FALSE)
plot(result, orient = "LR", width = "600px") %>% render()
```

```{r}
result <- crtree(
  dvd_wrk, 
  rvar = "buy", 
  evar = c(
    "coupon", "purch", "last", "rnd1", "rnd2", "rnd3", 
    "rnd4", "rnd5"
  ), 
  type = "classification", 
  lev = "yes", 
  nodes = 4, 
  prior = 0.5, 
  data_filter = "training == 1"
)
summary(result, prn = FALSE)
pred <- predict(result, pred_data = dvd_wrk)
print(pred, n = 10)
dvd_wrk <- store(dvd_wrk, pred, name = "pred_crtree")
plot(result, orient = "LR") %>% render()
```

### Evaluate model performance 

We can evaluate model performance in the training and validation sample using _Model > Evaluate classification_.

First, lets see if there is any evidence of overfitting for the neural network with 8 nodes in the hidden layer. The gains chart shown below indicates that performance in the training dataset is better than in the validation dataset.

```{r fig.width = 7, fig.height = 6}
result <- evalbin(
  dvd_wrk, 
  pred = "pred_nn", 
  rvar = "buy", 
  lev = "yes", 
  qnt = 50, 
  cost = 1,
  margin = 8, 
  train = "Both", 
  data_filter = "training == 1"
)
summary(result, prn = FALSE)
plot(result, plots = "gains")
```

Next, lets evaluate the Decision Tree predictions in training and validation. The first gains chart shows that the large decision tree fits the training data much better compared to the pruned decision tree.

```{r fig.width = 7, fig.height = 6}
result <- evalbin(
  dvd_wrk, 
  pred = c("pred_crtree", "pred_crtree_large"), 
  rvar = "buy", 
  lev = "yes", 
  qnt = 50, 
  cost = 1,
  margin = 8, 
  train = "Training", 
  data_filter = "training == 1"
)
summary(result, prn = FALSE)
plot(result, plots = "gains")
```

However, when we look at the validation dataset the conclusion is reversed. Now the pruned decision tree fits (slightly) better than the the large (un-pruned) decision tree.

```{r fig.width = 7, fig.height = 6}
result <- evalbin(
  dvd_wrk, 
  pred = c("pred_crtree", "pred_crtree_large"), 
  rvar = "buy", 
  lev = "yes", 
  qnt = 50, 
  cost = 1,
  margin = 8, 
  train = "Validation", 
  data_filter = "training == 1"
)
summary(result, prn = FALSE)
plot(result, plots = "gains")
```

Finally, lets compare different models in the validation sample using a gains chart and a profit curve. Both plots suggest that, at least for the `dvd` data, the Logistic Regression performs best. 

```{r fig.width = 7, fig.height = 11}
result <- evalbin(
  dvd_wrk, 
  pred = c("pred_logit", "pred_nn", "pred_crtree_large", "pred_crtree"), 
  rvar = "buy", 
  lev = "yes", 
  qnt = 50, 
  margin = 8, 
  train = "Validation", 
  data_filter = "training == 1"
)
summary(result, prn = FALSE)
plot(result, plots = c("gains", "profit"), custom = FALSE)
```

Results from the _Model > Evaluate classification > Confusion_ tab confirm that the Logisitic Regression has the best fit in the validation sample (AUC).

```{r fig.width = 7, fig.height = 11}
result <- confusion(
  dvd_wrk, 
  pred = c(
    "pred_logit", "pred_nn", "pred_crtree_large",
    "pred_crtree"
  ), 
  rvar = "buy", 
  lev = "yes", 
  qnt = 50, 
  margin = 8, 
  train = "Validation", 
  data_filter = "training == 1"
)
summary(result)
```