Elements of Data Science
SDS 322E

H. Sherry Zhang
Department of Statistics and Data Sciences
University of Texas at Austin

Get the repository for today:
library(usethis)
create_from_github("SDS322E-26FALL/1303-ml4", fork = FALSE)

Today

So far, for your group’s question:

• ML in practice I: a dataset, a question, a response variable, and candidate predictors
• ML in practice II and III: a tuned KNN, and a tuned random forest

In Comparing models with tidymodels (Nov 18), we compared three models for the penguins.

Today, do the same for your group’s question: compare at least three models, and choose one.

• Again, most of today is working time in your group

The task

The same steps as the penguins example (compare-models.R):

• Step 0: Training-testing split, shared by all models
• Steps 1-3, for each model: the model, the recipe (depends on the model), the workflow
• Step 4: Cross-validation, for all models:
  • 4a: the cross-validation folds, shared by all models
  • 4b: the hyperparameter grid for each model
  • 4c: fit each model on the folds
  • 4d: average the metrics over the folds
  • 4e: the best version of each model
• Step 4.5: Compare the models on their cross-validation results, and choose one
• Step 5/6: Refit the chosen model on the training data, and evaluate it once on the testing data

Which models?

Use the three models from class, in the mode that matches your response:

• Binary response (e.g. is_canceled, SUCCESS1, rap vs rock)
  • logistic_reg()
  • nearest_neighbor(mode = "classification")
  • rand_forest(mode = "classification")
  • Metrics: metric_set(accuracy, roc_auc): higher is better
  • The response must be a factor, with the positive class (1) as the second level
• Numeric response (e.g. total_cup_points, track_popularity)
  • linear_reg()
  • nearest_neighbor(mode = "regression")
  • rand_forest(mode = "regression")
  • Metrics: metric_set(rmse, rsq): for RMSE, lower is better

A recipe for each model

Start from the same data, and add the steps each model needs:

• Linear / logistic regression: step_dummy(all_nominal_predictors()): categorical predictors become 0/1 dummy variables
• KNN: step_dummy() and step_normalize(all_numeric_predictors()): distances need numbers on the same scale
• Random forest: nothing extra: trees split on categorical predictors directly, and don’t care about scale

And for all models, if your data needs it:

• Missing values: step_naomit()
• A categorical predictor with many rare levels: step_other(all_nominal_predictors(), threshold = 0.05) (before step_dummy())

Check what a recipe does with prep() and bake().

Before you start

• Reuse your code: your KNN and random forest from parts II and III are Steps 1-4 for two of the models
• One split, one set of folds: create them once, and use them for all three models
• Uneven response? Use strata in initial_split() and vfold_cv(), e.g. 37% of the hotel bookings are canceled, 71% of the expeditions succeed
• Large data? Work with a random subsample, e.g. slice_sample(n = 5000): 30 values of \(k\) \(\times\) 10 folds is a lot of KNN fits
• Reproducible: set.seed() before the split, the folds, and tuning the random forest

Your turn

In your group, compare the three models for your question. Consider:

1. The cross-validation comparison: the best version of each model, with its mean and standard error, and a plot
2. For a binary response: the ROC curves of the three models
3. Your choice and the justification
4. The chosen model’s performance on the testing data, evaluated once
5. For the penguins, the flexible models won because the boundary depends on species. Did they win for your question? Why do you think so?

The script ml-practice-4.R in the repository loads the four datasets and has the steps as comments. compare-models.R from Nov 18 has all the code you need.