Elements of Data Science
SDS 322E

H. Sherry Zhang
Department of Statistics and Data Sciences
University of Texas at Austin

Get the repository for today:
library(usethis)
create_from_github("SDS322E-26FALL/1301-ml3", fork = FALSE)

Today

So far, for your group’s question:

• ML in practice I: a dataset, a question, a response variable, and candidate predictors
• ML in practice II: a tuned KNN model

In Decision trees and random forest (Nov 11), we tuned a random forest for the penguins.

Today: build a tuned random forest for your group’s question.

• Most of today is working time in your group
• Tuning a random forest with cross-validation can take a long time: start small

The task

The same workflow as the penguins example (trees.R):

• Step 0: Training-testing split

• Step 1: Specify the model, with mtry and min_n left to tune

• Step 2: Specify the pre-processing recipe

• Step 3: Build a workflow

• Step 4: Fit the model

  • 4a: Generate the cross validation folds from the training data
  • 4b: Generate the hyperparameter grid
  • 4c: Fit the model on the cv folds and hp grid
  • 4d: Average the metrics over the folds
  • 4e: Find the best model
  • 4f: Plug in the best mtry and min_n and refit on the whole training data

• Step 5/6: Predict and calculate the metrics on the testing data

Regression or classification?

Response	Step 1	Step 6
numeric	rand_forest(mode = "regression", trees = 500, mtry = tune(), min_n = tune())	rmse, rsq
binary	rand_forest(mode = "classification", trees = 500, mtry = tune(), min_n = tune())	accuracy, roc_auc

• set_engine("ranger"), as in the lecture
• For classification, the response must be a factor, with the positive class (1) as the second level, as for KNN

Step 4b: the hyperparameter grid

• mtry: from 1 to \(p\), the number of predictors in your recipe. Check \(p\) with prep() and bake()
• min_n: e.g. from 2 to 20
• trees: don’t tune it, 500 is plenty

rf_grid <- grid_regular(mtry(range = c(1, ...)), min_n(range = c(2, 20)), levels = c(..., 4))

Computing time: start small

Tuning fits (number of combinations) \(\times\) (number of folds) forests, each with 500 trees. This adds up quickly: 6 values of mtry \(\times\) 4 values of min_n \(\times\) 10 folds = 240 forests = 120,000 trees

Start small, check that everything works, then scale up:

1. Time one fit first: system.time(fit(rf_wf_fixed, data = ..._train)) with fixed values, e.g. mtry = 3, min_n = 10
2. A small grid and few folds: e.g. levels = c(3, 2) and vfold_cv(v = 5)
3. A random subsample of large data, e.g. slice_sample(n = 5000)
4. Then make the grid finer, or use more data and folds, only if you have time

• set.seed() before the folds and before tune_grid(), so the results are reproducible
• Save the tuning results, e.g. saveRDS(rf_tune_res, "rf_tune_res.rds"), so you don’t have to rerun them

Your turn

In your group, use your question from part I and build a tuned random forest. Share at the end:

1. Your recipe: which steps did you need?
2. Your grid: how many combinations, how many folds, and how long did tuning take?
3. The cross-validation plot (autoplot()) and the best mtry and min_n
4. The test-set metrics, compared with the baseline
5. One thing that surprised you

The script ml-practice-3.R in the repository loads the four datasets and has the steps as comments. The penguins script from Nov 11 (trees.R) has all the code you need.