Elements of Data Science
SDS 322E

H. Sherry Zhang
Department of Statistics and Data Sciences
University of Texas at Austin

Get the repository for today:
library(usethis)
create_from_github("SDS322E-26FALL/1103-ml2", fork = FALSE)

Today

In ML in practice I, your group picked a dataset and turned it into a question: one row, a response variable, and candidate predictors.

In K-nearest neighbor and cross validation, we tuned a KNN model for the penguins.

Today, put the two together: build a tuned KNN model for your group’s question.

  • Most of today is working time in your group
  • In part III, we fit a random forest to the same question and compare it with today’s KNN

The task

The same workflow as the penguins example:

  • Step 0: Training-testing split

  • Step 1: Specify the model, with the number of neighbors left to tune

  • Step 2: Specify the pre-processing recipe

  • Step 3: Build a workflow

  • Step 4: Fit the model

    • 4a: Generate the cross validation folds from the training data
    • 4b: Generate the hyperparameter grid
    • 4c: Fit the model on the cv folds and hp grid
    • 4d: Average the metrics over the folds
    • 4e: Find the best model
    • 4f: Plug in the best \(k\) and refit on the whole training data

  • Step 5/6: Predict and calculate classification metrics on the testing data

Regression or classification?

Response Step 1 Step 6
numeric nearest_neighbor(mode = "regression", neighbors = tune()) rmse, rsq
binary nearest_neighbor(mode = "classification", neighbors = tune()) accuracy, roc_auc

For classification, the response must be a factor, with the positive class (1) as the second level:

hotels <- hotels |> mutate(is_canceled = factor(is_canceled, levels = c(0, 1)))

Step 2: the recipe for real data

Penguins only needed step_normalize(). Real data usually needs more:

Problem Recipe step
Categorical predictors: KNN needs numbers to compute distances step_dummy(all_nominal_predictors())
A categorical predictor with many rare levels (e.g. country) step_other(all_nominal_predictors(), threshold = 0.05)
Missing values step_naomit()
Predictors on different scales step_normalize(all_numeric_predictors())
A column with only one value step_zv(all_predictors())

For example, for the hotel bookings:

hotels_rec <- recipe(..., data = ..._train) |>
  step_...(...) |>
  step_...(...) |>
  step_...(...)

Before you start

Large data: KNN computes the distance to every training row, so it is slow on big data. Work with a random subsample first:

set.seed(123)
hotels_small <- hotels |> slice_sample(n = 5000)

Your turn

In your group, use your question from part I and build a tuned KNN model. Share at the end:

  1. Your question, the response variable, and the predictors you used
  2. Your recipe: which steps did you need, and why?
  3. The cross-validation plot (autoplot()) and the best \(k\)
  4. The test-set metrics, compared with the baseline
  5. One thing that surprised you

The script ml2.R in the repository loads the four datasets and has the steps as comments. The penguins script from Nov 4 (knn.R) has all the code you need.