Elements of Data Science
SDS 322E

H. Sherry Zhang
Department of Statistics and Data Sciences
The University of Texas at Austin

Fall 2026

Learning objectives

• Understand the idea of K-nearest neighbor (KNN) classification
• Fit a KNN classifier with tidymodels: nearest_neighbor() and the kknn engine
• Understand that the number of neighbors \(k\) is a hyperparameter, and why we can’t choose it with the training data or a single test set
• Tune \(k\) with cross-validation: vfold_cv(), tune(), grid_regular(), tune_grid(), select_best(), finalize_workflow(), last_fit()

KNN

x	-5	-3	-1	0	1	2	3	4	6	8	9	10	12	15
y	1	1	1	1	1	1	0	1	0	1	0	0	0	0

Previously from a logistic regression

for x = 3.5, y = 0.72, predict 1.

Maybe there are other things to make a prediction …

How about we look at the 3 points that are closest to x = 3.5:

x	2	3	4
y	1	0	1

What’s the “average”/ majority vote?

It should get predicted to 1

This is k-nearest neighbor with k = 3.

What would you predict at x = 3 with k = 1?

Binary prediction with KNN

Follow along the class example with

usethis::create_from_github("SDS322E-26FALL/1102-knn")

penguins_clean <- as_tibble(datasets::penguins) |> na.omit()
head(penguins_clean, 3)

# A tibble: 3 × 8
  species island    bill_len bill_dep flipper_len body_mass sex     year
  <fct>   <fct>        <dbl>    <dbl>       <int>     <int> <fct>  <int>
1 Adelie  Torgersen     39.1     18.7         181      3750 male    2007
2 Adelie  Torgersen     39.5     17.4         186      3800 female  2007
3 Adelie  Torgersen     40.3     18           195      3250 female  2007

Goal: predict sex based on bill_len and bill_dep with KNN.

The same tidymodels workflow as last time, with two changes:

• the model is nearest_neighbor() with the kknn engine, in "classification" mode
• KNN is based on distances, so the recipe scales the predictors with step_normalize()

A first KNN workflow

Step 1: Specify the model, with \(k = 5\) neighbors:

knn_mod <- nearest_neighbor(mode = "classification", neighbors = 5) |>
  set_engine("kknn")

Step 2: Specify the pre-processing recipe:

knn_rec <- recipe(sex ~ bill_len + bill_dep, data = penguins_clean) |>
  step_normalize(all_predictors())

Step 3: Build a workflow:

knn_wf <- workflow() |> add_recipe(knn_rec) |> add_model(knn_mod)

A first KNN workflow

Step 4: Fit the model:

knn_fit <- knn_wf |> fit(data = penguins_clean)

Step 5: Predict:

knn_pred <- augment(knn_fit, penguins_clean)
knn_pred |> select(sex, .pred_class, .pred_female, .pred_male) |> head(5)

# A tibble: 5 × 4
  sex    .pred_class .pred_female .pred_male
  <fct>  <fct>              <dbl>      <dbl>
1 male   male                0.12       0.88
2 female female              1          0   
3 female female              0.64       0.36
4 female female              0.52       0.48
5 male   male                0.28       0.72

Step 6: Calculate classification metrics:

conf_mat(knn_pred, truth = sex, estimate = .pred_class)

          Truth
Prediction female male
    female    151   13
    male       14  155

accuracy(knn_pred, truth = sex, estimate = .pred_class)

# A tibble: 1 × 3
  .metric  .estimator .estimate
  <chr>    <chr>          <dbl>
1 accuracy binary         0.919

\(k\) is a hyperparameter

We picked \(k = 5\), but nothing in the data told us to. \(k\) is a hyperparameter: a setting we choose before fitting. Each value of \(k\) gives a different model:

• Small \(k\): the boundary bends around individual penguins, so the model is very flexible
• Large \(k\): each prediction averages over many penguins, so the boundary is smooth

Which \(k\) is best? Not on the training data

Try \(k = 1, 2, \ldots, 20\), and compute the accuracy on the same data we fit on:

With \(k = 1\), every penguin’s nearest neighbor is itself, so the training accuracy is almost perfect. The training data always rewards the most flexible model: it can’t tell us how well the model predicts new penguins.

Which \(k\) is best? Not on a single test set

Split the data, fit on the training set, and compute the accuracy on the test set for each \(k\). Here are three different random splits:

• The best \(k\) (circled) changes from split to split: one test set is too small and too noisy to choose \(k\)
• If we choose \(k\) with the test set, the test set is no longer unseen data, so its accuracy is too optimistic

Cross-validation

Instead, keep the test set aside and split the training set into, say, 10 folds. Each fold takes one turn as the validation set while the model is fit on the other 9:

For each \(k\), average the 10 validation accuracies, then pick the \(k\) with the best average. This uses the training set only, and averaging over 10 folds is much more stable than a single test set.

The workflow with tuning

• Step 0: Training-testing split

• Step 1: Specify the model, with the number of neighbors left to tune

• Step 2: Specify the pre-processing recipe

• Step 3: Build a workflow

• Step 4: Fit the model

  • Generate the cross validation folds from the training data
  • Generate the hyperparameter grid
  • Fit the model on the cv folds and hp grid
  • Find the best model

• Step 5/6: Predict and calculate classification metrics on the testing data

The workflow with tuning

Step 0: Training-testing split: new

set.seed(123)
penguins_split <- initial_split(penguins_clean, prop = 0.75)
penguins_train <- training(penguins_split)
penguins_test  <- testing(penguins_split)

Step 1/2/3: Specify the model, the pre-processing recipe, and the workflow - same, except neighbors = tune()

knn_tune_mod <- nearest_neighbor(mode = "classification", neighbors = tune()) |>
  set_engine("kknn")
knn_rec <- recipe(sex ~ bill_len + bill_dep, data = penguins_train) |>
  step_normalize(all_predictors())
knn_tune_wf <- workflow() |> add_recipe(knn_rec) |> add_model(knn_tune_mod)

The workflow with tuning

Step 4: Fit the model: cross validation folds, hyperparameter grid, fit on both, and find the best model

penguins_folds <- vfold_cv(penguins_train, v = 10)
knn_grid <- grid_regular(neighbors(range = c(1, 20)), levels = 20)
knn_tune_res <- tune_grid(knn_tune_wf, resamples = penguins_folds, grid = knn_grid)
best_k <- select_best(knn_tune_res, metric = "accuracy")
best_k

# A tibble: 1 × 2
  neighbors .config         
      <int> <chr>           
1        14 pre0_mod14_post0

knn_tune_res |>
  collect_metrics() |>
  filter(.metric == "accuracy") |>
  ggplot(aes(x = neighbors, y = mean)) +
  geom_line() +
  geom_point() +
  geom_point(data = \(d) filter(d, neighbors == best_k$neighbors), size = 5, shape = 21, stroke = 1.5) +
  scale_x_continuous(breaks = 1:20) +
  labs(x = "k", y = "Cross validation accuracy")

The workflow with tuning

Step 5/6: Predict and calculate classification metrics on the testing data: plug the best \(k\) into the workflow, refit on the whole training data, and evaluate once on the testing data

final_wf <- finalize_workflow(knn_tune_wf, best_k)
final_fit <- fit(final_wf, data = penguins_train)
res_final <- augment(final_fit, penguins_test)
metrics(res_final, truth = sex, estimate = .pred_class)

# A tibble: 2 × 3
  .metric  .estimator .estimate
  <chr>    <chr>          <dbl>
1 accuracy binary         0.798
2 kap      binary         0.593

last_fit(final_wf, split = penguins_split) does the refit and the testing-data evaluation in one step.

This follows the big picture:

• Training set: preprocess (step_normalize()), list candidate values of \(k\), choose the best with cross validation, refit
• Test set: used once, at the end, to report how the final model does on new data