# A tibble: 344 × 8
species island bill_len bill_dep flipper_len body_mass sex year
<fct> <fct> <dbl> <dbl> <int> <int> <fct> <int>
1 Adelie Torgersen 39.1 18.7 181 3750 male 2007
2 Adelie Torgersen 39.5 17.4 186 3800 female 2007
3 Adelie Torgersen 40.3 18 195 3250 female 2007
4 Adelie Torgersen NA NA NA NA <NA> 2007
5 Adelie Torgersen 36.7 19.3 193 3450 female 2007
# ℹ 339 more rowsElements of Data Science
SDS 322E
H. Sherry Zhang
Department of Statistics and Data Sciences
The University of Texas at Austin
Fall 2026
Department of Statistics and Data Sciences
The University of Texas at Austin
Fall 2026
Learning objectives
- Understand what problem dimension reduction solves, and what principal components (PCs) are
- Perform principal component analysis (PCA) in R:
- prepare the data: remove missing values and scale the variables (
prcomp(scale = TRUE)) - compute PCA with
prcomp()and read its output: standard deviations, rotation (loadings), center/scale - choose the number of PCs with a scree plot (
factoextra::fviz_eig()) - get the projected data with
broom::augment()and plot the first two PCs - interpret the PCs with a biplot (
factoextra::fviz_pca_biplot())
- prepare the data: remove missing values and scale the variables (
Motivation: dimension reduction
Hand shadow puppets: a 3D hand is projected onto a 2D wall as a shadow.
- From a good angle, the shadow still looks like a rabbit: most of the information is kept.
- From a bad angle, it’s just a blob.
Principal component analysis (PCA)
PCA is a linear dimension reduction method: it finds a set of new variables, the principal components (PCs), that capture as much of the variation in the data as possible.
- Mathematically, PCA solves an optimization problem: maximize the variance of the projected data, subject to some constraints.
- (Optional, see the Appendix) The solution is given by the singular value decomposition (SVD) of the centered data matrix \(X\).
- In R,
prcomp()does all of this for you.
PCA pre-processing: missing values
Do you think missing values will be a problem for PCA?
species island bill_len bill_dep flipper_len body_mass sex year
1 Adelie Torgersen 39.1 18.7 181 3750 male 2007
2 Adelie Torgersen 39.5 17.4 186 3800 female 2007
3 Adelie Torgersen 40.3 18.0 195 3250 female 2007
5 Adelie Torgersen 36.7 19.3 193 3450 female 2007
6 Adelie Torgersen 39.3 20.6 190 3650 male 2007
7 Adelie Torgersen 38.9 17.8 181 3625 female 2007Yes: prcomp() needs complete rows, since each PC score uses all the variables.
Variable scaling in PCA
prcomp() performs PCA on the numeric variables.
- Centering (
center = TRUE) is on by default - see the math in the Appendix for why. - Scaling (
scale = TRUE) is not the default, but we almost always want it.
See the help page for prcomp():
Why scale? PCA looks for the directions with the most variance, and the variables are in different units. Variance of each variable:
bill_len bill_dep flipper_len body_mass
1 29.90633 3.877888 196.4417 648372.5Without scaling, body_mass (in g) has by far the largest variance, so PC1 is just body mass and explains 99.99% of the variance.
Alternatively, scale the data yourself with scale() first, then use prcomp(..., center = FALSE, scale = FALSE).
Fit PCA with prcomp()
With scaling, every variable has standard deviation 1:
Standard deviations (1, .., p=4):
[1] 1.6569115 0.8821095 0.6071594 0.3284579
Rotation (n x k) = (4 x 4):
PC1 PC2 PC3 PC4
bill_len 0.4537532 -0.60019490 -0.6424951 0.1451695
bill_dep -0.3990472 -0.79616951 0.4258004 -0.1599044
flipper_len 0.5768250 -0.00578817 0.2360952 -0.7819837
body_mass 0.5496747 -0.07646366 0.5917374 0.5846861Inspect the prcomp() output
sdev: standard deviations of the principal components. Their squares are the eigenvalues: the variance explained by each PC
rotation: principal components (eigenvectors) - directions of maximum variance
Scree plot with factoextra
The fviz_eig() function draws the scree plot in one command.
This plot is called a scree plot - used to tell the number of principal components to retain. Choose the number at the “elbow” point: k = 2 here.
Variable plot: how variables contribute to PC1 and PC2
Biplot: observations and variables together
Interpret the first two PCs
PC1 contrasts overall size (flipper length, body mass, bill length) with bill depth: Gentoo penguins are larger with shallower bills, so they have high PC1 scores.
PC2 is driven mostly by bill depth and bill length: Chinstrap penguins have much longer bills than Adelie, so they have lower PC2 scores.
This matches with the summary statistics:
# A tibble: 3 × 5
species bill_len bill_dep flipper_len body_mass
<fct> <dbl> <dbl> <dbl> <dbl>
1 Adelie 38.8 18.3 190. 3706.
2 Chinstrap 48.8 18.4 196. 3733.
3 Gentoo 47.6 15.0 217. 5092.Beyond PCA: other dimension reduction methods
- Principal component analysis (PCA) is a linear dimension reduction method, there are other linear methods, e.g.
- Projection Pursuit (PP)
- There are also non-linear dimension reduction methods, e.g.
- Multidimensional Scaling (MDS),
- T-stochastic Neighbourhood Embedding (t-SNE), and
- Uniform manifold Approximation and Projection (UMAP)
Appendix: data and projected data
Data:
\[\begin{aligned} X_{n\times p} & = [X_{1} \enspace X_{2} \enspace \dots \enspace X_{p}]_{n\times p} \\ & = \left[ \begin{array}{cccc} x_{11} & x_{12} & \dots & x_{1p} \\ x_{21} & x_{22} & \dots & x_{2p}\\ \vdots & \vdots & & \vdots \\ x_{n1} & x_{n2} & \dots & x_{np} \end{array} \right]_{n\times p} \end{aligned}\]We require column means of \(X\) to be zero (centered data): \(\bar{X_i} = 0\)
Projected data:
\[\begin{aligned}
Z_{n\times k} = X\Phi
&=[Z_{1} \enspace \dots \enspace Z_{k}]_{n\times k} \\
& = \left[
\begin{array}{cccc}
z_{11} & \dots & z_{1k} \\
z_{21} & \dots & z_{2k}\\
\vdots & & \vdots \\
z_{n1} & \dots & z_{nk}
\end{array} \right]_{n\times k}
\end{aligned}\]
The loading matrix \(\Phi\)
In PCA, \(\Phi\) is called the rotation or the loading matrix:
\[\begin{aligned} \Phi_{p \times k} & = [\phi_{1} \enspace \phi_{2} \enspace \dots \enspace \phi_{k}]_{p\times k} \\ & = \left[ \begin{array}{cccc} \phi_{11} & \phi_{12} & \dots & \phi_{1k} \\ \phi_{21} & \phi_{22} & \dots & \phi_{2k}\\ \vdots & \vdots & & \vdots \\ \phi_{p1} & \phi_{p2} & \dots & \phi _{pk} \end{array} \right]_{p\times k} \end{aligned}\]\(\Phi\) satisfies two conditions:
- It is normalized: \(\sum _{i=1}^{p} \phi_{ij}^2 = 1\) for \(j = 1, \dots, k\) and
- Vectors are orthogonal to each other: \(\sum_{i=1}^{p} \phi_{ij}\phi_{ik} = 0\) for \(j \neq k\).
(We will talk about why this is the case in a second.)
PC1 is a linear combination of the variables
Principal components are the linear combinations of the original variables:
The first principal component (PC1):
\[\begin{aligned} Z_1 = \begin{bmatrix} z_{11} \\ z_{21} \\ \vdots \\ z_{n1}\end{bmatrix} &= \phi_{11} \begin{bmatrix} x_{11} \\ x_{21} \\ \vdots \\ x_{n1}\end{bmatrix} + \phi_{21} \begin{bmatrix} x_{12} \\ x_{22} \\ \vdots \\ x_{n2}\end{bmatrix} + \dots + \phi_{p1}\begin{bmatrix} x_{1p} \\ x_{2p} \\ \vdots \\ x_{np}\end{bmatrix} \\ &= \phi_{11}X_1 + \phi_{21}X_2 + \dots + \phi_{p1}X_p \\ \end{aligned}\]where \(\sum _{i=1}^{p} \phi_{i1}^2 = 1\) and \(\bar{Z_i} = 0\) (since \(\bar{X_i} = 0\))
PC1 maximizes the variance
For each element in \(Z_1\): \(z_{i1} = \phi_{11}x_{i1} + \phi_{21}x_{i2} + \dots + \phi_{p1}x_{ip} = \sum_{j = 1}^p \phi_{j1} x_{ij}\)
Write down the variance of \(Z_1\):
\[\text{Var}(Z_1) = E(Z_1^2) = \frac{1}{n}\sum_{i = 1}^n z_{i1}^2=\frac{1}{n}\sum_{i = 1}^n (\sum_{j = 1}^p \phi_{j1} x_{ij})^2 \]
We would like to find the direction (the coefficients \(\phi_{j1}\)’s) that maximizes the variance of \(Z_1\):
\[\max_{\phi_{11}, \dots, \phi_{p1}} \text{Var}(Z_1) = \max_{\phi_{11}, \dots, \phi_{p1}} \frac{1}{n}\sum_{i = 1}^n (\sum_{j = 1}^p \phi_{j1} x_{ij})^2 \] subject to the normalization constraint \(\sum _{i=1}^{p} \phi_{i1}^2 = 1\).
Solving for PC1: an eigenvalue problem
\[\text{Var}(Z_1) = E(Z_1^2) = \frac{1}{n}\sum_{i = 1}^n z_{i1}^2 = \left\lVert z_{1} \right\rVert^2 = \left\lVert X\phi_1 \right\rVert^2 = \phi_1^T X^T X \phi_1\]
subject to \(\phi_1^T \phi_1 = 1\)
Use Lagrange multipliers to solve the constrained optimization problem:
\[ L(\phi_1, \lambda) = \phi_1^T X^T X \phi_1 - \lambda (\phi_1^T \phi_1 - 1)\] Take the derivative and set it to zero gives \[X^TX\phi_1 = \lambda \phi_1\]
This is the eigenvalue equation for the matrix \(X^TX\).
Recognize \(X^TX\) is related the covariance matrix: \(S = \frac{1}{n-1}X^TX\), the first principal component (\(\phi_1\)) is the eigenvector of \(X^TX\) corresponding to the largest eigenvalue.
Computing the PCs: SVD or eigen-decomposition
The first principal component (\(\phi_1\)) is the eigenvector of \(X^TX\) corresponding to the largest eigenvalue. This can be obtained through:
- Singular value decomposition (SVD) of \(X\) (always apply):
\[X = UDV^T\]
- Eigen-decomposition of \(X^TX = (n-1) S\) (since \(X^TX\) is square):
\[X^TX = V D^2 V^T\]
where
- \(X\) is a \(n \times p\) matrix
- \(U\) is a \(n \times n\) orthogonal matrix (\(U^TU = 1\))
- \(D\) is a \(n \times p\) diagonal matrix with non-negative elements (square root of the eigenvalues of \(X^TX\) arranged in descending order)
- \(V\) is a \(p \times p\) orthogonal matrix (eigenvectors: \(V^TV = 1\))
The second principal component (PC2)
The second principal component (PC2): \(Z_2 = \phi_{12}X_1 + \phi_{22}X_2 + \dots + \phi_{p2}X_p\)
The second principal component is the linear combination of \(X_1, X_2, \dots, X_p\) that has the largest variance out of all linear combinations that are uncorrelated with \(Z_1\).
subject to “\(Z_2\) being uncorrelated with \(Z_1\)”.
This is equivalent to constraining \(\phi_2\) to be orthogonal to \(\phi_1\), hence
\[\max_{\phi_{12}, \dots, \phi_{p2}} \text{Var}(Z_2) = \max_{\phi_{12}, \dots, \phi_{p2}} \frac{1}{n}\sum_{i = 1}^n (\sum_{j = 1}^p \phi_{j2} x_{ij})^2 \] subject to the normalization constraint \(\sum _{i=1}^{p} \phi_{i2}^2 = 1\) and the orthogonality constraint \(\sum_{i=1}^{p} \phi_{i1}\phi_{i2} = 0\).