GoKawiilTextbooks say LOOCV has the lowest bias but highest variance compared to 10 and 5-fold. Coded a K-Fold CV from scratch for learning to test that on simulated data 🔍📊 — and at n=1000 it holds up. At n=100? Not so much. 🤔
The above image was generated via chatGPT. Uploaded all the text of this blog post and asked it to generate a cartoon. Very impressive! It used to be spelling error and gibberish of text in the past, but now cohesive words on image. Just wow.
Motivations
Crossvalidation is such a crucial step in Machine Learning (and traditional methods) that nowadays is incorporated in easy to use sklearn or tidymodels without us needing to build one from scratch. As with my other learning experience, the best way to learn the concept (other than learning the concept 🤣) is to code it from the ground up and see how it works! In K-Fold CV, the training data is split into K chunks; the model is trained K times, each time holding out a different chunk. Performance is averaged across all K folds, giving a more stable estimate. A special case is Leave-One-Out CV (LOOCV), where each individual observation serves as its own validation set. It’s thorough but computationally expensive. I was told that, bias LOOCV < 10-fold < 5-fold; whereas variance LOOCV > 10-fold > 5-fold. Is that true? Also, what’s with the repeats, does that really reduce variance? Let’s check them out.
Simulate Data
library (tidyverse) set.seed ( 1 ) n <- 1000 x <- rnorm (n) w <- rnorm (n) y <- 0.5 * x^2 + -0.5 * w + 0.3 * w * x + rnorm (n) df <- tibble (x,y,w) idx <- sample ( 1 : n, size = 0.8 * n) train <- df[idx, ] test <- df[ - idx, ]
The above code simulates a dataset with 1000 observations, where the response variable y is generated based on a known data-generating process involving predictors x and w . The dataset is then split into a training set (80%) and a test set (20%). Let’s visualize.
df |> mutate (w_cut = cut_interval (w, n = 5 )) |> ggplot ( aes (x = x, y = y, color = w_cut, group = w_cut)) + geom_point (alpha = 0.5 ) + theme_bw () + geom_smooth (method = "gam" , se = F )
Wow, very interesting visualization where the relationships are definitely not linear here. It’s some form of interaction between x and w . Let’s see if we can recover the underlying data-generating process using K-Fold Cross-Validation.
K-Fold Cross-Validation From Scratch
... continue reading