GoKawiilIn Which I Lose My Mind Over Embeddings (HPLM Chapter 2)
Chapter 2 of The Hundred-Page Language Models Book is when things started to get really exciting for me.
Chapter 2 – Language Modeling Basics
Chapter 2 introduces the fundamental building blocks of machine learning with natural language input and outputs: tokenization, embeddings, and model evaluation frameworks. Tokenization is the process of converting input words into “tokens” that are recognizable by language models, and embeddings convert those tokens into dense numerical representations. Together, they tackle the problem of efficiently representing arbitrarily large vocabularies in machine-readable formats.
I had heard the word “embeddings”, of course, and I understood, at a high level, that embedding models are something important for LLMs, and are used to capture semantic similarity between words. More than that, I could not have told you. HPLM explains the concepts beautifully:
Imagine that you have a language with a vocabulary of 10000 words. (Note that this is a very small language. The Oxford English Dictionary contains over 500,000 words, and modern LLMs model multiple languages simultaneously.) The most straightforward way to represent a word in this vocabulary is with a one-hot vector, an array of 10000 bits, with each index in the array corresponding to a word in the vocabulary.
One-hot vectors are easy to understand and perfectly machine-readable, but they have two major drawbacks: they are incredibly memory inefficient, wasting most of their space storing useless 0s, and they fail to encode any useful relationships between words. Indices are assigned to words arbitrarily, and words that are very similar (e.g. “happy” and “glad”) may end up far away from each other, whereas words with adjacent indices may have no meaningful semantic relationship.
Word2vec
The primary insight of embeddings is that it is possible to generate an alternative representation of tokens which is both more dense than one-hot encoding and represents semantic relationships between words. HPLM goes into detail on Word2vec, a family of algorithms used to generate word embeddings. Word2vec comes in several flavors, but they all share the same core conceptual idea. Imagine a snippet of text containing a target word (in this case “cat”) and several surrounding context words:
If you were to train a neural network such that, when given the target word as input, it produces the context words with high likelihood, then if you were to pass in a similar word, one that is likely to appear in similar contexts (e.g. “kitten”), you should expect the network to produce similar output. The structure of the network looks like this: two layers, an input layer that scales from the vocabulary size down to the embedding dimension size and an output layer that does the reverse. The output of the first layer is a vector of floats, also called the embedding vector, which can be used as a dense representation of the input word when training or using a language model.
... continue reading