Horror wunderkind Ari Aster’s new movie Eddington has divided audiences and inspired plenty of online debate about what exactly the director is trying to say about our collective relationship to technology (hint: it’s probably not good). The story centers around a small town in Texas that descends into social-media-driven chaos during the covid-19 pandemic. The film stars Joaquin Phoenix as local sheriff Joe Cross, who tussles with the town’s mayor, played by Pedro Pascal, while the rest of the community loses its mind over various political and cultural disputes.
The film’s story is complicated and filled with subplots, but one of the more intriguing narrative strands involves the controversial installation of a new data center near the city limits. The company behind the ominous new facility has a very peculiar name: SolidGoldMagikarp. As it turns out, this name appears to be a reference to a relatively obscure concept in artificial intelligence development known as “glitch tokens.”
In the world of AI training, the process of “tokenization” involves translating human text into numerical data that can be interpreted by the LLM. To do this, researchers feed vast amounts of human text into software programs called “tokenizers.” Tokenizers translate the raw data from something humans can understand into parsed data designed for machine consumption, dubbed “tokens.” Tokens are then fed into a language model when it’s being “trained.” The training process involves teaching an algorithm to interact with a human—this is the way chatbots like ChatGPT learn to respond to a human’s prompt, and produce an output, or response.
Tokenization can have a fiscal benefit for companies, as it can make the effort to process large amounts of data more efficient and less costly, and a tokenizer might aptly be compared to a software compiler that translates human input into binary code that a computer can understand.
Large language models are just predictive algorithms designed to construct language based on human prompts. Like auto-correct, ChatGPT is designed to learn word sequences and regurgitate them. “That is literally all that they do,” Jessica Rumbelow, the CEO of AI firm Leap Labs, told Gizmodo. “So, you’d put in 1, 2, 3, 4, and the model is trained to predict 5,” she offered, as an example.
Tokens, entered into LLMs during the AI training process, are a critical part of teaching a program how to talk to humans. There’s a key exception to this rule, however, and that’s the phenomenon known as “glitch tokens.” Rumbelow said that the token system “typically works really well, except when you have tokens that the model has never seen” before. When an LLM encounters a token that it wasn’t trained on, it can produce very strange behavior in the program.
SolidGoldMagikarp is one such “glitch token” that, in the past, has introduced bizarre behavior into LLMs. Rumbelow would know. She and her research partner, Matthew Watkins, discovered this particular glitch token when they were looking into the anomalous token phenomenon. Rumbelow says that when the token was entered into the LLM, it produced bizarre results. The program may babble nonsense, utter oblique, ominous phrases and sentiments, or otherwise display aggressive and hostile behavior towards the user.
“We think it’s what happens when the model has not seen that token during its training process,” said Rumbelow. The AI “doesn’t know what to do with the input” because it’s never seen it before, she said.
Warning: Spoilers Ahead!
In short, phrases like SolidGoldMagikarp are obscure utterances that can make an AI model go temporarily crazy. How does SolidGoldMagikarp tie into the themes of Aster’s new movie? That’s anybody’s guess, but you could easily venture several interpretations.
... continue reading