Byteification

Researchers retrofit token-based LLMs to read text at the byte level

A study published in Nature by Minixhofer and colleagues introduces 'byteification,' a method for converting existing token-based large language models so they can process text as individual bytes rather than whole word fragments. The authors report that byteified models retain the ability to identify individual characters within words while still matching the performance of standard token-based models.