GoKawiilGigatoken
~1000x faster than HuggingFace's tokenizers, drop-in replacement. Tokenize your text data at GB/s! Note that both HF tokenizers and tiktoken are already running multithreaded Rust!
What is Gigatoken?
Gigatoken is the fastest tokenizer for language modeling. It supports a wide range of CPU hardware, and nearly all commonly used tokenizers. See the Benchmarks section for detailed throughput numbers across tokenizers and CPUs.
Installation
pip install gigatoken
Usage
Gigatoken can be used with its own API, or in compatibility mode with HuggingFace Tokenizers or Tiktoken.
Compatibility Mode (Easiest)
import gigatoken as gt # Minimum change from existing HuggingFace tokenizers usage (compatibility mode) hf_tokenizer = ... tokenizer = gt . Tokenizer ( hf_tokenizer ). as_hf () # tokenizer can be used in the same contexts as hf_tokenizer tokens = tokenizer . encode_batch ([ "This is a test string" , "And here is another" ]) # OR with tiktoken tiktokenizer = ... tokenizer = gt . Tokenizer ( tiktokenizer ). as_tiktoken () # Now works like existing tiktoken tokenizers tokens = tokenizer . encode_batch ([ "This is a test string" , "And here is another" ])
... continue reading