Hugging Face tokenizers v1 Cuts Encoding Time 3-30x With SIMD and Caching

Hugging Face released tokenizers v1, a new tokenization library that speeds up encoding by 3-30x and decoding by 5.4-8.8x across all languages without changing token IDs or API. Three key optimizations include SIMD-based regex replacement (bitcannon), a word cache for deterministic BPE merges, and a zero-allocation merge loop.