Hugging Face's tokenizers v1 Speeds Up Tokenization 3-30x with SIMD and Caching

Hugging Face released tokenizers v1, a new SOTA tokenization library that accelerates encoding by 3-30x and decoding by 5.4-8.8x across all languages without changing token IDs or the API. The speedup comes from three core technologies: bitcannon (SIMD-based pre-tokenization), word cache (caching BPE results), and a zero-allocation, branchless merge loop. The library also achieves smaller package size and memory usage through crate splitting.