Hugging Face tokenizers v1: 3-30x encoding speedup with SIMD and cache

Hugging Face released tokenizers v1, a new SOTA tokenization library that speeds up encoding by 3–30x and decoding by 5.4–8.8x across all languages without changing the API. Three core optimizations are used: bitcannon (SIMD bitstream for pre-tokenization), word cache (thread-local cache for deterministic BPE results), and merge loop (zero-allocation, branchless merge). The combination yields the dramatic speedups, with individual techniques addressing control flow, redundant computation, and memory allocation.