Hugging Face's tokenizers v1 achieves 3-30x speedup with SIMD and caching

Hugging Face released tokenizers v1, delivering 3-30x encoding and 5.4-8.8x decoding speed improvements across languages without changing token IDs or API. Key technical innovations include bitcannon (SIMD bitstream for pre-tokenization), word cache (reusing BPE results for repeated tokens), and a zero-allocation, branchless merge loop.