Hugging Face Releases tokenizers v1 with SIMD, Cache, and Merge Loop Optimizations

Hugging Face's tokenizers v1 achieves 3-30x encoding speedup and 5.4-8.8x decoding speedup across all languages without API changes. Three key optimizations: bitcannon SIMD-based pre-tokenization replacing regex, word cache for deterministic BPE results, and merge loop with zero allocation and branchless operations. These combine to deliver the speedups.