Hugging Face Tokenizers v1: 3-30x Speedup with SIMD and Cache

Hugging Face released tokenizers v1, achieving 3-30x encoding speedup and 5.4-8.8x decoding speedup using three core technologies: bitcannon SIMD for pre-tokenization, word cache for BPE, and zero-allocation merge loop. The library maintains full API compatibility and scales linearly across 8 threads.