Hugging Face Tokenizers v1: 3-30x Speedup with SIMD

Hugging Face released tokenizers v1, achieving 3-30x faster encoding and 5.4-8.8x faster decoding without API changes. Three core technologies drive the improvement: bitcannon SIMD-based splitting, word cache for deterministic BPE results, and a zero-allocation merge loop. The library maintains 76% linear scaling across eight threads.