Hugging Face tokenizers v1 achieves 3-30x speedup via SIMD and caching

Hugging Face released tokenizers v1, boosting encoding speed 3-30x and decoding 5.4-8.8x while keeping token IDs and API unchanged. Three core technologies drive gains: bitcannon (SIMD bitstreams replacing regex), word cache (caching deterministic BPE results), and merge loop (zero-allocation branchless operations).