Hugging Face tokenizers v1 boosts speed up to 30x with SIMD and caching

Hugging Face released tokenizers v1, a SOTA tokenization library that accelerates encoding by 3-30x and decoding by 5.4-8.8x across all languages without API changes. It uses bitcannon (SIMD bit streams) to replace regex, word cache for BPE, and zero-allocation merge loops.