Hugging Face releases tokenizers v1 with 3-30x faster encoding

Hugging Face released tokenizers v1, speeding up encoding 3-30x and decoding 5.4-8.8x across languages with unchanged token IDs/API. It keeps 76% linear scaling on eight threads and cuts size/memory via crate splitting. The gain comes from SIMD pre-tokenization, word caching, and zero-allocation merge loops.