Hugging Face tokenizers v1: 3-30x faster encoding with SIMD

Hugging Face released tokenizers v1, a new SOTA tokenization library that speeds up encoding by 3-30x and decoding by 5.4-8.8x across all languages. It uses bitcannon (SIMD bit streams), word cache, and zero-allocation merge loops, while maintaining the same API and token IDs.