Hugging Face Releases New SOTA Tokenizer 'tokenizers v1'

Hugging Face released 'tokenizers v1', a new SOTA tokenization library that speeds up encoding by 3-30x and decoding by 5.4-8.8x across all languages, with unchanged API and tokens. The speedup comes from three core technologies: bitcannon (SIMD bitstream), word cache (thread-local BPE cache), and merge loop (zero-allocation). It maintains 76% linear scaling on 8 threads.