XNews

Hugging Face tokenizers v1 boosts speed up to 30x with SIMD and caching

Hugging Face tokenizers v1 boosts speed up to 30x with SIMD and caching

Hugging Face released tokenizers v1, a SOTA tokenization library that accelerates encoding by 3-30x and decoding by 5.4-8.8x across all languages without API changes. It uses bitcannon (SIMD bit streams) to replace regex, word cache for BPE, and zero-allocation merge loops.

Source posts