XNews

Hugging Face tokenizers v1 achieves 3-30x speedup with SIMD and cache

Hugging Face tokenizers v1 achieves 3-30x speedup with SIMD and cache

Hugging Face released tokenizers v1, a new SOTA tokenization library that speeds up encoding by 3-30x and decoding by 5.4-8.8x across all languages, using bitcannon SIMD for pre-tokenization, word cache for BPE, and zero-allocation merge loops. It maintains API compatibility.

Source posts