XNews

Hugging Face tokenizers v1: 3-30x faster encoding with SIMD and cache

Hugging Face tokenizers v1: 3-30x faster encoding with SIMD and cache

Hugging Face released tokenizers v1, a new SOTA tokenization library that speeds up encoding by 3-30x and decoding by 5.4-8.8x across all languages. It uses three core techniques: bitcannon (SIMD bitstream for pre-tokenization), word cache (caching BPE results), and merge loop (zero-allocation, branchless).

Source posts