XNews

Hugging Face tokenizers v1: 3-30x faster encoding, 5.4-8.8x decoding

Hugging Face tokenizers v1: 3-30x faster encoding, 5.4-8.8x decoding

Hugging Face released tokenizers v1, a new SOTA tokenization library that speeds up encoding by 3-30x and decoding by 5.4-8.8x across all languages while keeping token IDs and API unchanged. It uses SIMD-based bitcannon for pre-tokenization, a word cache for deterministic BPE, and zero-allocation merge loops. The library maintains 76% linear scaling on eight threads and reduces package size and memory usage via crate splitting.

Source posts