XNews

Hugging Face tokenizers v1: 3-30x encoding speedup with SIMD and cache

Hugging Face tokenizers v1: 3-30x encoding speedup with SIMD and cache

Hugging Face released tokenizers v1, a new SOTA tokenizer library that achieves 3-30x encoding and 5.4-8.8x decoding speedups without API changes. Three core techniques: bitcannon (SIMD bitstream for pre-tokenization), word cache (thread-local cache for BPE deterministic outputs), and merge loop (zero-allocation, branchless merging).

Source posts