XNews

Hugging Face Releases Tokenizers v1 with 3-30x Speedup

Hugging Face Releases Tokenizers v1 with 3-30x Speedup

Hugging Face released tokenizers v1, achieving 3-30x encoding and 5.4-8.8x decoding speedups via SIMD bit streams, word caching, and zero-allocation merge loops, with full API compatibility.

Source posts