XNews

Vercel Engineer Ships gpu-lexer: 27.5KB Neural Syntax Highlighter on WebGPU

Vercel Engineer Ships gpu-lexer: 27.5KB Neural Syntax Highlighter on WebGPU

Vercel's Shu Ding released gpu-lexer, a 27.5KB neural syntax-highlighting model running on WebGPU. It targets three rule-based pain points: MB-scale grammar bundles (Shiki/TextMate), complex language detection with async loading, and main-thread blocking from CPU tokenizers. The model performs sequence labeling—predicting each token's syntax role—trained on 50+ languages and generalizing to unseen ones via cross-language patterns. Benchmarked against Shiki's real VS Code grammars on top-25 GitHub languages, with the author noting AST-based Shiki is structurally more precise and gpu-lexer cannot yet fully replace rule engines.

Source posts