2 ms·
Still a tiny fraction compared to available tokenizers in Pygments: https://pygments.org/languages/ https://pygments.org/languages/ (Or the vast number of Text
by WorldMaker 6y ago
Still a tiny fraction compared to available tokenizers in Pygments: https://pygments.org/languages/ https://pygments.org/languages/
(Or the vast number of TextMate grammars in the wild, such as editors like VS Code have standardized on.)
The "universality" of tokenizers is tough to beat, and though the tokenizers seem "dumber" than parser brethren, it's at a nice maxima for "good enough" for things like character-based diffs.