4 ms·

First, syntect can use both onig (an adaptation of Ruby regex, same as Sublime) and fancy-regex. The latter is yet another xi-instigated, and was motivated by t

First, syntect can use both onig (an adaptation of Ruby regex, same as Sublime) and fancy-regex. The latter is yet another xi-instigated, and was motivated by trying to do a better job at this. Second, the design of fancy-regex is precisely to delegate to the regex crate when features like lookahead aren't needed. I had high hopes for the performance, but it turned out to be lackluster, a highly tuned backtracking engine beats it in most cases. And much of the problem is indeed that RE2-style approaches are a lot faster when captures aren't required. And, by nature, Sublime-style syntax definitions rely extremely heavily on captures. A bit of discussion (including comments from burntsushi) here: https://www.reddit.com/r/rust/comments/5zit0e/regex_captures_slow_compared_to_python/ https://www.reddit.com/r/rust/comments/5zit0e/regex_captures... Third, the question of whether "regular languages" would support faster parsing is somewhat academic, though perhaps you could build something. Though the languages being parsed often have some regular features, as a rule there are exceptions. Markdown is an extreme example, and cannot be parsed well with either regular languages or Perl-style regexes, though that hasn't stopped people from trying.

chubot 6y ago
OK thanks for the clarifications. I responded to BurntSushi and others here: https://lobste.rs/s/fq8uil/aho_corasick https://lobste.rs/s/fq8uil/aho_corasick https://news.ycombinator.com/item?id=23665341 https://news.ycombinator.com/item?id=23665341 I don't have experience with captures in re2c, but it's entirely automata-based, and they apparently implement something "fast" for captures and published it in this 2017 paper, so it's pretty new: https://arxiv.org/abs/1907.08837 https://arxiv.org/abs/1907.08837 I'm sort of interested in benchmarking some Sublime-ish workloads with it, but I'm not sure I'll have time. ---- And the other claim from that comment is that while regular languages aren't powerful enough by themselves for syntax highlighting, you can add a trivial state machine on top of them (perhaps Vim-like), and get something more powerful and faster than Sublime's model. I'd have to look at the captures vs. lookahead issue more closely, but the general claim is that I think Sublime has a bad parsing model based on a sloppy application of Perl-style regexes (e.g. if you are really forced to read every byte multiple times, that seems dumb, and can be avoided with regular languages)

chubot 6y ago

OK thanks for the clarifications. I responded to BurntSushi and others here: https://lobste.rs/s/fq8uil/aho_corasick https://lobste.rs/s/fq8uil/aho_corasick https://news.ycombinator.com/item?id=23665341 https://news.ycombinator.com/item?id=23665341 I don't have experience with captures in re2c, but it's entirely automata-based, and they apparently implement something "fast" for captures and published it in this 2017 paper, so it's pretty new: https://arxiv.org/abs/1907.08837 https://arxiv.org/abs/1907.08837 I'm sort of interested in benchmarking some Sublime-ish workloads with it, but I'm not sure I'll have time. ---- And the other claim from that comment is that while regular languages aren't powerful enough by themselves for syntax highlighting, you can add a trivial state machine on top of them (perhaps Vim-like), and get something more powerful and faster than Sublime's model. I'd have to look at the captures vs. lookahead issue more closely, but the general claim is that I think Sublime has a bad parsing model based on a sloppy application of Perl-style regexes (e.g. if you are really forced to read every byte multiple times, that seems dumb, and can be avoided with regular languages)