3 ms·

I'm interested in your package, but as a go programmer. I spend my hollidays writing a [colorizer](https://github.com/chmike/clrz https://github.com/chmike/clrz

by chmike 9y ago

I'm interested in your package, but as a go programmer. I spend my hollidays writing a [colorizer](https://github.com/chmike/clrz https://github.com/chmike/clrz) package in go. I compared many colorizer packages and tried to make something original. I also wanted to support faster lexers than the one based on regex. I had to stop by the end of hollidays. Do you have a documentation on how your lexer works ? Does it use regex only ? I don't understand rust and don't want to learn it.

thesmallestcat 9y ago
> I don't understand rust and don't want to learn it. Tread lightly, for one more comment about Rust will summon the Rust Evangelism Strike Force.
- hnbroseph 9y ago
  rust programs have zero bugs, don't you know. and they practically write themselves in moments.
  - sctb 9y ago
    Could you please stop posting like this and instead comment civilly and substantively? https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- chmike 9y ago
  Sorry. You are right. I'm close to 55year old, and start to feel the limit of number of neurons available. I have to use them sparingly. That is why I enjoy so much Go. There is no judgment of Rust. It's just because of me. :)
- blaenk 9y ago
  The only ones that ever show are the harbingers of the purported evangelism strike force :)
trishume 9y ago
Syntect's grammars are regex based, they're based on ST3's grammars which are an extension of the fundamental model used by Textmate/Atom/VSCode. They're stronger than just regexes though, there's a fancy stack machine with all sorts of features laid on top that allows it to do full parsing of a lot of languages. The more modern grammars are written with a regex style that makes heavier use of the stack machine and so only uses regexes that can be turned into a DFA. There's ongoing work to use a layer on top of Rust's super-fast DFA-based regex engine to accelerate these grammars https://github.com/trishume/syntect/pull/34 https://github.com/trishume/syntect/pull/34. The problem with using non-regex based grammars is that you have to write them yourself. Syntect is something like 4000 lines of code but the grammars it uses total around 35k lines, and that's just the included ones, not the full ecosystem of online grammars. Basically unless you only want to support a small set of languages, a non-regex-based highlighting library is fairly infeasibly for a single hobbyist.
- GrayShade 9y ago
  > There's ongoing work to use a layer on top of Rust's super-fast DFA-based regex engine to accelerate these grammars Will it really be faster? It didn't seem so from that GitHub thread. As a data point for anyone curious, I'm using Syntect myself in a toy project. With Oniguruma (C NFA regexes), it highlights 200 lines of Rust in 40 ms on a 10 W TDP Celeron, which is all right, but a bit slower than I expected.
  - trishume 9y ago
    It'll hopefully be faster. The Rust regex engine is quite fast, I haven't done any profiling to figure out why performance was the same in my initial test. There might be something easy to fix. It's definitely possible to get better performance out of the underlying model, since Sublime does, but they have a custom DFA-based engine that can test regexes in parallel with captures, which Rust's regex engine can't.

by chmike 9y ago

thesmallestcat 9y ago

> I don't understand rust and don't want to learn it. Tread lightly, for one more comment about Rust will summon the Rust Evangelism Strike Force.

hnbroseph 9y ago
rust programs have zero bugs, don't you know. and they practically write themselves in moments.
- sctb 9y ago
  Could you please stop posting like this and instead comment civilly and substantively? https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
chmike 9y ago
Sorry. You are right. I'm close to 55year old, and start to feel the limit of number of neurons available. I have to use them sparingly. That is why I enjoy so much Go. There is no judgment of Rust. It's just because of me. :)
blaenk 9y ago
The only ones that ever show are the harbingers of the purported evangelism strike force :)

trishume 9y ago

Syntect's grammars are regex based, they're based on ST3's grammars which are an extension of the fundamental model used by Textmate/Atom/VSCode. They're stronger than just regexes though, there's a fancy stack machine with all sorts of features laid on top that allows it to do full parsing of a lot of languages. The more modern grammars are written with a regex style that makes heavier use of the stack machine and so only uses regexes that can be turned into a DFA. There's ongoing work to use a layer on top of Rust's super-fast DFA-based regex engine to accelerate these grammars https://github.com/trishume/syntect/pull/34 https://github.com/trishume/syntect/pull/34. The problem with using non-regex based grammars is that you have to write them yourself. Syntect is something like 4000 lines of code but the grammars it uses total around 35k lines, and that's just the included ones, not the full ecosystem of online grammars. Basically unless you only want to support a small set of languages, a non-regex-based highlighting library is fairly infeasibly for a single hobbyist.

GrayShade 9y ago
> There's ongoing work to use a layer on top of Rust's super-fast DFA-based regex engine to accelerate these grammars Will it really be faster? It didn't seem so from that GitHub thread. As a data point for anyone curious, I'm using Syntect myself in a toy project. With Oniguruma (C NFA regexes), it highlights 200 lines of Rust in 40 ms on a 10 W TDP Celeron, which is all right, but a bit slower than I expected.
- trishume 9y ago
  It'll hopefully be faster. The Rust regex engine is quite fast, I haven't done any profiling to figure out why performance was the same in my initial test. There might be something easy to fix. It's definitely possible to get better performance out of the underlying model, since Sublime does, but they have a custom DFA-based engine that can test regexes in parallel with captures, which Rust's regex engine can't.