4 ms·

Shameless plug: my implementation of Sublime's syntax highlighting engine in Rust has similar optimizations and more. I'm not at my computer to benchmark on the

Shameless plug: my implementation of Sublime's syntax highlighting engine in Rust has similar optimizations and more. I'm not at my computer to benchmark on the same files but it should be >2x as fast as their "after" numbers just based on lines/second for JS-like files. This evening I'm even trying to port it to a pure Rust regex engine that should eliminate non-Rust code and make it substantially faster. It also implements the sublime-syntax format which is a superset of tmlanguage that allows even nicer highlighting. https://github.com/trishume/syntect https://github.com/trishume/syntect

octref 10y ago
Sorry but I'm not familiar with Rust -- is it possible to run your lib in all platforms in a Node environment? And, do you have an issue for the regex port? Would love to see the benchmark.
- trishume 10y ago
  Yes it would be possible to write a C wrapper for my Rust library and link that from Node, however I expect you might lose a lot of performance to data conversion. I have yet to do the regex engine port, I'm doing that later this evening. I'm porting it from Oniguruma to https://github.com/google/fancy-regex https://github.com/google/fancy-regex which accelerates common types of regexes using the awesome Rust regex crate which is super fast.
  - dgfgfdagasdfgfa 10y ago
    Doesn't rust export the C abi? Or does the "c wrapper" just involve converting c types to rust domain types>
    - kevincox 10y ago
      You can very easily export a C api from rust. And fact you would have to if you wanted to write a wrapper anyways so I think the wrapper is probably unnecessary.
    - trishume 10y ago
      Sorry that's what I meant. Just export c ABI functions in a Rust crate.
nhaehnle 10y ago
Thanks for this. Clearly the original post describes good work, but I can't help feeling the JS community is slacking off when it comes to performance. Just eyeballing the cited numbers, they take 3939ms to handle a 1.18MB input on "a somewhat powerful desktop machine". Assuming that that means a chip running at 2GHz, we're talking about over 6300 cycles per byte! That's quite frankly ridiculous. An improvement by at least one order of magnitude should be possible. Where's the ambition? (Yes, there's always a trade-off with these things. But I feel someone has to point this out when the OP is explicitly about getting kudos for performance work.)
- shanemhansen 10y ago
  The dynamic languages people are always saying programmer time is more important than computer time. Yet they put out an editor that is wasteful with both.
- trishume 10y ago
  It's harder than it sounds if you want to support many languages. The Sublime syntaxes repo I use has 34,000 lines of grammars whereas my engine is only 3000 lines of code. If you count all the tmLanguage files for nice languages available online it's probably hundreds of thousands of lines, and that's in a pretty dense format. The whole point of using tmLanguage files is that people don't care about how fast other languages are if there is no highlighting for their language. I could get way better performance by rewriting all those grammars using compiled parsers in Rust (like Xi has as an option https://github.com/google/xi-editor/blob/master/rust/lang/Cargo.toml https://github.com/google/xi-editor/blob/master/rust/lang/Ca...) but it would take an absurd amount of effort. The speed of highlighting tmLanguage files is limited mostly by the regex library. My code spends 50% of its time in Oniguruma. You can improve that a bit by using a fancy regex engine, which is how Sublime is faster than my engine, but this evening I'm going to try to port my library to a faster engine based on Rust's regex crate. But that library (https://github.com/google/fancy-regex https://github.com/google/fancy-regex) is brand new and almost untested. It's the only open source library that is faster than Oniguruma while supporting all the right features. In the future VSCode may be able to gain some speed by switching to it, but they have much more overhead over the regex library to eat away first than I do.
trishume 10y ago
So I ran the sqlite3.c benchmark on my machine (a comparable "somewhat powerful" machine) and it took 6.7s with my engine vs 10.9s with the new VSCode one. Both are doing tokenization+theme matching but the machines and exact grammars are not necessarily the same. I'm using ST3's C grammar but they are using a different one. It could be that I ran it on a faster computer with an easier grammar, or it could be the opposite. It's close enough I'm not willing to claim my engine is substantially faster. For context Sublime Text 3 takes 2 seconds with the same grammar and same file on the same computer due to using a better custom regex engine written specifically for highlighting. Given what alexdima mentioned in a different comment about spending most of the time in the regex engine, I'm not sure that my engine would be substantially faster under exactly identical conditions since I'm also bottlenecked by Oniguruma. However, maybe after I port my engine to https://github.com/google/fancy-regex https://github.com/google/fancy-regex I'll be substantially faster. And if I do it is likely they could also benefit from a fancy-regex port.
HeyImAlex 10y ago
That moment when someone else already did your hobby project :congrats through tears:

octref 10y ago

Sorry but I'm not familiar with Rust -- is it possible to run your lib in all platforms in a Node environment? And, do you have an issue for the regex port? Would love to see the benchmark.

trishume 10y ago
Yes it would be possible to write a C wrapper for my Rust library and link that from Node, however I expect you might lose a lot of performance to data conversion. I have yet to do the regex engine port, I'm doing that later this evening. I'm porting it from Oniguruma to https://github.com/google/fancy-regex https://github.com/google/fancy-regex which accelerates common types of regexes using the awesome Rust regex crate which is super fast.
- dgfgfdagasdfgfa 10y ago
  Doesn't rust export the C abi? Or does the "c wrapper" just involve converting c types to rust domain types>
  - kevincox 10y ago
    You can very easily export a C api from rust. And fact you would have to if you wanted to write a wrapper anyways so I think the wrapper is probably unnecessary.
  - trishume 10y ago
    Sorry that's what I meant. Just export c ABI functions in a Rust crate.

nhaehnle 10y ago

Thanks for this. Clearly the original post describes good work, but I can't help feeling the JS community is slacking off when it comes to performance. Just eyeballing the cited numbers, they take 3939ms to handle a 1.18MB input on "a somewhat powerful desktop machine". Assuming that that means a chip running at 2GHz, we're talking about over 6300 cycles per byte! That's quite frankly ridiculous. An improvement by at least one order of magnitude should be possible. Where's the ambition? (Yes, there's always a trade-off with these things. But I feel someone has to point this out when the OP is explicitly about getting kudos for performance work.)

shanemhansen 10y ago
The dynamic languages people are always saying programmer time is more important than computer time. Yet they put out an editor that is wasteful with both.
trishume 10y ago
It's harder than it sounds if you want to support many languages. The Sublime syntaxes repo I use has 34,000 lines of grammars whereas my engine is only 3000 lines of code. If you count all the tmLanguage files for nice languages available online it's probably hundreds of thousands of lines, and that's in a pretty dense format. The whole point of using tmLanguage files is that people don't care about how fast other languages are if there is no highlighting for their language. I could get way better performance by rewriting all those grammars using compiled parsers in Rust (like Xi has as an option https://github.com/google/xi-editor/blob/master/rust/lang/Cargo.toml https://github.com/google/xi-editor/blob/master/rust/lang/Ca...) but it would take an absurd amount of effort. The speed of highlighting tmLanguage files is limited mostly by the regex library. My code spends 50% of its time in Oniguruma. You can improve that a bit by using a fancy regex engine, which is how Sublime is faster than my engine, but this evening I'm going to try to port my library to a faster engine based on Rust's regex crate. But that library (https://github.com/google/fancy-regex https://github.com/google/fancy-regex) is brand new and almost untested. It's the only open source library that is faster than Oniguruma while supporting all the right features. In the future VSCode may be able to gain some speed by switching to it, but they have much more overhead over the regex library to eat away first than I do.

trishume 10y ago

So I ran the sqlite3.c benchmark on my machine (a comparable "somewhat powerful" machine) and it took 6.7s with my engine vs 10.9s with the new VSCode one. Both are doing tokenization+theme matching but the machines and exact grammars are not necessarily the same. I'm using ST3's C grammar but they are using a different one. It could be that I ran it on a faster computer with an easier grammar, or it could be the opposite. It's close enough I'm not willing to claim my engine is substantially faster. For context Sublime Text 3 takes 2 seconds with the same grammar and same file on the same computer due to using a better custom regex engine written specifically for highlighting. Given what alexdima mentioned in a different comment about spending most of the time in the regex engine, I'm not sure that my engine would be substantially faster under exactly identical conditions since I'm also bottlenecked by Oniguruma. However, maybe after I port my engine to https://github.com/google/fancy-regex https://github.com/google/fancy-regex I'll be substantially faster. And if I do it is likely they could also benefit from a fancy-regex port.

HeyImAlex 10y ago

That moment when someone else already did your hobby project :congrats through tears: