6 ms·

Sourcegraph CEO here. @slimsag is the Sourcegrapher who wrote https://github.com/sourcegraph/syntect_server https://github.com/sourcegraph/syntect_server in Rus

by sqs 8y ago

Sourcegraph CEO here. @slimsag is the Sourcegrapher who wrote https://github.com/sourcegraph/syntect_server https://github.com/sourcegraph/syntect_server in Rust, based on @trishume's awesome https://github.com/trishume/syntect https://github.com/trishume/syntect lib. He chose to use Rust for many of the same reasons that other people mention in this thread: https://news.ycombinator.com/item?id=15324787 https://news.ycombinator.com/item?id=15324787. I'll ping him in case there are any more details.

slimsag 8y ago
Sourcegraph dev here :) At Sourcegraph we've actually tried a lot of different syntax highlighters in the past. Syntax highlighting is actually a _really_ hard problem. Syntax highlighting for basically all editors today is provided by TextMate language grammers[1] (`.tmLanguage` files) ultimately, which are just giant regex definition files, basically. These are the ubiquitous files for syntax highlighting, and you can almost always find a TextMate grammer for any programming language. Some 'modern' syntax highlighters, mostly ones designed for use in the web, try not to use these TextMate files -- but in doing so they lose support for TONS of languages, and the languages that they do support are often very buggy / incorrect because the grammers are just simply not that well tested (nobody is testing them in their editor daily). The annoyance with TextMate grammers is that they are based on Oniguruma Regular Expressions, which means that while they are written in very powerful regex syntax, you cannot really use them with any regex engine built in to most languages today. Almost all editors directly use the original C implementaton of Oniguruma, even editors like Visual Studio Code which are based on web tech primarily. And even once you have a working Oniguruma regex engine, it is still very non-trivial to use it for highlighting actual code because the TextMate files themselves are pretty complex on their own. To make matters more interesting, Sublime created their own language grammer format (`.sublime-syntax` files), which is a superset of the TextMate language grammers and supports more features. Sublime can convert any TextMate language grammer into the new .sublime-syntax format. So when I started seriously looking into syntax highlighting about a year ago, the Pygments syntax highlighter was my first choice and it is pretty good, but I found it to be lacking support for languages we cared about such as TypeScript at the time. Additionally, Pygments was pretty slow when we tried it out. GitHub switched away from it for a similar reason[2] This was about the time that I ran into the Syntect[3] project. For us, this had huge benefits: - It was by far the fastest syntax highlighter we found, which meant page load times improved a lot for us. - Its results were really, really good. I found multiple cases where the output was better than other highlighters such as Pygments. - The built-in language support, all language Sublime Text 3 supports by default, was great out of the box. - Because Sublime can convert TextMate grammers to .sublime-text grammers, we ultimately get support for basically all languages. - It took me _one day_ to write `syntect_server`[4] which highlights arbitrary code for our Go application, which was a spectacular turnaround time given how much time I'd already invested elsewhere in researching all of this. Overall, we really couldn't be more pleased with Syntect and Rust for modern syntax highlighting. It's a beautiful, fast, high-quality combination. [1] https://macromates.com/manual/en/language_grammars https://macromates.com/manual/en/language_grammars [2] https://github.com/github/linguist/issues/1717#issuecomment-63681177 https://github.com/github/linguist/issues/1717#issuecomment-... [3] https://github.com/trishume/syntect https://github.com/trishume/syntect [4] https://github.com/sourcegraph/syntect_server https://github.com/sourcegraph/syntect_server
- euyyn 8y ago
  nit, but it affected me: grammar
- CupOfJava 8y ago
  Well done! Thank you for the write up. How much did it improve load times? Why do you think syntect was faster than what you tried previously?
  - slimsag 8y ago
    I don't have hard numbers on me at the moment, but I remember it going from roughly ~2-4s with Highlight.js on some poor file cases down to ~500ms with Syntect. From what I remember, Pygments was better than Highlight.js by a factor of roughly ~2x, but with Syntect still outperforming it significantly in the general cases. I think Syntect is faster in general just because of the nature of the work (syntax highlighting is basically just a stack machine + running regex 24/7) and Rust being a really performant choice for this in comparison to Python/JS (although I am sure some less idiomatic code could improve the perf of a Python/JS implementation). Hopefully one day I'll do a proper write-up / blog post on this topic and provide some more hard numbers / stats for everyone. :)

by sqs 8y ago

slimsag 8y ago

Sourcegraph dev here :) At Sourcegraph we've actually tried a lot of different syntax highlighters in the past. Syntax highlighting is actually a _really_ hard problem. Syntax highlighting for basically all editors today is provided by TextMate language grammers[1] (`.tmLanguage` files) ultimately, which are just giant regex definition files, basically. These are the ubiquitous files for syntax highlighting, and you can almost always find a TextMate grammer for any programming language. Some 'modern' syntax highlighters, mostly ones designed for use in the web, try not to use these TextMate files -- but in doing so they lose support for TONS of languages, and the languages that they do support are often very buggy / incorrect because the grammers are just simply not that well tested (nobody is testing them in their editor daily). The annoyance with TextMate grammers is that they are based on Oniguruma Regular Expressions, which means that while they are written in very powerful regex syntax, you cannot really use them with any regex engine built in to most languages today. Almost all editors directly use the original C implementaton of Oniguruma, even editors like Visual Studio Code which are based on web tech primarily. And even once you have a working Oniguruma regex engine, it is still very non-trivial to use it for highlighting actual code because the TextMate files themselves are pretty complex on their own. To make matters more interesting, Sublime created their own language grammer format (`.sublime-syntax` files), which is a superset of the TextMate language grammers and supports more features. Sublime can convert any TextMate language grammer into the new .sublime-syntax format. So when I started seriously looking into syntax highlighting about a year ago, the Pygments syntax highlighter was my first choice and it is pretty good, but I found it to be lacking support for languages we cared about such as TypeScript at the time. Additionally, Pygments was pretty slow when we tried it out. GitHub switched away from it for a similar reason[2] This was about the time that I ran into the Syntect[3] project. For us, this had huge benefits: - It was by far the fastest syntax highlighter we found, which meant page load times improved a lot for us. - Its results were really, really good. I found multiple cases where the output was better than other highlighters such as Pygments. - The built-in language support, all language Sublime Text 3 supports by default, was great out of the box. - Because Sublime can convert TextMate grammers to .sublime-text grammers, we ultimately get support for basically all languages. - It took me _one day_ to write `syntect_server`[4] which highlights arbitrary code for our Go application, which was a spectacular turnaround time given how much time I'd already invested elsewhere in researching all of this. Overall, we really couldn't be more pleased with Syntect and Rust for modern syntax highlighting. It's a beautiful, fast, high-quality combination. [1] https://macromates.com/manual/en/language_grammars https://macromates.com/manual/en/language_grammars [2] https://github.com/github/linguist/issues/1717#issuecomment-63681177 https://github.com/github/linguist/issues/1717#issuecomment-... [3] https://github.com/trishume/syntect https://github.com/trishume/syntect [4] https://github.com/sourcegraph/syntect_server https://github.com/sourcegraph/syntect_server

euyyn 8y ago
nit, but it affected me: grammar
CupOfJava 8y ago
Well done! Thank you for the write up. How much did it improve load times? Why do you think syntect was faster than what you tried previously?
- slimsag 8y ago
  I don't have hard numbers on me at the moment, but I remember it going from roughly ~2-4s with Highlight.js on some poor file cases down to ~500ms with Syntect. From what I remember, Pygments was better than Highlight.js by a factor of roughly ~2x, but with Syntect still outperforming it significantly in the general cases. I think Syntect is faster in general just because of the nature of the work (syntax highlighting is basically just a stack machine + running regex 24/7) and Rust being a really performant choice for this in comparison to Python/JS (although I am sure some less idiomatic code could improve the perf of a Python/JS implementation). Hopefully one day I'll do a proper write-up / blog post on this topic and provide some more hard numbers / stats for everyone. :)