7 ms·

Optimizations in Syntax Highlighting

257 points by riejo 10y ago

mrgalaxy 10y ago
> there is no feasible way to interpret TextMate grammars in the browser even today That doesn't sound right... but then again I don't know enough about TextMate grammars to argue.
- mikewhy 10y ago
  Yeah I'm not totally sure what was meant by that. They're plain text, thus parsable. Maybe they meant that browsers don't usually have access to the file system, but that's changing and also not applicable since they're using Electron and have NodeJS at their disposal.
  - lanna 10y ago
    plain text doesn't imply parsable; natural languages, for instance.
- trishume 10y ago
  The reason is that they basically rely on using the Oniguruma regret engine, and reimplementing that in JS would be hella slow.
  - algorithmsRcool 10y ago
    > using the Oniguruma regret engine I'm surprised I've never seen that typo for regex before. It's wonderful.
    - WildUtah 10y ago
      Sometimes you feel a regret. "I know!" you say, "I'll fix things with regretular expressions." Now you have two regrets.
      - nickpeterson 10y ago
        Maybe "I'll fix this with expressions of regret."
    - trishume 10y ago
      lol I typed that on an IOS device correctly but it autocompleted it since "regex" isn't a real word apparently.
      - Sean1708 10y ago
        Well no, it's a portmanteau of "regular" and "expression".
- alexdima 10y ago
  - all the regular expressions in TM grammars are based on oniguruma, a regular expression library written in C. - the only way to interpret the grammars and get anywhere near original fidelity is to use the exact same regular expression library (with its custom syntax constructs) in VSCode, our runtime is node.js and we can use a node native module that exposes the library to JavaScript - in the Monaco Editor, we are constrained to a browser environment where we cannot do anything similar - we have experimented with Emscripten to compile the C library to asm.js, but performance was very poor even in Firefox (10x slower) and extremely poor in Chrome (100x slower). - we can revisit this once WebAssembly gets traction in the major browsers, but we will still need to consider the browser matrix we support. i.e. if we support IE11 and only Edge will add WebAssembly support, what will the experience be in IE11, etc.
  - alimbada 10y ago
    Sorry if I'm missing something, but why do you care about supporting other browsers? Isn't VSCode built on Electron which is a self contained server/browser environment?
    - eduren 10y ago
      Because the base Monaco editor is used in other products besides Code. The example I've seen thrown around is configuration editing in Azure consoles.
    - wolfgang42 10y ago
      Monaco is a standalone component which also works in the browser: https://microsoft.github.io/monaco-editor/ https://microsoft.github.io/monaco-editor/ The article says "It was shipped in the form of the Monaco Editor in various Microsoft projects, including Internet Explorer's F12 tools"; presumably some of those projects also embed it into webpages.
  - gcp 10y ago
    Maybe slightly offtopic, but is there any chance we could see a VSCode based on Edge/Chakra? I prefer your font rendering over your competitions', and that's a pretty big issue for an editor.
- jbmorgado 10y ago
  What I don't understand is why do we still use TextMate grammars. Can anyone explain why this is the case? It's not only in VSCode, I remember seeing something about TextMate grammars also in other editors.
  - trishume 10y ago
    Copy-pasted from a comment I made elsewhere: It's harder than it sounds if you want to support many languages. The Sublime syntaxes repo I use has 34,000 lines of grammars whereas my engine is only 3000 lines of code. If you count all the tmLanguage files for nice languages available online it's probably hundreds of thousands of lines, and that's in a pretty dense format. The whole point of using tmLanguage files is that people don't care about how fast other languages are if there is no highlighting for their language. I could get way better performance by rewriting all those grammars using compiled parsers in Rust (like Xi has as an option https://github.com/google/xi-editor https://github.com/google/xi-editor) but it would take an absurd amount of effort.
    - jbmorgado 10y ago
      Thank you, that explains a lot, but now a new doubt came from it: Why don't we use other text editors grammars that are simpler/quicker to parse in JS? I have no idea on the technicalities, but for instance, Vim or Emacs grammars instead?
      - okdana 10y ago
        I can't speak to Emacs, but i know that Vim's syntax definitions are (a) not as powerful as TM's, (b) a nightmare to maintain, and (c) heavily reliant on features specific to Vim's loony regular-expression engine (like variable-width look-around). My experience is that most syntax highlighters and their definition formats (Scintilla, nano, almost anything that's based on JavaScript) are very limited/naïve compared to TextMate's. It doesn't have to be that way -- TM can certainly be improved upon -- but it is.
trishume 10y ago
Shameless plug: my implementation of Sublime's syntax highlighting engine in Rust has similar optimizations and more. I'm not at my computer to benchmark on the same files but it should be >2x as fast as their "after" numbers just based on lines/second for JS-like files. This evening I'm even trying to port it to a pure Rust regex engine that should eliminate non-Rust code and make it substantially faster. It also implements the sublime-syntax format which is a superset of tmlanguage that allows even nicer highlighting. https://github.com/trishume/syntect https://github.com/trishume/syntect
- octref 10y ago
  Sorry but I'm not familiar with Rust -- is it possible to run your lib in all platforms in a Node environment? And, do you have an issue for the regex port? Would love to see the benchmark.
  - trishume 10y ago
    Yes it would be possible to write a C wrapper for my Rust library and link that from Node, however I expect you might lose a lot of performance to data conversion. I have yet to do the regex engine port, I'm doing that later this evening. I'm porting it from Oniguruma to https://github.com/google/fancy-regex https://github.com/google/fancy-regex which accelerates common types of regexes using the awesome Rust regex crate which is super fast.
    - dgfgfdagasdfgfa 10y ago
      Doesn't rust export the C abi? Or does the "c wrapper" just involve converting c types to rust domain types>
      - kevincox 10y ago
        You can very easily export a C api from rust. And fact you would have to if you wanted to write a wrapper anyways so I think the wrapper is probably unnecessary.
pjmlp 10y ago
Great article, specially regarding the type of optimisations that were applied.
eduren 10y ago
So with the comparisons at the end of the article, does this mean that there were a lot of edge cases where the theming wasn't being correctly applied prior to 1.9? Were there themes that incorporated less stylistic choices because of the limitations?
- alexdima 10y ago
  Yes, those comparisons at the end show differences in rendering caused by the "approximations" used prior to VS Code 1.9. They were all caused by the difference between the ranking rules of CSS selectors and the ranking rules of TM scope selectors
nojvek 10y ago
I once dabbled inside vscode tokenizer code. There is a lot going but putting tokens in a buffer took me by a surprise. It was a very smart implementation. I think it's their focus on performance and good architecture that makes vscode stand out. Sure sublime also has great engineering behind it, but being able to contribute and look under the hood of tools, we developers use is very exciting. It feels like a very democratic process. I remember filing the mini-map bug in Monaco, I'm so glad to see they are working on implementing it in a performant way even though it will require large rewrites of their editor rendering code.
Animats 10y ago
(Syntax highlighting) is the one feature that turns a text editor into a code editor. Syntax highlighting is eye candy. Automatic indentation is what turns a text editor into a code editor.
- farnsworth 10y ago
  Automatic indentation saves a few keystrokes. A languages service (go to definition, etc) is what turns a text editor into a code editor!
  - whatever_dude 10y ago
    A language service is for people with poor memory. A terminal window is what turns a text editor into a code editor.
    - flamedoge 10y ago
      I need OOP navigator to use C++/Java/C#
      - hex13 10y ago
        I think support for butterflies is what turns a text editor into a code editor (xkcd#378) XD
iveqy 10y ago
So VSCode is great in many ways, and the article might be interesting. But I would never call it fast. It's still really really slow. Just see this comparision: https://www.youtube.com/watch?v=nDRBxtEUOFE https://www.youtube.com/watch?v=nDRBxtEUOFE
- zitterbewegung 10y ago
  The article only makes the claim that they have made it faster that the previous technique. Most people say VSCode is faster than other electron editors like Atom. I am not sure who has said that VSCode is faster than Vim and it would be safe to assume that Vim would be faster.
  - nerdponx 10y ago
    Vim syntax highlighting is painfully slow on long lines, which makes it unusable as a Markdown or Tex editor unless you enable automatic explicit line breaking at 80 or 150 characters, instead of letting paragraphs wrap naturally.
- trishume 10y ago
  That's not a perfectly fair comparison. Vim's syntaxes are often super simple and do a much less nice job at highlighting than most tmLanguage syntaxes. Also all that video tests for is the presence of an optimization where it updates the on-screen colours as soon as that part of the file is done instead of after the entire file is done. It tells nothing about the underlying speed of the highlighting engines. Perhaps an important optimization, but not much information here.
- alexdima 10y ago
  When we started the project, we did write tokenizers by hand. I mention that in the blog post. You can write some very fast tokenizers by hand, even in JavaScript. Of course they won't be as fast as hand written tokenizers in C, but you'd be surprised how well the code of a hand written tokenizer in JavaScript can be optimized by a JS engine, at least I was :). IR Hydra 2 is a great tool to visualise v8's IR representation of JS code [1]. It is a shame it is not built into the Chrome Dev Tools. In the end, we simply could not write tokenizers for all languages by hand. And our users wanted to take their themes with them when switching to VS Code. That's why we added support for TM grammars and TM themes, and in hindsight I still consider it to be a very smart decision. [1] http://mrale.ph/irhydra/2/ http://mrale.ph/irhydra/2/
joshschreuder 10y ago
Those who switched from ST to VS Code, did you stick with VS Code? Do you have any advice for making the transition easier - key bindings, packages etc.?
- bicubic 10y ago
  I stuck with VS Code. To be honest, I don't think it's even in the same league as ST or Atom due to the amount of integrated tooling. The integrated debugger, git, and task runner management is a god send. VS Code is also the only editor I've used where javascript type lookups and auto completion/code doc parsing Just Works out of the box.
- laurencerowe 10y ago
  I've switched over in the past couple of months. The motivating factor for me was my switch to writing more JS (the debugger works really well.) But I'm increasingly finding myself using it for everything I used to use Sublime for. It's come a long way in the past year or so since I last gave it a try.
ex3ndr 10y ago
Does VS Code support highlighting for "non-regexp" cases. For example: in code i can reference a Class by it's name, but in the same time if could be a function - how you can distinct one from another by just regexp when you don't use capitalization marker for class names? Some times Class is an Object in some languages (Scala, Kotlin...), how this case is handled?

257 points by riejo 10y ago

> there is no feasible way to interpret TextMate grammars in the browser even today That doesn't sound right... but then again I don't know enough about TextMate grammars to argue.

mikewhy 10y ago
Yeah I'm not totally sure what was meant by that. They're plain text, thus parsable. Maybe they meant that browsers don't usually have access to the file system, but that's changing and also not applicable since they're using Electron and have NodeJS at their disposal.
- lanna 10y ago
  plain text doesn't imply parsable; natural languages, for instance.
trishume 10y ago
The reason is that they basically rely on using the Oniguruma regret engine, and reimplementing that in JS would be hella slow.
- algorithmsRcool 10y ago
  > using the Oniguruma regret engine I'm surprised I've never seen that typo for regex before. It's wonderful.
  - WildUtah 10y ago
    Sometimes you feel a regret. "I know!" you say, "I'll fix things with regretular expressions." Now you have two regrets.
    - nickpeterson 10y ago
      Maybe "I'll fix this with expressions of regret."
  - trishume 10y ago
    lol I typed that on an IOS device correctly but it autocompleted it since "regex" isn't a real word apparently.
    - Sean1708 10y ago
      Well no, it's a portmanteau of "regular" and "expression".
alexdima 10y ago
- all the regular expressions in TM grammars are based on oniguruma, a regular expression library written in C. - the only way to interpret the grammars and get anywhere near original fidelity is to use the exact same regular expression library (with its custom syntax constructs) in VSCode, our runtime is node.js and we can use a node native module that exposes the library to JavaScript - in the Monaco Editor, we are constrained to a browser environment where we cannot do anything similar - we have experimented with Emscripten to compile the C library to asm.js, but performance was very poor even in Firefox (10x slower) and extremely poor in Chrome (100x slower). - we can revisit this once WebAssembly gets traction in the major browsers, but we will still need to consider the browser matrix we support. i.e. if we support IE11 and only Edge will add WebAssembly support, what will the experience be in IE11, etc.
- alimbada 10y ago
  Sorry if I'm missing something, but why do you care about supporting other browsers? Isn't VSCode built on Electron which is a self contained server/browser environment?
  - eduren 10y ago
    Because the base Monaco editor is used in other products besides Code. The example I've seen thrown around is configuration editing in Azure consoles.
  - wolfgang42 10y ago
    Monaco is a standalone component which also works in the browser: https://microsoft.github.io/monaco-editor/ https://microsoft.github.io/monaco-editor/ The article says "It was shipped in the form of the Monaco Editor in various Microsoft projects, including Internet Explorer's F12 tools"; presumably some of those projects also embed it into webpages.
- gcp 10y ago
  Maybe slightly offtopic, but is there any chance we could see a VSCode based on Edge/Chakra? I prefer your font rendering over your competitions', and that's a pretty big issue for an editor.
jbmorgado 10y ago
What I don't understand is why do we still use TextMate grammars. Can anyone explain why this is the case? It's not only in VSCode, I remember seeing something about TextMate grammars also in other editors.
- trishume 10y ago
  Copy-pasted from a comment I made elsewhere: It's harder than it sounds if you want to support many languages. The Sublime syntaxes repo I use has 34,000 lines of grammars whereas my engine is only 3000 lines of code. If you count all the tmLanguage files for nice languages available online it's probably hundreds of thousands of lines, and that's in a pretty dense format. The whole point of using tmLanguage files is that people don't care about how fast other languages are if there is no highlighting for their language. I could get way better performance by rewriting all those grammars using compiled parsers in Rust (like Xi has as an option https://github.com/google/xi-editor https://github.com/google/xi-editor) but it would take an absurd amount of effort.
  - jbmorgado 10y ago
    Thank you, that explains a lot, but now a new doubt came from it: Why don't we use other text editors grammars that are simpler/quicker to parse in JS? I have no idea on the technicalities, but for instance, Vim or Emacs grammars instead?
    - okdana 10y ago
      I can't speak to Emacs, but i know that Vim's syntax definitions are (a) not as powerful as TM's, (b) a nightmare to maintain, and (c) heavily reliant on features specific to Vim's loony regular-expression engine (like variable-width look-around). My experience is that most syntax highlighters and their definition formats (Scintilla, nano, almost anything that's based on JavaScript) are very limited/naïve compared to TextMate's. It doesn't have to be that way -- TM can certainly be improved upon -- but it is.

trishume 10y ago

Shameless plug: my implementation of Sublime's syntax highlighting engine in Rust has similar optimizations and more. I'm not at my computer to benchmark on the same files but it should be >2x as fast as their "after" numbers just based on lines/second for JS-like files. This evening I'm even trying to port it to a pure Rust regex engine that should eliminate non-Rust code and make it substantially faster. It also implements the sublime-syntax format which is a superset of tmlanguage that allows even nicer highlighting. https://github.com/trishume/syntect https://github.com/trishume/syntect

octref 10y ago
Sorry but I'm not familiar with Rust -- is it possible to run your lib in all platforms in a Node environment? And, do you have an issue for the regex port? Would love to see the benchmark.
- trishume 10y ago
  Yes it would be possible to write a C wrapper for my Rust library and link that from Node, however I expect you might lose a lot of performance to data conversion. I have yet to do the regex engine port, I'm doing that later this evening. I'm porting it from Oniguruma to https://github.com/google/fancy-regex https://github.com/google/fancy-regex which accelerates common types of regexes using the awesome Rust regex crate which is super fast.
  - dgfgfdagasdfgfa 10y ago
    Doesn't rust export the C abi? Or does the "c wrapper" just involve converting c types to rust domain types>
    - kevincox 10y ago
      You can very easily export a C api from rust. And fact you would have to if you wanted to write a wrapper anyways so I think the wrapper is probably unnecessary.

pjmlp 10y ago

Great article, specially regarding the type of optimisations that were applied.

eduren 10y ago

So with the comparisons at the end of the article, does this mean that there were a lot of edge cases where the theming wasn't being correctly applied prior to 1.9? Were there themes that incorporated less stylistic choices because of the limitations?

alexdima 10y ago
Yes, those comparisons at the end show differences in rendering caused by the "approximations" used prior to VS Code 1.9. They were all caused by the difference between the ranking rules of CSS selectors and the ranking rules of TM scope selectors

nojvek 10y ago

I once dabbled inside vscode tokenizer code. There is a lot going but putting tokens in a buffer took me by a surprise. It was a very smart implementation. I think it's their focus on performance and good architecture that makes vscode stand out. Sure sublime also has great engineering behind it, but being able to contribute and look under the hood of tools, we developers use is very exciting. It feels like a very democratic process. I remember filing the mini-map bug in Monaco, I'm so glad to see they are working on implementing it in a performant way even though it will require large rewrites of their editor rendering code.

Animats 10y ago

(Syntax highlighting) is the one feature that turns a text editor into a code editor. Syntax highlighting is eye candy. Automatic indentation is what turns a text editor into a code editor.

farnsworth 10y ago
Automatic indentation saves a few keystrokes. A languages service (go to definition, etc) is what turns a text editor into a code editor!
- whatever_dude 10y ago
  A language service is for people with poor memory. A terminal window is what turns a text editor into a code editor.
  - flamedoge 10y ago
    I need OOP navigator to use C++/Java/C#
    - hex13 10y ago
      I think support for butterflies is what turns a text editor into a code editor (xkcd#378) XD

iveqy 10y ago

So VSCode is great in many ways, and the article might be interesting. But I would never call it fast. It's still really really slow. Just see this comparision: https://www.youtube.com/watch?v=nDRBxtEUOFE https://www.youtube.com/watch?v=nDRBxtEUOFE

zitterbewegung 10y ago
The article only makes the claim that they have made it faster that the previous technique. Most people say VSCode is faster than other electron editors like Atom. I am not sure who has said that VSCode is faster than Vim and it would be safe to assume that Vim would be faster.
- nerdponx 10y ago
  Vim syntax highlighting is painfully slow on long lines, which makes it unusable as a Markdown or Tex editor unless you enable automatic explicit line breaking at 80 or 150 characters, instead of letting paragraphs wrap naturally.
trishume 10y ago
That's not a perfectly fair comparison. Vim's syntaxes are often super simple and do a much less nice job at highlighting than most tmLanguage syntaxes. Also all that video tests for is the presence of an optimization where it updates the on-screen colours as soon as that part of the file is done instead of after the entire file is done. It tells nothing about the underlying speed of the highlighting engines. Perhaps an important optimization, but not much information here.
alexdima 10y ago
When we started the project, we did write tokenizers by hand. I mention that in the blog post. You can write some very fast tokenizers by hand, even in JavaScript. Of course they won't be as fast as hand written tokenizers in C, but you'd be surprised how well the code of a hand written tokenizer in JavaScript can be optimized by a JS engine, at least I was :). IR Hydra 2 is a great tool to visualise v8's IR representation of JS code [1]. It is a shame it is not built into the Chrome Dev Tools. In the end, we simply could not write tokenizers for all languages by hand. And our users wanted to take their themes with them when switching to VS Code. That's why we added support for TM grammars and TM themes, and in hindsight I still consider it to be a very smart decision. [1] http://mrale.ph/irhydra/2/ http://mrale.ph/irhydra/2/

joshschreuder 10y ago

Those who switched from ST to VS Code, did you stick with VS Code? Do you have any advice for making the transition easier - key bindings, packages etc.?

bicubic 10y ago
I stuck with VS Code. To be honest, I don't think it's even in the same league as ST or Atom due to the amount of integrated tooling. The integrated debugger, git, and task runner management is a god send. VS Code is also the only editor I've used where javascript type lookups and auto completion/code doc parsing Just Works out of the box.
laurencerowe 10y ago
I've switched over in the past couple of months. The motivating factor for me was my switch to writing more JS (the debugger works really well.) But I'm increasingly finding myself using it for everything I used to use Sublime for. It's come a long way in the past year or so since I last gave it a try.

ex3ndr 10y ago

Does VS Code support highlighting for "non-regexp" cases. For example: in code i can reference a Class by it's name, but in the same time if could be a function - how you can distinct one from another by just regexp when you don't use capitalization marker for class names? Some times Class is an Object in some languages (Scala, Kotlin...), how this case is handled?