5 ms·

I'm not trying to bash Emacs or treesitter or anyone. But I find it mildly amusing that after so many decades, parsing and syntax highlighting aren't a perfectl

I'm not trying to bash Emacs or treesitter or anyone. But I find it mildly amusing that after so many decades, parsing and syntax highlighting aren't a perfectly solved problem, considering programming languages are the most used tools for developers.

thih9 3y ago
What would you consider a perfectly solved problem in this case? I.e. how is current development experience bad and how it could be better?
- raincole 3y ago
  In a perfect world of emacs/tree-sitter, I would imagine tree-sitter to be a single minor mode. You don't need 'java-ts-mode' and 'c-ts-mode'. Just a single 'treesitter-mode' toggle and it will call the treesitter binary to do the hard lifting. I guess the reality just isn't so rosy.
  - roenxi 3y ago
    A `treesitter-mode` is a so-so idea, because the way different languages are interacted with is different. The way lisp sexp handle in an IDE is different from Python's syntax. It doesn't make sense to try and have one mode that handles both - it is better to have many modes that interact with the common data structures that treesitter provides and then starts to provide specific convenience functions. Even if all it was supposed to do is syntax highlighting, different languages are usually highlighted differently and presumably the person maintianing the bindings would need to be a language expert with strong opinions. The options are they either maintain their own mode or a module in `treesitter-mode` which is basically 2 ways of describing the same situation. Although to address your other point it is quite funny that after 40 years and 29 versions of experimenting in OS design, GNU Emacs is starting to implement syntax highlighting properly for its system text editor.
    - PurpleRamen 3y ago
      A treesitter-mode would make sense for organizational reasons and user-experience. But under the hood it would be just a proxy-mode which figures out the language and load the appropriate treesitter-sub-modes. And maybe in a more advanced version it could also mix modes for different languages.
      - roenxi 3y ago
        But you're describing Emacs. Emacs figures out the language and loads the appropriate sub-modes. It even calls them modes. Modes are mixable. We're in a transition period while everything is rewritten to use tree-sitter. In a few years all the default major modes in Emacs are probably going to be tree-sitter based - unless the maintainers believe them to be better than tree-sitter.
        PurpleRamen 3y ago
        > Emacs figures out the language and loads the appropriate sub-modes. Not really. You setup hooks, usually based by fileextension, and emacs execute the hooks. Emacs itself has no understanding of languages. And this will not work well when there is no hook executed. > Modes are mixable. You can load multiple minor-modes, but there is always just one major-mode per buffer, and languages are usually major-modes. But ok, treesitter can be handled as a minor. But it still needs explicit support. You cannot give treesitter control over a range of text, let if figure out the language automatically and let it make it's thing. This needs explicit support.
    - pulpfictional 3y ago
      > GNU Emacs is starting to implement syntax highlighting properly for its system text editor What do you mean?
      - roenxi 3y ago
        It is a classic joke. Emacs is often a little behind competing text editors (Vim has better keybindings IMO, things like Visual studio have better IDE support, Atom text editor turns out to have better syntax highlighting, etc, etc). So, the line of the joke goes, it must not really be a text editor because it is not very good at it and has a wild array of other capabilities. Like "M-x dunnet". Must be an OS. Of course the reality is that Emacs just copies good ideas from other people. A pretty normal Emacs setup uses Vim keybindings, LSP and now tree-sitter to get a good experience programming.
- craftkiller 3y ago
  I'm not the GP, but personally I'd consider syntax highlighting perfectly solved if it was handled in the language server. That way we could have highlighting performed by 100% accurate parsers instead of approximations. It wouldn't be as fast as in-editor simple tree-sitter highlighting, but I don't see any issue with my text appearing uncolored at first and gaining color as my editor polls the language server as I type. Currently language servers support semantic highlighting which is close, but as far as I can tell, it seems meant to be supplemental to in-editor local highlighting rather than a full replacement.
  - pawelmurias 3y ago
    You would only get the top half of your code highlighted because a compiler doesnt usually continue when encountering syntax errors due to partial code
    - craftkiller 3y ago
      That's sort-of already the case whenever we start a string in any language that supports multi-line strings. You type the first " and then the rest of your file gets re-interpreted as a string. It is annoying but I still think its a worthy trade-off, because I spend a lot more time reading code than being mid-edit with invalid syntax.
      - troupo 3y ago
        Syntax errors are not limited to just typing a quote
        craftkiller 3y ago
        I never said they were. I'm just pointing out that we already experience the same problem in the case of quotes without the world ending, and I don't think expanding the problem to all syntax errors will make it much worse.
        3y ago
troupo 3y ago
The problem is that a good editor-compatible tool for parsing and syntax highlighting is at cross-purposes with what you want from a compiler. A good overview here: https://matklad.github.io/2022/04/25/why-lsp.html https://matklad.github.io/2022/04/25/why-lsp.html
- aidos 3y ago
  Interestingly, treesitter is designed for this space (parsing invalid structures with a time component) but it’s still not used as a base for LSPs. I once asked why on HN and people that know more than me said it wasn’t suitable.
  - vkazanov 3y ago
    tree-sitter definitely can be used as a incremental parser for an LSP. But the original purpose was and still is to provide a standard format for an editor-agnostic way to parse languages. As somebody who wrote a tree-sitter grammar recently I can confirm that the library definitely has its share of... Interesting choices. But there's nothing else like it as most parser generators are non-incremental, don't do generalized parsing and don't provide decent error recovery.
    - bch 3y ago
      > has its share of… Interesting choices Curious to hear more.
      - vkazanov 3y ago
        Well, nothing truly horrible but here are things I find inconvenient (but working nevertheless) when using the sdk. 1. NodeJS-based utilities. The tooling around js doesn't lean itself well to integration with with the dev's environment. 2. The sdk itself is a mix of Rust, javascript and C. I'd say that this is at least one language too many - makes it harder to contribute. Ideally such fundamental projects should be as simple and homogeneous as possible. 3. I keep wondering if generating languages other than C should be supported. The core idea of the library is brilliant either way so it sort of compensates all of the above.
kristopolous 3y ago
The languages extension strategies for CSS pretifiers are usually pretty reasonable. In editors this always seems extremely esoteric comparatively: I've tried doing it in a few. I'm sure brilliant people find it easy, but I'm merely average on a good day. I haven't tried extending any of these modern electron based editors, can anyone speak to that?
petepete 3y ago
Until TreeSitter came along the effort to add support for new languages to your editor would be gargantuan. Now it's much, much easier providing there's a TreeSitter parser for your language. I don't know of anything else that bridges the gap like this.
- tgv 3y ago
  Note that not that long ago, computers were simply not powerful enough to run that in acceptable time. Dedicated parsers can be much faster.
mickeyp 3y ago
I clarified exactly why it is the way it is here: https://www.masteringemacs.org/article/tree-sitter-complications-of-parsing-languages https://www.masteringemacs.org/article/tree-sitter-complicat... And also why using LSP to furnish your editor with highlight markers is an inelegant solution for many languages.
- deleted 3y ago
  [deleted]
vidarh 3y ago
Parsing of a correct program is a pretty "solved" problem. But fast enough re-parsing of fragments and recovery from errors is a much more complex problem, that often doesn't have a single correct answer, and it's also a much newer problem in as much as syntax-highlighting is much newer feature, being preceded largely by "offline" pretty-printers with very different constraints. The extent to which modern compilers try to parse past errors still varies greatly, with a whole lot not even trying to. But just any recovering parser also does not mean the problem is solved. E.g. you've typed "foo". Now you type "(". It'd be very annoying if your editor now re-colors everything as an error, so you typically want some error recovery. But how soon? Do you assume the tokens immediate afterwards are par of what was a valid expression until you typed "foo", or are they a valid part of an argument list? And where do they end? Do you just delay re-parsing until the user has typed more? Or left the line? Sometimes that can help, sometimes it will just make things worse. Parsing methods that work fine if you assume you can "reset" the parse at many different points which tend to constrain the area considered an error and so reducing the size of a typical re-parse will fail badly if you want stricter re-parsing that frequently may trigger reparsing most of a file, for example. A lot of this is subjective, and picking the "right" way of handling it largely comes down to unpacking humans unstated preferences, and trying to reconcile competing and possibly contradictory preferences.
- aeonik 3y ago
  I wonder why there aren't more tools to allow structural editing like Lisp has. This way the file can never be in an unparsable state.
  - ghosty141 3y ago
    This is what treesitter enables!
  - nequo 3y ago
    You may be aware but the author of TFA also has a tree-sitter based minor mode called Combobulate for exactly that: https://www.masteringemacs.org/article/combobulate-structured-movement-editing-treesitter https://www.masteringemacs.org/article/combobulate-structure... There is also evil-textobj-tree-sitter for tree-sitter based text objects for Evil mode: https://github.com/meain/evil-textobj-tree-sitter https://github.com/meain/evil-textobj-tree-sitter
PurpleRamen 3y ago
There will never be "perfectly" solved problems at that complexity-level. There are always changing requirements and space for improvement. Make it faster, add new features, use new hardware-abilities, follow the flavors of this decade, this is an eternally going game of catching up.
olau 3y ago
It's a tough problem. Steve Yegge blogged about the complexities involved when he wrote js2-mode: https://steve-yegge.blogspot.com/2008/03/js2-mode-new-javascript-mode-for-emacs.html https://steve-yegge.blogspot.com/2008/03/js2-mode-new-javasc... I guess comp. sci. people studying languages have been more interested in syntactically valid programs than the opposite.
- mbork_pl 3y ago
  Well, I bet most syntactically valid programs are not interested in comp-sci people!
mbork_pl 3y ago
Well, there are more problems like that. You'd think diffing is a solved problem, and yet we still struggle with syntax-aware diffs (I use difftastic, which is great, but doesn't always work well, and is under constant development).
- tuukkah 3y ago
  Interesting, it's also based on parsing by Tree-sitter: https://difftastic.wilfred.me.uk/parsing.html https://difftastic.wilfred.me.uk/parsing.html
nerdponx 3y ago
This is that. Tree Sitter has become one of the foundational advances that is allowing us to make progress on solving that problem.
hardwaregeek 3y ago
It's not really a solved problem in general. Most editors appear to use TextMate grammars which nobody likes. Otherwise you have to implement it using whatever custom setup your specific editor uses. It just happens that most languages have some poor soul who set this up already. Emacs is actually on the better side because tree-sitter is a much better setup for writing grammars.

thih9 3y ago

What would you consider a perfectly solved problem in this case? I.e. how is current development experience bad and how it could be better?

raincole 3y ago
In a perfect world of emacs/tree-sitter, I would imagine tree-sitter to be a single minor mode. You don't need 'java-ts-mode' and 'c-ts-mode'. Just a single 'treesitter-mode' toggle and it will call the treesitter binary to do the hard lifting. I guess the reality just isn't so rosy.
- roenxi 3y ago
  A `treesitter-mode` is a so-so idea, because the way different languages are interacted with is different. The way lisp sexp handle in an IDE is different from Python's syntax. It doesn't make sense to try and have one mode that handles both - it is better to have many modes that interact with the common data structures that treesitter provides and then starts to provide specific convenience functions. Even if all it was supposed to do is syntax highlighting, different languages are usually highlighted differently and presumably the person maintianing the bindings would need to be a language expert with strong opinions. The options are they either maintain their own mode or a module in `treesitter-mode` which is basically 2 ways of describing the same situation. Although to address your other point it is quite funny that after 40 years and 29 versions of experimenting in OS design, GNU Emacs is starting to implement syntax highlighting properly for its system text editor.
  - PurpleRamen 3y ago
    A treesitter-mode would make sense for organizational reasons and user-experience. But under the hood it would be just a proxy-mode which figures out the language and load the appropriate treesitter-sub-modes. And maybe in a more advanced version it could also mix modes for different languages.
    - roenxi 3y ago
      But you're describing Emacs. Emacs figures out the language and loads the appropriate sub-modes. It even calls them modes. Modes are mixable. We're in a transition period while everything is rewritten to use tree-sitter. In a few years all the default major modes in Emacs are probably going to be tree-sitter based - unless the maintainers believe them to be better than tree-sitter.
      - PurpleRamen 3y ago
        > Emacs figures out the language and loads the appropriate sub-modes. Not really. You setup hooks, usually based by fileextension, and emacs execute the hooks. Emacs itself has no understanding of languages. And this will not work well when there is no hook executed. > Modes are mixable. You can load multiple minor-modes, but there is always just one major-mode per buffer, and languages are usually major-modes. But ok, treesitter can be handled as a minor. But it still needs explicit support. You cannot give treesitter control over a range of text, let if figure out the language automatically and let it make it's thing. This needs explicit support.
  - pulpfictional 3y ago
    > GNU Emacs is starting to implement syntax highlighting properly for its system text editor What do you mean?
    - roenxi 3y ago
      It is a classic joke. Emacs is often a little behind competing text editors (Vim has better keybindings IMO, things like Visual studio have better IDE support, Atom text editor turns out to have better syntax highlighting, etc, etc). So, the line of the joke goes, it must not really be a text editor because it is not very good at it and has a wild array of other capabilities. Like "M-x dunnet". Must be an OS. Of course the reality is that Emacs just copies good ideas from other people. A pretty normal Emacs setup uses Vim keybindings, LSP and now tree-sitter to get a good experience programming.
craftkiller 3y ago
I'm not the GP, but personally I'd consider syntax highlighting perfectly solved if it was handled in the language server. That way we could have highlighting performed by 100% accurate parsers instead of approximations. It wouldn't be as fast as in-editor simple tree-sitter highlighting, but I don't see any issue with my text appearing uncolored at first and gaining color as my editor polls the language server as I type. Currently language servers support semantic highlighting which is close, but as far as I can tell, it seems meant to be supplemental to in-editor local highlighting rather than a full replacement.
- pawelmurias 3y ago
  You would only get the top half of your code highlighted because a compiler doesnt usually continue when encountering syntax errors due to partial code
  - craftkiller 3y ago
    That's sort-of already the case whenever we start a string in any language that supports multi-line strings. You type the first " and then the rest of your file gets re-interpreted as a string. It is annoying but I still think its a worthy trade-off, because I spend a lot more time reading code than being mid-edit with invalid syntax.
    - troupo 3y ago
      Syntax errors are not limited to just typing a quote
      - craftkiller 3y ago
        I never said they were. I'm just pointing out that we already experience the same problem in the case of quotes without the world ending, and I don't think expanding the problem to all syntax errors will make it much worse.
        3y ago

troupo 3y ago

The problem is that a good editor-compatible tool for parsing and syntax highlighting is at cross-purposes with what you want from a compiler. A good overview here: https://matklad.github.io/2022/04/25/why-lsp.html https://matklad.github.io/2022/04/25/why-lsp.html

aidos 3y ago
Interestingly, treesitter is designed for this space (parsing invalid structures with a time component) but it’s still not used as a base for LSPs. I once asked why on HN and people that know more than me said it wasn’t suitable.
- vkazanov 3y ago
  tree-sitter definitely can be used as a incremental parser for an LSP. But the original purpose was and still is to provide a standard format for an editor-agnostic way to parse languages. As somebody who wrote a tree-sitter grammar recently I can confirm that the library definitely has its share of... Interesting choices. But there's nothing else like it as most parser generators are non-incremental, don't do generalized parsing and don't provide decent error recovery.
  - bch 3y ago
    > has its share of… Interesting choices Curious to hear more.
    - vkazanov 3y ago
      Well, nothing truly horrible but here are things I find inconvenient (but working nevertheless) when using the sdk. 1. NodeJS-based utilities. The tooling around js doesn't lean itself well to integration with with the dev's environment. 2. The sdk itself is a mix of Rust, javascript and C. I'd say that this is at least one language too many - makes it harder to contribute. Ideally such fundamental projects should be as simple and homogeneous as possible. 3. I keep wondering if generating languages other than C should be supported. The core idea of the library is brilliant either way so it sort of compensates all of the above.

kristopolous 3y ago

The languages extension strategies for CSS pretifiers are usually pretty reasonable. In editors this always seems extremely esoteric comparatively: I've tried doing it in a few. I'm sure brilliant people find it easy, but I'm merely average on a good day. I haven't tried extending any of these modern electron based editors, can anyone speak to that?

petepete 3y ago

Until TreeSitter came along the effort to add support for new languages to your editor would be gargantuan. Now it's much, much easier providing there's a TreeSitter parser for your language. I don't know of anything else that bridges the gap like this.

tgv 3y ago
Note that not that long ago, computers were simply not powerful enough to run that in acceptable time. Dedicated parsers can be much faster.

mickeyp 3y ago

I clarified exactly why it is the way it is here: https://www.masteringemacs.org/article/tree-sitter-complications-of-parsing-languages https://www.masteringemacs.org/article/tree-sitter-complicat... And also why using LSP to furnish your editor with highlight markers is an inelegant solution for many languages.

deleted 3y ago
[deleted]

vidarh 3y ago

Parsing of a correct program is a pretty "solved" problem. But fast enough re-parsing of fragments and recovery from errors is a much more complex problem, that often doesn't have a single correct answer, and it's also a much newer problem in as much as syntax-highlighting is much newer feature, being preceded largely by "offline" pretty-printers with very different constraints. The extent to which modern compilers try to parse past errors still varies greatly, with a whole lot not even trying to. But just any recovering parser also does not mean the problem is solved. E.g. you've typed "foo". Now you type "(". It'd be very annoying if your editor now re-colors everything as an error, so you typically want some error recovery. But how soon? Do you assume the tokens immediate afterwards are par of what was a valid expression until you typed "foo", or are they a valid part of an argument list? And where do they end? Do you just delay re-parsing until the user has typed more? Or left the line? Sometimes that can help, sometimes it will just make things worse. Parsing methods that work fine if you assume you can "reset" the parse at many different points which tend to constrain the area considered an error and so reducing the size of a typical re-parse will fail badly if you want stricter re-parsing that frequently may trigger reparsing most of a file, for example. A lot of this is subjective, and picking the "right" way of handling it largely comes down to unpacking humans unstated preferences, and trying to reconcile competing and possibly contradictory preferences.

aeonik 3y ago
I wonder why there aren't more tools to allow structural editing like Lisp has. This way the file can never be in an unparsable state.
- ghosty141 3y ago
  This is what treesitter enables!
- nequo 3y ago
  You may be aware but the author of TFA also has a tree-sitter based minor mode called Combobulate for exactly that: https://www.masteringemacs.org/article/combobulate-structured-movement-editing-treesitter https://www.masteringemacs.org/article/combobulate-structure... There is also evil-textobj-tree-sitter for tree-sitter based text objects for Evil mode: https://github.com/meain/evil-textobj-tree-sitter https://github.com/meain/evil-textobj-tree-sitter

PurpleRamen 3y ago

There will never be "perfectly" solved problems at that complexity-level. There are always changing requirements and space for improvement. Make it faster, add new features, use new hardware-abilities, follow the flavors of this decade, this is an eternally going game of catching up.

olau 3y ago

It's a tough problem. Steve Yegge blogged about the complexities involved when he wrote js2-mode: https://steve-yegge.blogspot.com/2008/03/js2-mode-new-javascript-mode-for-emacs.html https://steve-yegge.blogspot.com/2008/03/js2-mode-new-javasc... I guess comp. sci. people studying languages have been more interested in syntactically valid programs than the opposite.

mbork_pl 3y ago
Well, I bet most syntactically valid programs are not interested in comp-sci people!

mbork_pl 3y ago

Well, there are more problems like that. You'd think diffing is a solved problem, and yet we still struggle with syntax-aware diffs (I use difftastic, which is great, but doesn't always work well, and is under constant development).

tuukkah 3y ago
Interesting, it's also based on parsing by Tree-sitter: https://difftastic.wilfred.me.uk/parsing.html https://difftastic.wilfred.me.uk/parsing.html

nerdponx 3y ago

This is that. Tree Sitter has become one of the foundational advances that is allowing us to make progress on solving that problem.

hardwaregeek 3y ago

It's not really a solved problem in general. Most editors appear to use TextMate grammars which nobody likes. Otherwise you have to implement it using whatever custom setup your specific editor uses. It just happens that most languages have some poor soul who set this up already. Emacs is actually on the better side because tree-sitter is a much better setup for writing grammars.