3 ms·
Parsing of a correct program is a pretty "solved" problem. But fast enough re-parsing of fragments and recovery from errors is a much more complex problem, tha
by vidarh 3y ago
Parsing of a correct program is a pretty "solved" problem.
But fast enough re-parsing of fragments and recovery from errors is a much more complex problem, that often doesn't have a single correct answer, and it's also a much newer problem in as much as syntax-highlighting is much newer feature, being preceded largely by "offline" pretty-printers with very different constraints.
The extent to which modern compilers try to parse past errors still varies greatly, with a whole lot not even trying to.
But just any recovering parser also does not mean the problem is solved. E.g. you've typed "foo". Now you type "(". It'd be very annoying if your editor now re-colors everything as an error, so you typically want some error recovery. But how soon? Do you assume the tokens immediate afterwards are par of what was a valid expression until you typed "foo", or are they a valid part of an argument list? And where do they end? Do you just delay re-parsing until the user has typed more? Or left the line? Sometimes that can help, sometimes it will just make things worse.
Parsing methods that work fine if you assume you can "reset" the parse at many different points which tend to constrain the area considered an error and so reducing the size of a typical re-parse will fail badly if you want stricter re-parsing that frequently may trigger reparsing most of a file, for example.
A lot of this is subjective, and picking the "right" way of handling it largely comes down to unpacking humans unstated preferences, and trying to reconcile competing and possibly contradictory preferences.
- aeonik 3y agoI wonder why there aren't more tools to allow structural editing like Lisp has. This way the file can never be in an unparsable state.
- ghosty141 3y agoThis is what treesitter enables!
- nequo 3y agoYou may be aware but the author of TFA also has a tree-sitter based minor mode called Combobulate for exactly that: https://www.masteringemacs.org/article/combobulate-structured-movement-editing-treesitter https://www.masteringemacs.org/article/combobulate-structure... There is also evil-textobj-tree-sitter for tree-sitter based text objects for Evil mode: https://github.com/meain/evil-textobj-tree-sitter https://github.com/meain/evil-textobj-tree-sitter
- rprospero 3y agoI spent a couple of months working with a tool along those lines. Unfortunately, it's one of those issues where it won't work specifically without the galactic emperor mandating these tools under pain of death. A colleague using VS Code would commit code with unbalance parentheses. The tooling was smart enough that it could load the file, even though it wasn't parsable, but there was no structural was to fix the missing closing brace. Adding a closing brace automatically inserted another opening brace, leaving the original mismatch.
- taeric 3y agoThis is not fully correct. You are right that lisp has very regular atoms that you can read, but the full parsing is different. Consider (let (a 0) (1+ a)) is not valid syntax. Such that if you were to add color for the different parts, it would fail. Indeed, if you want to label the parts of the program tree, you have to parse it more than just "lists of atoms."
- Barrin92 3y agobecause in most languages there's a level of abstraction between the syntax of the language and the actual data structure of the program. To use the much maligned term, most languages aren't homoiconic, the internal structure of the program is hidden away from the programmer. Lisps essentially have no syntax, that's why it is trivial to manipulate a lisp program structurally.
- vidarh 3y agoYou can do this with most other language too, but I think the challenge is that a lot of us have habits that we'd need to change. E.g. with structural editing I couldn't easily decide I want to add exception handling around a block in Ruby by writing "begin", then moving down, and writing "rescue ..." and "end" - I'd likely need to mark the block first. That may well end up being faster, but it's a fairly big change. I wonder, though, if a "semi-structural" editor with a keypress to "complete what is otherwise an error" would work well. E.g. I write "begin" and so imbalances an expression, and the editor looked at what I typed that changed the parse from successful to an error and puts up options to insert "rescue ... end" or just "end". Maybe coupled with a warning on save if the file does not parse.
- jlokier 3y ago> it's also a much newer problem in as much as syntax-highlighting is much newer feature Just an aside. I wouldn't associate syntax highlighting with new. It's almost as old as text editors. Accurate syntax highlighting in real-time visual text editors is over 40 years old by now and was becoming common by the late 1980s. GNU Emacs gained syntax highlighting in 1989 and it was considered late to the party. Many programming editors and IDEs had syntax highlighting by then.