4 ms·

I don't think it's easy to write a good syntax coloring engine like the one in Vim. Syntax coloring has to handle context: different rules for material nested

I don't think it's easy to write a good syntax coloring engine like the one in Vim. Syntax coloring has to handle context: different rules for material nested in certain ways. Vim's syntax higlighter lets you declare two kinds of items: matches and regions. Matches are simpler lexical rules, whereas regions have separate expressions for matching the start and end and middle. There are ways to exclude leading and trailing material from a region. Matches and regions can declare that they are contained. In that case they are not active unless they occur in a containing region. Contained matches declare which regions contain them. Regions declare which other regions they contain. That's the basic semantic architecture; there are bells and whistles in the system due to situations that arise. I don't think even Justine could develop that in an interview, other than as an overnight take home.

kazinator 2y ago
Here is an example of something hard to handle: TXR language with embedded TXR Lisp. This is the "genman" script which takes the raw output of a manpage to HTML converter, and massages it to form the HTML version of the TXR manual: https://www.kylheku.com/cgit/txr/tree/genman.txr https://www.kylheku.com/cgit/txr/tree/genman.txr Everything that is white (not colored) is literal template material. Lisp code is embedded in directives, like @(do ...). In this scheme, TXR keywords appear purple, TXR Lisp ones green. They can be the same; see the (and ...) in line 149, versus numerous occurrences of @(and). Quasistrings contain nested syntax: see 130 where `<a href ..> ... </a>` contains an embedded (if ...). That could itself contain a quasistring with more embedded code. TXR's txr.vim" and tl.vim* syntax definition files are both generated by this: https://www.kylheku.com/cgit/txr/tree/genvim.txr https://www.kylheku.com/cgit/txr/tree/genvim.txr
saghm 2y ago
Naively, I would have assumed that the "correct" way to write a syntax highlighter would be to parse into an AST and then iterate over the tokens and update the color of a token based on the type of node (and maybe just tracking a diff to avoid needing to recolor things that haven't changed). I'm guessing that if this isn't done, it's for efficiency reasons (e.g. due to requiring parsing the whole file to highlight rather than just the part currently visible on the screen)?
- Someone 2y ago
  > I would have assumed that the "correct" way to write a syntax highlighter would be to parse into an AST and then […] I'm guessing that if this isn't done, it's for efficiency reasons It’s not only running time, but also ease of implementation. A good syntax highlighter should do a decent job highlighting both valid and invalid programs (rationale: in most (editor, language) pairs, writing a program involves going through moments where the program being written isn’t a valid program) If you decide to use an AST, that means you need to have good heuristics for turning invalid programs into valid ones that best mimic what the programmer intended. That can be difficult to achieve (good compilers have such heuristics, but even if you have such a compiler, chances are it isn’t possible to reuse them for syntax coloring) If this simpler approach gives you most of what you can get with the AST approach, why bother writing that? Also, there are languages where some programs can’t be perfectly parsed or syntax colored without running them. For those, you need this approach.
deleted 2y ago
[deleted]
tomcam 2y ago
> I don't think even Justine could develop that in an interview Not so sure I’d put money on that opinion ;)

kazinator 2y ago

Here is an example of something hard to handle: TXR language with embedded TXR Lisp. This is the "genman" script which takes the raw output of a manpage to HTML converter, and massages it to form the HTML version of the TXR manual: https://www.kylheku.com/cgit/txr/tree/genman.txr https://www.kylheku.com/cgit/txr/tree/genman.txr Everything that is white (not colored) is literal template material. Lisp code is embedded in directives, like @(do ...). In this scheme, TXR keywords appear purple, TXR Lisp ones green. They can be the same; see the (and ...) in line 149, versus numerous occurrences of @(and). Quasistrings contain nested syntax: see 130 where `<a href ..> ... </a>` contains an embedded (if ...). That could itself contain a quasistring with more embedded code. TXR's txr.vim" and tl.vim* syntax definition files are both generated by this: https://www.kylheku.com/cgit/txr/tree/genvim.txr https://www.kylheku.com/cgit/txr/tree/genvim.txr

saghm 2y ago

Naively, I would have assumed that the "correct" way to write a syntax highlighter would be to parse into an AST and then iterate over the tokens and update the color of a token based on the type of node (and maybe just tracking a diff to avoid needing to recolor things that haven't changed). I'm guessing that if this isn't done, it's for efficiency reasons (e.g. due to requiring parsing the whole file to highlight rather than just the part currently visible on the screen)?

Someone 2y ago
> I would have assumed that the "correct" way to write a syntax highlighter would be to parse into an AST and then […] I'm guessing that if this isn't done, it's for efficiency reasons It’s not only running time, but also ease of implementation. A good syntax highlighter should do a decent job highlighting both valid and invalid programs (rationale: in most (editor, language) pairs, writing a program involves going through moments where the program being written isn’t a valid program) If you decide to use an AST, that means you need to have good heuristics for turning invalid programs into valid ones that best mimic what the programmer intended. That can be difficult to achieve (good compilers have such heuristics, but even if you have such a compiler, chances are it isn’t possible to reuse them for syntax coloring) If this simpler approach gives you most of what you can get with the AST approach, why bother writing that? Also, there are languages where some programs can’t be perfectly parsed or syntax colored without running them. For those, you need this approach.

deleted 2y ago

[deleted]

tomcam 2y ago

> I don't think even Justine could develop that in an interview Not so sure I’d put money on that opinion ;)