4 ms·

This point in the article is pretty weird (the rest is good). I don't think IDEs do a lot of parsing to do syntax highlighting, isn't it all just regex matching

This point in the article is pretty weird (the rest is good). I don't think IDEs do a lot of parsing to do syntax highlighting, isn't it all just regex matching to identify the types of tokens? I'd be interested in real-world examples of IDEs doing something more complex to achieve syntax highlighting. And conversely, examples of imperfect syntax highlighting of C++ due to the undecidability of its input language. I mean, yes, an actual parser for C++ must be able to execute template computations, and that's a pain, but why would that be relevant to syntax highlighting? It isn't. Now, there are valid reasons for running a proper language frontend from the IDE: Error reporting of all kinds, not just syntax but also type errors and whatever else the compiler likes to complain about. But there the IDE should not try to replicate the parser. Parsing only a context-free surface language will not catch type errors, exactly because enforcing a static type system requires one to "look things up in a symbol table". So the actual compiler should provide an IDE-friendly mode where it runs its frontend and reports errors back, and the IDE should not try to roll its own version of this. In other words, the IDE is not relevant either way. > Any programming language that is massively adopted is context free? Statically typed languages require semantic analysis and symbol table lookups, so they are out. And yet we have context-free grammars for all widely adopted statically typed programming languages. And we also have additional semantic checks for all of them. The notion of a "context free programming language" being one that has context free syntax and no semantic checks at all is not a useful notion. Don't worry about it while designing your language.

alpaca128 6y ago
> I don't think IDEs do a lot of parsing to do syntax highlighting, isn't it all just regex matching to identify the types of tokens? IntelliJ highlights variables etc. differently if they're unused and can highlight different identifiers in different colors. Not sure about other current IDEs. That said I never felt like I needed such features, normal syntax highlighting does the job pretty well for me.
- dehrmann 6y ago
  IntelliJ has context-aware autocomplete. Type this.field.get and it will show methods and fields starting with get for whatever type field is. There's a lot of context needed to make that happen.
pasquinelli 6y ago
>> Any programming language that is massively adopted is context free? > Statically typed languages require semantic analysis and symbol table lookups, so they are out. Isn't that the typechecker, not the parser?
- tom_mellior 6y ago
  Depends on what level you're thinking of: "Pure" grammar or "well-formed programs acceptable to the compiler". C++ does have a context-free grammar (it's in the standard), and also a lot of additional constraints on what constitutes a well-formed programs (this includes type checking). Since we're discussing a claim that "C++ is famously not a context free grammar", we must be thinking of the latter notion and include the type system in our considerations. In which case, we must consider the entire frontend as a whole.
neutronicus 6y ago
Emacs definitely does parsing of C/C++ (or delegates to an LSP server, depending) in order to do indentation, users' preferences for which tend to depend heavily on syntactic context.
- tom_mellior 6y ago
  Indentation is a good point, you need a proper parse for that. But for that you don't need the ability to consult symbol tables, or resolve nasty C++ template metaprogramming stuff. A context-free grammar suffices, and C++ has that. C++ also has some very complicated (undecidable) rules for what programs are well-formed and should be accepted by the compiler, but the editor's indentation tool doesn't need to worry about those.
geophile 6y ago
I'm pretty sure that the JetBrains products (Intellij, PyCharm, etc.) have a pretty deep understanding of the languages they handle. Java is strongly typed, and Intellij does a perfect job, (as far as I can tell), of highlighting, and more importantly, refactoring. Hard to see how they could do that with just regex matching. By contrast, Python is more dynamic, and so the refactoring in PyCharm is pretty weak. It usually catches some of what it needs, but I often need to find the rest. It does appear to be the case that PyCharm is doing flow analysis to infer types. I.e., even PyCharm is doing more than regex matching.
- tom_mellior 6y ago
  > highlighting, and more importantly, refactoring. Hard to see how they could do that with just regex matching. The former is easy, the latter is impossible with regexes only. Which tells you that they are very different things. For refactoring you need not only a proper parse, but also the ability to "look things up in symbol tables". Which is what I said: There are good reasons to run a proper language frontend from the IDE. But syntax highlighting isn't one of these reasons. And refactoring isn't possible with only a simple parse without using symbol tables.
harry8 6y ago
Eclipse has refactoring told for at least Java and c{,++} Variable renaming respecting scope, function renaming that updates the name at the call sights but ignoring overloads, function extraction... and so on. You may not use it or indeed use eclipse at all. I can't tell you about the details, clearly something going on beyond syntax highlighting via keyword regex matching. With a beefy enough box it all works well enough.
johnmaguire2013 6y ago
Many syntax highlighters actually interpret the language to some extent. For example, Atom: https://github.blog/2018-10-31-atoms-new-parsing-system/ https://github.blog/2018-10-31-atoms-new-parsing-system/ Anyway I'm just posting now as I first read this comment when you posted it and tried to recall this blog post by the infamous Steve Yegge, and just finally did: http://steve-yegge.blogspot.com/2007/06/rich-programmer-food.html?m=1 http://steve-yegge.blogspot.com/2007/06/rich-programmer-food...

alpaca128 6y ago

> I don't think IDEs do a lot of parsing to do syntax highlighting, isn't it all just regex matching to identify the types of tokens? IntelliJ highlights variables etc. differently if they're unused and can highlight different identifiers in different colors. Not sure about other current IDEs. That said I never felt like I needed such features, normal syntax highlighting does the job pretty well for me.

dehrmann 6y ago
IntelliJ has context-aware autocomplete. Type this.field.get and it will show methods and fields starting with get for whatever type field is. There's a lot of context needed to make that happen.

pasquinelli 6y ago

>> Any programming language that is massively adopted is context free? > Statically typed languages require semantic analysis and symbol table lookups, so they are out. Isn't that the typechecker, not the parser?

tom_mellior 6y ago
Depends on what level you're thinking of: "Pure" grammar or "well-formed programs acceptable to the compiler". C++ does have a context-free grammar (it's in the standard), and also a lot of additional constraints on what constitutes a well-formed programs (this includes type checking). Since we're discussing a claim that "C++ is famously not a context free grammar", we must be thinking of the latter notion and include the type system in our considerations. In which case, we must consider the entire frontend as a whole.

neutronicus 6y ago

Emacs definitely does parsing of C/C++ (or delegates to an LSP server, depending) in order to do indentation, users' preferences for which tend to depend heavily on syntactic context.

tom_mellior 6y ago
Indentation is a good point, you need a proper parse for that. But for that you don't need the ability to consult symbol tables, or resolve nasty C++ template metaprogramming stuff. A context-free grammar suffices, and C++ has that. C++ also has some very complicated (undecidable) rules for what programs are well-formed and should be accepted by the compiler, but the editor's indentation tool doesn't need to worry about those.

geophile 6y ago

I'm pretty sure that the JetBrains products (Intellij, PyCharm, etc.) have a pretty deep understanding of the languages they handle. Java is strongly typed, and Intellij does a perfect job, (as far as I can tell), of highlighting, and more importantly, refactoring. Hard to see how they could do that with just regex matching. By contrast, Python is more dynamic, and so the refactoring in PyCharm is pretty weak. It usually catches some of what it needs, but I often need to find the rest. It does appear to be the case that PyCharm is doing flow analysis to infer types. I.e., even PyCharm is doing more than regex matching.

tom_mellior 6y ago
> highlighting, and more importantly, refactoring. Hard to see how they could do that with just regex matching. The former is easy, the latter is impossible with regexes only. Which tells you that they are very different things. For refactoring you need not only a proper parse, but also the ability to "look things up in symbol tables". Which is what I said: There are good reasons to run a proper language frontend from the IDE. But syntax highlighting isn't one of these reasons. And refactoring isn't possible with only a simple parse without using symbol tables.

harry8 6y ago

Eclipse has refactoring told for at least Java and c{,++} Variable renaming respecting scope, function renaming that updates the name at the call sights but ignoring overloads, function extraction... and so on. You may not use it or indeed use eclipse at all. I can't tell you about the details, clearly something going on beyond syntax highlighting via keyword regex matching. With a beefy enough box it all works well enough.

johnmaguire2013 6y ago

Many syntax highlighters actually interpret the language to some extent. For example, Atom: https://github.blog/2018-10-31-atoms-new-parsing-system/ https://github.blog/2018-10-31-atoms-new-parsing-system/ Anyway I'm just posting now as I first read this comment when you posted it and tried to recall this blog post by the infamous Steve Yegge, and just finally did: http://steve-yegge.blogspot.com/2007/06/rich-programmer-food.html?m=1 http://steve-yegge.blogspot.com/2007/06/rich-programmer-food...