4 ms·

A context-free grammar, besides making things a lot simpler, means that IDEs can do syntax highlighting without integrating most of a compiler front end. As a r

by chubot 9y ago

A context-free grammar, besides making things a lot simpler, means that IDEs can do syntax highlighting without integrating most of a compiler front end. As a result, third-party tools become much more likely to exist. This statement feels imprecise in a couple ways. It seems to imply that some IDEs actually use context-free grammars for syntax highlighting? Which ones? As far as I can tell, Vim and Textmate bundles (i.e. what Github uses for syntax highlighting) don't use anything close to a context-free grammar for their syntax highlighting models. They are more like ad hoc lexers -- a collection of rules and regular expressions. Certainly an editor doesn't want to parse the entire file to highlight text, because it has to potentially re-highlight at every keystroke. Also, you want to be able to highlight malformed programs (i.e. code with syntax errors). As far as I understand, that's generally why they don't use grammars. I think you might mean that a language should be designed with a concise grammar so that somebody else can reimplement it by hand more easily? That is, you want multiple independent implementations. If that's true, "context-free grammars" is the wrong term to express that notion. Context-free grammars don't handle many real languages, not just C and C++, but also Python/Perl/Ruby, and even JavaScript and Go (semi-colon insertion.) Also, a lot of tools are enabled not by grammars but by producing specific data structures for your front end. (e.g. here is the way I think about it: http://www.oilshell.org/blog/2017/02/11.html http://www.oilshell.org/blog/2017/02/11.html ) The different between Clang and GCC is that Clang is a library that enables a whole ecosystem of fantastic tools, including IDE support. But Clang doesn't do much at all with CFGs. The real difference is that the front end is a library that produces a very rich representation of the code. The C# front end does similar things. More links here: https://github.com/oilshell/oil/wiki/Lossless-Syntax-Tree-Pattern https://github.com/oilshell/oil/wiki/Lossless-Syntax-Tree-Pa... In other words, I would say that integrating the compiler front end into the IDE became the more popular and successful approach. And of course the compiler very much has to be designed with this use case in mind.

WalterBright 9y ago
> integrating the compiler front end into the IDE became the more popular and successful approach C and C++ syntax highlighting suffered for decades because it was not possible to do a complete job of it without integrating a compiler front end into the IDE.
- chubot 9y ago
  I think you're talking about Intellisense-like functionality, not syntax highlighting. Syntax highlighting works just fine in C++ in many editors. The red squigglies in Visual Studio is a completely different technology than syntax highlighting, which is lexical. Intellisense has only a rough relation to context-free grammars. Some of the red squigglies are errors in semantic analysis, not parsing. That is why the more fruitful approach was to integrate the actual compiler and the IDE -- not have two separate parsers/compilers, as was the case with Java IDEs. Also, I think it was always "possible" to integrate a C++ front end and an IDE -- just nobody did it in an open source compiler until Clang.
  - WalterBright 9y ago
    No, I'm not talking about Intellisense. Just highlighting the code correctly. The C++ editors that don't integrate a compiler front end tend to get confused when you do tricks with macros, backslash line splicing, and trigraphs, for example. They'll also have trouble with the >> thing and templates, and preprocessor metaprogramming stuff like: #define BEGIN { #define END } They do work fine with conventional code, but if one knows the darker corners of the Standard, they can be broken. One could argue "don't write code like that", but as a tool developer there is always someone that does. When designing a language, though, one can design out all those problems.
    - chubot 9y ago
      OK, but highlighting the code correctly has essentially nothing to do with context free grammars. This happens in Vim and Emacs with languages other than C and C++ -- here docs in shell, multiline strings in Python -- and Python does use a CFG, etc. I agree it's annoying although I think most people view it as a minor thing. They stick with Vim and Emacs for other reasons. I'm not sure anyone has based their language design around Vim/Emacs syntax highlighting, although ironically that is one of my criteria for language design. I was just confused by the advice to use a CFG, since it's not the relevant issue. I would say the relevant issue is that your lexer shouldn't be too clever and have too many modes. And to avoid mixing languages in the same file, or have a very obvious lexical construct to mix languages. The C preprocessor is an entirely separate language than C or C++, so that is the core of the issue in your example. Likewise, it is usually hard to highlight CSS and JavaScript embedded within HTML.

by chubot 9y ago

WalterBright 9y ago

> integrating the compiler front end into the IDE became the more popular and successful approach C and C++ syntax highlighting suffered for decades because it was not possible to do a complete job of it without integrating a compiler front end into the IDE.

chubot 9y ago
I think you're talking about Intellisense-like functionality, not syntax highlighting. Syntax highlighting works just fine in C++ in many editors. The red squigglies in Visual Studio is a completely different technology than syntax highlighting, which is lexical. Intellisense has only a rough relation to context-free grammars. Some of the red squigglies are errors in semantic analysis, not parsing. That is why the more fruitful approach was to integrate the actual compiler and the IDE -- not have two separate parsers/compilers, as was the case with Java IDEs. Also, I think it was always "possible" to integrate a C++ front end and an IDE -- just nobody did it in an open source compiler until Clang.
- WalterBright 9y ago
  No, I'm not talking about Intellisense. Just highlighting the code correctly. The C++ editors that don't integrate a compiler front end tend to get confused when you do tricks with macros, backslash line splicing, and trigraphs, for example. They'll also have trouble with the >> thing and templates, and preprocessor metaprogramming stuff like: #define BEGIN { #define END } They do work fine with conventional code, but if one knows the darker corners of the Standard, they can be broken. One could argue "don't write code like that", but as a tool developer there is always someone that does. When designing a language, though, one can design out all those problems.
  - chubot 9y ago
    OK, but highlighting the code correctly has essentially nothing to do with context free grammars. This happens in Vim and Emacs with languages other than C and C++ -- here docs in shell, multiline strings in Python -- and Python does use a CFG, etc. I agree it's annoying although I think most people view it as a minor thing. They stick with Vim and Emacs for other reasons. I'm not sure anyone has based their language design around Vim/Emacs syntax highlighting, although ironically that is one of my criteria for language design. I was just confused by the advice to use a CFG, since it's not the relevant issue. I would say the relevant issue is that your lexer shouldn't be too clever and have too many modes. And to avoid mixing languages in the same file, or have a very obvious lexical construct to mix languages. The C preprocessor is an entirely separate language than C or C++, so that is the core of the issue in your example. Likewise, it is usually hard to highlight CSS and JavaScript embedded within HTML.