3 ms·

I think you're talking about Intellisense-like functionality, not syntax highlighting. Syntax highlighting works just fine in C++ in many editors. The red squ

by chubot 9y ago

I think you're talking about Intellisense-like functionality, not syntax highlighting. Syntax highlighting works just fine in C++ in many editors. The red squigglies in Visual Studio is a completely different technology than syntax highlighting, which is lexical. Intellisense has only a rough relation to context-free grammars. Some of the red squigglies are errors in semantic analysis, not parsing. That is why the more fruitful approach was to integrate the actual compiler and the IDE -- not have two separate parsers/compilers, as was the case with Java IDEs. Also, I think it was always "possible" to integrate a C++ front end and an IDE -- just nobody did it in an open source compiler until Clang.

WalterBright 9y ago
No, I'm not talking about Intellisense. Just highlighting the code correctly. The C++ editors that don't integrate a compiler front end tend to get confused when you do tricks with macros, backslash line splicing, and trigraphs, for example. They'll also have trouble with the >> thing and templates, and preprocessor metaprogramming stuff like: #define BEGIN { #define END } They do work fine with conventional code, but if one knows the darker corners of the Standard, they can be broken. One could argue "don't write code like that", but as a tool developer there is always someone that does. When designing a language, though, one can design out all those problems.
- chubot 9y ago
  OK, but highlighting the code correctly has essentially nothing to do with context free grammars. This happens in Vim and Emacs with languages other than C and C++ -- here docs in shell, multiline strings in Python -- and Python does use a CFG, etc. I agree it's annoying although I think most people view it as a minor thing. They stick with Vim and Emacs for other reasons. I'm not sure anyone has based their language design around Vim/Emacs syntax highlighting, although ironically that is one of my criteria for language design. I was just confused by the advice to use a CFG, since it's not the relevant issue. I would say the relevant issue is that your lexer shouldn't be too clever and have too many modes. And to avoid mixing languages in the same file, or have a very obvious lexical construct to mix languages. The C preprocessor is an entirely separate language than C or C++, so that is the core of the issue in your example. Likewise, it is usually hard to highlight CSS and JavaScript embedded within HTML.
  - WalterBright 9y ago
    > nothing to do with context free grammars This is incorrect. Some languages have user-defined tokens. Some have contextual keywords. Both require a semantic understanding of the code to highlight them correctly. And it isn't just the preprocessor with C++. There's the >> problem. It's not just me talking through my hat - tools for C++, such as pretty-printers and refactoring tools - have been very slow to appear, and fragile. But with a language like Java the tooling is quick & easy to write. You don't have to believe me. Write a tool that reads C++ source code and inserts boilerplate at the beginning and end of each function, and works 100% of the time.
    - chubot 9y ago
      I know exactly what problem you are talking about. It's exactly the problem that Clang solves. With the Clang front end, you can write a tool to read C++ source code and insert boilerplate at the beginning and end of each function, and it will work 100% of the time. There are dozens of such tools in active use at Google and I'm sure many other places. But it has nothing to do with context free grammars -- really. Clang uses a recursive descent parser. GCC used to use a yacc-style grammar (which BTW is only context free-ish because of semantic actions), but it could NOT perform the task you are talking about. In fact that was largely the motivation for Clang. It also doesn't really have to do with syntax highlighting as practiced by any editor or IDE I know of. Even though Clang has the power that you want ("semantic understanding"), I don't know any editor that uses it for syntax highlighting. Instead they use approximate lexical hacks. This is probably because of the need to highlight partial files and malformed files, as I mentioned. You don't want your syntax highlighting to turn off in the middle of typing a code fragment. But editors DO use Clang for semantic understanding, e.g. the YCM plugin for Vim. But they use CFGs for NEITHER problem. You're conflating two different issues and suggesting the wrong solution for both of them. There are a lot of links about this issue with regard to languages like C#, Scala, Go, JavaScript, etc. in the wiki page I linked. I agree with your general point about language design, but the terminology you're using is wrong and confusing.
      - WalterBright 9y ago

by chubot 9y ago

WalterBright 9y ago

No, I'm not talking about Intellisense. Just highlighting the code correctly. The C++ editors that don't integrate a compiler front end tend to get confused when you do tricks with macros, backslash line splicing, and trigraphs, for example. They'll also have trouble with the >> thing and templates, and preprocessor metaprogramming stuff like: #define BEGIN { #define END } They do work fine with conventional code, but if one knows the darker corners of the Standard, they can be broken. One could argue "don't write code like that", but as a tool developer there is always someone that does. When designing a language, though, one can design out all those problems.

chubot 9y ago
OK, but highlighting the code correctly has essentially nothing to do with context free grammars. This happens in Vim and Emacs with languages other than C and C++ -- here docs in shell, multiline strings in Python -- and Python does use a CFG, etc. I agree it's annoying although I think most people view it as a minor thing. They stick with Vim and Emacs for other reasons. I'm not sure anyone has based their language design around Vim/Emacs syntax highlighting, although ironically that is one of my criteria for language design. I was just confused by the advice to use a CFG, since it's not the relevant issue. I would say the relevant issue is that your lexer shouldn't be too clever and have too many modes. And to avoid mixing languages in the same file, or have a very obvious lexical construct to mix languages. The C preprocessor is an entirely separate language than C or C++, so that is the core of the issue in your example. Likewise, it is usually hard to highlight CSS and JavaScript embedded within HTML.
- WalterBright 9y ago
  > nothing to do with context free grammars This is incorrect. Some languages have user-defined tokens. Some have contextual keywords. Both require a semantic understanding of the code to highlight them correctly. And it isn't just the preprocessor with C++. There's the >> problem. It's not just me talking through my hat - tools for C++, such as pretty-printers and refactoring tools - have been very slow to appear, and fragile. But with a language like Java the tooling is quick & easy to write. You don't have to believe me. Write a tool that reads C++ source code and inserts boilerplate at the beginning and end of each function, and works 100% of the time.
  - chubot 9y ago
    I know exactly what problem you are talking about. It's exactly the problem that Clang solves. With the Clang front end, you can write a tool to read C++ source code and insert boilerplate at the beginning and end of each function, and it will work 100% of the time. There are dozens of such tools in active use at Google and I'm sure many other places. But it has nothing to do with context free grammars -- really. Clang uses a recursive descent parser. GCC used to use a yacc-style grammar (which BTW is only context free-ish because of semantic actions), but it could NOT perform the task you are talking about. In fact that was largely the motivation for Clang. It also doesn't really have to do with syntax highlighting as practiced by any editor or IDE I know of. Even though Clang has the power that you want ("semantic understanding"), I don't know any editor that uses it for syntax highlighting. Instead they use approximate lexical hacks. This is probably because of the need to highlight partial files and malformed files, as I mentioned. You don't want your syntax highlighting to turn off in the middle of typing a code fragment. But editors DO use Clang for semantic understanding, e.g. the YCM plugin for Vim. But they use CFGs for NEITHER problem. You're conflating two different issues and suggesting the wrong solution for both of them. There are a lot of links about this issue with regard to languages like C#, Scala, Go, JavaScript, etc. in the wiki page I linked. I agree with your general point about language design, but the terminology you're using is wrong and confusing.
    - WalterBright 9y ago