3 ms·

The stuff on screen would be still (rich) text, of course. Only that it would be a rendering of the actually underlying data structure. Linters and auto-format

The stuff on screen would be still (rich) text, of course. Only that it would be a rendering of the actually underlying data structure. Linters and auto-formatters didn't solve anything. They are just a constant PITA, and a source of arguments. And to quibble a little bit: I said "faster". I did not say how much… Just parsing is indeed very cheep by now. But a major performance issue was unmentioned: Type checking. In languages with advanced type systems that's the most costly part. Putting all that information into a database and carry it along "the code" would make things significantly faster. It would be like there would be only incremental compilation of the few line you're just editing.

duped 4y ago
I've actually worked on this problem quite a bit and taken this approach, but it's not as obvious as you're making it out to be - I'm going to talk in generalizations about optimization, which is obviously bad so take this at face value. Type checking is usually dwarfed by macro expansion, import resolution, optimization, and code generation. There's very little I/O, not that much work to do, and it is pretty straightforward to cache (*). There's a trap to thinking that complex syntax and type systems are potential bottlenecks - it's true the compiler has to do more work, but it's still not that much compared to all the other time it has to waste doing mundane things. An example is import resolution. In order to type check a compilation unit you need to type check its imports. In order to do that, the compilation unit and its dependencies needs to be macro expanded. If the language supports some kind of scripted macro-programming then you need to evaluate the macros and possibly cannot cache the results (performance pitfall #1). If the language does not support explicit imports, then the entire dependency graph may need be scanned and checked to resolve the imports (performance pitfall #2). This is a case where the simplest and most expressive language designs can have the absolute worst performance regardless of the compiler's design - the language definition forces the compiler to do a ton of mundane work, in particular I/O and cache invalidation. A central in-memory database holding a binary representation of the AST does not solve this problem and is not necessarily faster than redoing all of the work up front. In my experience, something as mundane as "import x from y" has a much larger impact over the performance of a compiler than any type checker or parser, and incrementalism strategies are tightly coupled to the design of the language being compiled and not the storage layer. It has to be approached holistically for the language being compiled. Features like "style sheets" for code are just linter and auto formatter config files. It may be useful to reuse the compiler internals for this, but in practice every tooling author eventually learns not to do that. You will need to modify the IR generated by the compiler, and it probably doesn't have an API that's anything close to what you need for that problem (*) Of course I'm glossing over type systems that allow higher-kinded dependent types, but those are going to suck to check the same that macro heavy code sucks and currently isn't that widely used.
- still_grokking 4y ago
  You're talking about C++ issues here. Those are very specific to C++. No other languages besides C/C++ have such a broken import and dependency mechanism. No other language than C++ has such broken "macros" like the template mechanism. Also I was explicitly talking about advanced type systems. C/C++ have a very shallow and simple type system. But OTOH C/C++ have the most advanced optimizers. So the whole picture gets quite bended when looking at it form that perspective. Scala for example has an IR called TASTy which can be stored along the compile artifacts. It's the AST after type checking (Typed Abstract Syntax Tree). Recompiling form that is very fast as it's (almost) only the relatively cheep code gen step. Also incremental compilation works differently for a lot of languages because those languages have small and independent compilation units. The situation in C/C++ is again a very specific mess as there are really no modules (until now). Usually you don't need to potentially recompile the world on every code change. > Features like "style sheets" for code are just linter and auto formatter config files. No, they aren't. If I "change the style of my code" the actual code gets changed! In case I would change the style sheet used for rendering only a local config value would get switched. The "binary code" on disc would stay exactly the same. (Still I could than have "Python with braces" after the switch if this is what I prefer to look at). > It may be useful to reuse the compiler internals for this, but in practice every tooling author eventually learns not to do that. Everybody involved in the "LSP revolution" would disagree, I think. The whole point of LSPs is to reuse the already existing tools (usually the compiler) built by the language devs.
  - duped 4y ago
    I'm not talking about C++ at all! My entire point is that caching is hard and tightly coupled to the design of the language, and things like typechecking are not really that slow compared to the other work a compiler has to do. Choosing an in memory representation is not really a secret recipe for performance or utility, designing the way the cache gets updated and what it stores is. I think the idea of a core language with multiple syntaxes is interesting, but I'd contest that the system to implement it is any faster. > Everybody involved in the "LSP revolution" would disagree, I think. I reached this conclusion after writing a language server and researching other language servers. The tide is changing to where the primary compilers for languages are query-oriented compilers like Roslyn that allow for this kind of thing, but they aren't nearly as common as batch compilers so language server authors find themselves writing everything from scratch. Interestingly though, big portions like type checkers can be reused if they have stable input/output IRs because they're not slow (however, compiler authors are loathe to expose those details or make stability guarantees there).

duped 4y ago

I've actually worked on this problem quite a bit and taken this approach, but it's not as obvious as you're making it out to be - I'm going to talk in generalizations about optimization, which is obviously bad so take this at face value. Type checking is usually dwarfed by macro expansion, import resolution, optimization, and code generation. There's very little I/O, not that much work to do, and it is pretty straightforward to cache (*). There's a trap to thinking that complex syntax and type systems are potential bottlenecks - it's true the compiler has to do more work, but it's still not that much compared to all the other time it has to waste doing mundane things. An example is import resolution. In order to type check a compilation unit you need to type check its imports. In order to do that, the compilation unit and its dependencies needs to be macro expanded. If the language supports some kind of scripted macro-programming then you need to evaluate the macros and possibly cannot cache the results (performance pitfall #1). If the language does not support explicit imports, then the entire dependency graph may need be scanned and checked to resolve the imports (performance pitfall #2). This is a case where the simplest and most expressive language designs can have the absolute worst performance regardless of the compiler's design - the language definition forces the compiler to do a ton of mundane work, in particular I/O and cache invalidation. A central in-memory database holding a binary representation of the AST does not solve this problem and is not necessarily faster than redoing all of the work up front. In my experience, something as mundane as "import x from y" has a much larger impact over the performance of a compiler than any type checker or parser, and incrementalism strategies are tightly coupled to the design of the language being compiled and not the storage layer. It has to be approached holistically for the language being compiled. Features like "style sheets" for code are just linter and auto formatter config files. It may be useful to reuse the compiler internals for this, but in practice every tooling author eventually learns not to do that. You will need to modify the IR generated by the compiler, and it probably doesn't have an API that's anything close to what you need for that problem (*) Of course I'm glossing over type systems that allow higher-kinded dependent types, but those are going to suck to check the same that macro heavy code sucks and currently isn't that widely used.

still_grokking 4y ago
You're talking about C++ issues here. Those are very specific to C++. No other languages besides C/C++ have such a broken import and dependency mechanism. No other language than C++ has such broken "macros" like the template mechanism. Also I was explicitly talking about advanced type systems. C/C++ have a very shallow and simple type system. But OTOH C/C++ have the most advanced optimizers. So the whole picture gets quite bended when looking at it form that perspective. Scala for example has an IR called TASTy which can be stored along the compile artifacts. It's the AST after type checking (Typed Abstract Syntax Tree). Recompiling form that is very fast as it's (almost) only the relatively cheep code gen step. Also incremental compilation works differently for a lot of languages because those languages have small and independent compilation units. The situation in C/C++ is again a very specific mess as there are really no modules (until now). Usually you don't need to potentially recompile the world on every code change. > Features like "style sheets" for code are just linter and auto formatter config files. No, they aren't. If I "change the style of my code" the actual code gets changed! In case I would change the style sheet used for rendering only a local config value would get switched. The "binary code" on disc would stay exactly the same. (Still I could than have "Python with braces" after the switch if this is what I prefer to look at). > It may be useful to reuse the compiler internals for this, but in practice every tooling author eventually learns not to do that. Everybody involved in the "LSP revolution" would disagree, I think. The whole point of LSPs is to reuse the already existing tools (usually the compiler) built by the language devs.
- duped 4y ago
  I'm not talking about C++ at all! My entire point is that caching is hard and tightly coupled to the design of the language, and things like typechecking are not really that slow compared to the other work a compiler has to do. Choosing an in memory representation is not really a secret recipe for performance or utility, designing the way the cache gets updated and what it stores is. I think the idea of a core language with multiple syntaxes is interesting, but I'd contest that the system to implement it is any faster. > Everybody involved in the "LSP revolution" would disagree, I think. I reached this conclusion after writing a language server and researching other language servers. The tide is changing to where the primary compilers for languages are query-oriented compilers like Roslyn that allow for this kind of thing, but they aren't nearly as common as batch compilers so language server authors find themselves writing everything from scratch. Interestingly though, big portions like type checkers can be reused if they have stable input/output IRs because they're not slow (however, compiler authors are loathe to expose those details or make stability guarantees there).