3 ms·

This is the difference between building a parser for a compiler and for an IDE. For a compiler, you want to stop at the first error. The parser can also emit a

• nicoburns 5y ago
  Why don't you want to do the IDE-style parsing in a compiler?
  • kkdaemas 5y ago
    That can be desirable but there are a few challenges: - The compiler code becomes more complicated, making correctness harder - The compiler might become slower to run - Introducing new languages features may become harder, again due to code complexity
    • oauea 5y ago
      > The compiler code becomes more complicated, making correctness harder > Introducing new languages features may become harder, again due to code complexity It'll be written for IDEs anyway. Might as well reuse if possible, right?
    • phillipcarter 5y ago
      Compiler will get plenty complicated without IDE scenarios, trust me on that one. Slowness is also never really a thing to worry about here, especially because usage patterns in an IDE vs. a batch process are so different. It's almost always the other way around: someone writes something that's completely fine for a batch process but tanks IDE performance.
  • onei 5y ago
    I was a little curious about this too. It's contrary to what I see in the Go and Rust compilers. My understanding was that it's good to have a go at parsing all input if possible so the end-user can batch fix mistakes, but it's unreasonable to expect error checking in post-parsing steps to occur if there are parse errors because the AST is almost certainly incomplete.
  • mannykannot 5y ago
    From time to time, I see errors in IDE parsing. It's not a big deal there, but it would be in a compiler or interpreter.
    • duped 5y ago
      What case would introduce a parsing error in an IDE that isn't the case in a compiler?
      • hombre_fatal 5y ago
        Bad language plugins in an IDE can show you this. Sometimes I'll be using a niche language with someone's side-project plugin that has some issues even though it's correct, like when its file-formatter can't parse the code and fails with an error even though it's valid code for the compiler.
        • duped 5y ago
          If the plugins used the same parser as the compiler this wouldn't be an issue?
      • deleted 5y ago
        [deleted]
      • mannykannot 5y ago
        I figure it is on account of the desirable situation you describe in your other post not obtaining: in order to satisfy the goals of the IDE, it attempts to go beyond where the compiler parser would stop, as the compiler is more of a batch than a responsive one, and sometimes the IDE gets it wrong. As you say, batch to responsive is the difficult way to go. In addition, I suppose that there are people hard at work applying ML in tools to help understand incomplete code and mitigate the false positive problem of traditional static analysis. I can imagine probabilistic parsing being useful in this case, but not so much in compiling.
• curun1r 5y ago
  > makes it difficult to share parser logic between the two It’s difficult to reuse traditional compiler logic in an IDE, but there’s good examples that the reverse isn’t true. IDE validation of language semantics is a strictly more complex problem, but if you start by solving that, it’s not as hard to add a compiler backend. A compiler’s job is to either take correct code and translate to its compiled form or take incorrect code and report errors. There’s no reason an IDE-focused parser/compiler can’t do both. IIRC, Microsoft talked publicly about how they built the C# compiler as IDE-first and found that it simplified things greatly. And I think there has been substantive discussions within the Rust community about bringing parts of rust-analyzer into the official compiler whereas the RLS approach of reusing compiler APIs wasn’t able to provide a reasonable IDE experience.
  • rhdunn 5y ago
    My perspective comes from writing an XPath and XQuery lexer and parser for IntelliJ, which has its own lexer, parser, and AST APIs. The XPath lexer and parser are designed to be overridden where needed to implement the XQuery lexer and parser. The lexer itself has state as a stack-based lexer in order to tokenize the different structures (string literals, comments, embedded XML) correctly. A compiler could use the parse state as the context to drive the tokenizer without needing a state/stack-based lexer. The lexer also treats keyword tokens as an identifier type as keywords can be used as identifiers. This is not necessary in a compiler as it knows when it is reading/expects a keyword. My parser handles the different versions of XPath/XQuery, the different extensions, and vendor-specific extensions all in a unified lexer/parser. A compiler could ignore the bits it does not support and simplify some of the logic. My QName parser is very complex due to providing error recovery and reporting for things like spaces, etc. -- Other parsers (e.g. Saxon) treat the QName as a single token. I'm also generating a full AST with single nodes removed, e.g.: XPath InstanceofExpr IntegerLiteral "5" XmlNCName "instance" XmlNCName "of" SequenceType AtomicOrUnionType QName XmlNCName "xs" Token ":" XmlNCName "string" Token "?" I'm traversing this AST to do things like variable and namespace resolution. For the modules, I'm using the IDE's mechanisms to search the project files. -- In a compiler, these would be collated and built as the file is parsed, which does not work with incremental/partial parsing. I'm getting to the stage where I can evaluate several static programs due to the need of implementing IDE features, and providing static analysis.
• phillipcarter 5y ago
  This isn't necessarily correct. Many modern compilers (e.g., C# and F# compilers) will do error recovery and keep processing as far as they can go, accumulating errors in the process. These same compilers are not different from that used in the IDE - they are one and the same. And finally, modern compilers can also be tuned based on usage, such as enabling batch mode to optimize for speed in a single thread or using as many threads as are available to optimize for IDE scenarios.
• duped 5y ago
  It's only difficult to reuse logic from a batch compiler in a responsive compiler. It's trivial to derive a batch compiler from a responsive one. You do not actually want to stop at the first error in either case. You want to accumulate all the errors at a given phase of the compilation and halt. Sometimes that allows other phases to progress (for example, you do not want an error in one compilation unit to halt the compilation of any other units until linking in an incremental compiler). You do actually want to reuse IRs in the IDE, otherwise it can be extremely difficult to get certain things correct (and some are next to impossible, like macro expansion/syntax extensions, decompilation of libraries, etc). Unification of the reference compiler and IDE backend are extremely desirable, in my opinion. Very few languages take that tact (C#/.NET being a major exception) but not because it's a bad design - it's because it's hard. Writing a lexer and parser is easy if you don't care about edits and and updates. And there are very few parser generators that do make that possible (tree sitter being the major exception). And once you have a working lexer/parser it is difficult to replace it in your compiler, so few language devs ever take that approach. It's essentially a massive engineering effort in something that is rather boring for low payoff in the early life of a language implementation, the RoI is only obvious much later when usage scales up. So it's unsurprising many languages do not do it early on, and like objects, most languages die young.