5 ms·

Well, it's the first thing in the pipeline, so why not?

mpd 5y ago
I'm not your parent commentor, but imo, it's because there's is an enormous amount of existing info out there about these two steps. For these kinds of projects, they are necessary whether you're translating to another language (i.e. generating code) or not. And everyone starts with lexing and parsing. And just repeats the same old shit.
dmitriid 5y ago
There's too much emphasis on parsing in all the teaching materials on compiling. And there's rarely any good stuff on the actual hard problems
- habibur 5y ago
  > And there's rarely any good stuff on the actual hard problems So what are those hard problems?
  - sim_card_map 5y ago
    codegen, optimizations, ast transformations
  - dmitriid 5y ago
    The actual compilation. Actual programming language design. Types, type inference, function implementations, error handling and recovery, incremental compilation, interfacing with the external world, code generation, fixing problems in code generation tools, CPU architecture concerns... The toy languages that most learning materials make can be parsed with regexps and is definitely not something to spend a lot of time on at the beginning. Because actual parsing problems are never addressed in these materials, and can come later (example of concerns: https://matklad.github.io//2018/06/06/modern-parser-generator.html https://matklad.github.io//2018/06/06/modern-parser-generato...).
    - rualca 5y ago
      > The actual compilation. Actual programming language design. Isn't that intimately tied to parsing? I mean, how do you expect to implement a parser if you have no idea if a specific programming construct can actually be parsed, or what are the implications in compilation time of doing things one way or another?
      - dmitriid 5y ago
        > Isn't that intimately tied to parsing? Not entirely > how do you expect to implement a parser if you have no idea if a specific programming construct can actually be parsed Every construct can be parsed. However, there are significantly more aspects to language design than "parsing programming constructs". For example, C++ header/implementation split with no real module system affects compilation orders of magnitudes more than the parsing rules for C++.
kragen 5y ago
I think there are several good reasons not to start with the lexer and parser: 1. If you write a code generator whose input is something like Forth ("operators are separated by spaces") you have written a compiler. If you write a lexer and parser, you have not written a compiler. So, if we want to develop the compiler incrementally, we should start with a back end, then add a front end. 2. Parsing and lexing are much more theoretically tractable than code generation, which is only now beginning to succumb to systematic formalization after nearly 50 years of efforts. Consequently there are enormous edifices of theory and tooling built up around parsing and lexing, which are fascinating and delightful. The problems is that people tend to teach all this stuff, which is not actually necessary to write a useful parser. Rather than studying Chomsky's hierarchy of languages and automata theory, you're probably better off just learning how context-free languages work, a little about refactoring them to eliminate left recursion, and about chronological backtracking search and applying it to the parsing problem, and then just write a recursive-descent parser. If you want to study the LALR algorithm and the refinements that have made Earley's algorithm practical, that's great, but you shouldn't really have to puzzle over shift-reduce conflicts in order to get up and running. 3. It's very tempting for novices to see syntax as the important aspect of programming, because it's what you can see most easily; but of course it is purely superficial. By teaching lexers and parsers as the first step, and thus implicitly the most important step, in writing a compiler, you're hurting novice programmers by reinforcing that misconception instead of helping them by ripping it away.
- tester756 5y ago
  >3. It's very tempting for novices to see syntax as the important aspect of programming, because it's what you can see most easily; but of course it is purely superficial. By teaching lexers and parsers as the first step, and thus implicitly the most important step, in writing a compiler, you're hurting novice programmers by reinforcing that misconception instead of helping them by ripping it away. I'm newbie in those topics and I don't see it this way for me lexer is way of building abstraction above syntax. Syntax is important from perspective of language design team, meanwhile compiler is operating on abstraction
  - 5y ago

mpd 5y ago

I'm not your parent commentor, but imo, it's because there's is an enormous amount of existing info out there about these two steps. For these kinds of projects, they are necessary whether you're translating to another language (i.e. generating code) or not. And everyone starts with lexing and parsing. And just repeats the same old shit.

dmitriid 5y ago

There's too much emphasis on parsing in all the teaching materials on compiling. And there's rarely any good stuff on the actual hard problems

habibur 5y ago
> And there's rarely any good stuff on the actual hard problems So what are those hard problems?
- sim_card_map 5y ago
  codegen, optimizations, ast transformations
- dmitriid 5y ago
  The actual compilation. Actual programming language design. Types, type inference, function implementations, error handling and recovery, incremental compilation, interfacing with the external world, code generation, fixing problems in code generation tools, CPU architecture concerns... The toy languages that most learning materials make can be parsed with regexps and is definitely not something to spend a lot of time on at the beginning. Because actual parsing problems are never addressed in these materials, and can come later (example of concerns: https://matklad.github.io//2018/06/06/modern-parser-generator.html https://matklad.github.io//2018/06/06/modern-parser-generato...).
  - rualca 5y ago
    > The actual compilation. Actual programming language design. Isn't that intimately tied to parsing? I mean, how do you expect to implement a parser if you have no idea if a specific programming construct can actually be parsed, or what are the implications in compilation time of doing things one way or another?
    - dmitriid 5y ago
      > Isn't that intimately tied to parsing? Not entirely > how do you expect to implement a parser if you have no idea if a specific programming construct can actually be parsed Every construct can be parsed. However, there are significantly more aspects to language design than "parsing programming constructs". For example, C++ header/implementation split with no real module system affects compilation orders of magnitudes more than the parsing rules for C++.

kragen 5y ago

I think there are several good reasons not to start with the lexer and parser: 1. If you write a code generator whose input is something like Forth ("operators are separated by spaces") you have written a compiler. If you write a lexer and parser, you have not written a compiler. So, if we want to develop the compiler incrementally, we should start with a back end, then add a front end. 2. Parsing and lexing are much more theoretically tractable than code generation, which is only now beginning to succumb to systematic formalization after nearly 50 years of efforts. Consequently there are enormous edifices of theory and tooling built up around parsing and lexing, which are fascinating and delightful. The problems is that people tend to teach all this stuff, which is not actually necessary to write a useful parser. Rather than studying Chomsky's hierarchy of languages and automata theory, you're probably better off just learning how context-free languages work, a little about refactoring them to eliminate left recursion, and about chronological backtracking search and applying it to the parsing problem, and then just write a recursive-descent parser. If you want to study the LALR algorithm and the refinements that have made Earley's algorithm practical, that's great, but you shouldn't really have to puzzle over shift-reduce conflicts in order to get up and running. 3. It's very tempting for novices to see syntax as the important aspect of programming, because it's what you can see most easily; but of course it is purely superficial. By teaching lexers and parsers as the first step, and thus implicitly the most important step, in writing a compiler, you're hurting novice programmers by reinforcing that misconception instead of helping them by ripping it away.

tester756 5y ago
>3. It's very tempting for novices to see syntax as the important aspect of programming, because it's what you can see most easily; but of course it is purely superficial. By teaching lexers and parsers as the first step, and thus implicitly the most important step, in writing a compiler, you're hurting novice programmers by reinforcing that misconception instead of helping them by ripping it away. I'm newbie in those topics and I don't see it this way for me lexer is way of building abstraction above syntax. Syntax is important from perspective of language design team, meanwhile compiler is operating on abstraction
- 5y ago