13 ms·

> However, once I start constructing a parser, progress slows to a crawl I can't really relate here. The parser is the easiest part of a compiler; the work onl

> However, once I start constructing a parser, progress slows to a crawl I can't really relate here. The parser is the easiest part of a compiler; the work only increases from there. I feel like if you ran out of steam at the parser, you never had enough steam to write a whole compiler. I don't think removing the parser will take you across the finish line if you otherwise were running out of steam. My advice is to write your language in vertical slices. Write the parsing, semantic checking, and code generation for the simplest features first and progressively add feature slices, rather than trying to write the entire parser for a fully-baked language before proceeding. Consider including "print" as a built-in statement so you can print things out (and thus write tests) before you have working expressions and function calls.

tester756 3y ago
Parser for "4-lulz language" or something that will also be used in IDEs, so has robust error recovery, can perform partial updates, etc? I feel it's just that it is possible to say that work on the parser side is completed meanwhile optimizations? you can probably endlessly improve stuff
- electroly 3y ago
  Talking about the same thing as the article: hobby/learning languages. If you click the links to the author's projects you can see they have yet to finish a compiler or really even come close. Definitely not talking about sophisticated production-grade languages here; OP is trying to complete their first compiler at all. I think writing compilers is a really neat and useful learning project but you have to be smart about not biting off more than you can chew. I would absolutely recommend not supporting incremental compilation or error recovery in your first compiler. Just stop everything at the first error. Save that for your second hobby compiler, or better yet, the first commercial compiler that you get paid to work on.
  - 082349872349872 3y ago
    > writing compilers is a really neat and useful learning project but you have to be smart about not biting off more than you can chew. Writing compilers is an especially useful project when you learn roughly how much you can chew. (Algol 68 is a nice example of a bunch of smart people, with relevant domain expertise, biting off way too much for the machines of the time.)
- paulddraper 3y ago
  I think you're reaching higher
smasher164 3y ago
> I don't think removing the parser will take you across the finish line if you otherwise were running out of steam. Everyone's different. The problem with parsing isn't the difficulty, but rather the potential for endless bikeshedding. You're having to make a ton of opinionated decisions that in turn produce more questions about your syntax. If your personality is like mine in that it's a bit obsessive "completing" a phase, then parsing feels like an endless quagmire. In comparison, AST -> type inference -> codegen feels more structured and straightforward.
- rewmie 3y ago
  > The problem with parsing isn't the difficulty, but rather the potential for endless bikeshedding. You're having to make a ton of opinionated decisions that in turn produce more questions about your syntax. I fail to see your point. OP was commenting on the technical difficulty of actually rolling out a parser, but you're not arguing the technical side. You're arguing the project management sides where "endless bike shedding" results in code churn. That's not the parser's fault, but the person's fault. You're bound to get hung up on details regardless of what part of the project you're working on, because it's not the parser compelling you to get hung up on details.
  - convolvatron 3y ago
    but it is. language syntax design is a total sinkhole. you give me a spec - I'll crap out a parser for you. tell me to design a textual language and chances are pretty good I won't get back to you. and if I do, chances are really high that you're gonna want me to do something different.
    - rewmie 3y ago
      > language syntax design is a total sinkhole. Sintax design, not implementing a parser. You're arguing that you might not make up your mind and feel compelled to churn parts of your implementation due to your indecisiveness. That's not the parser's fault, and I fail to see how anyone avoids this indecisiveness by replacing a proper parser with a bunch of ad-hoc hacks. > chances are really high that you're gonna want me to do something different. So what? Just work on it. How many parser generator tools are there? It's as if people have been doing this for decades.
throwaway894345 3y ago
I disagree. Parsers are tedious to write and obscure corner cases crop up over the course of development that make us completely change our approach to parsing or the grammar. You have to be pretty expert to get the grammar and parsing to work right the first time. Type inference is tough, but it’s not difficult the way writing a parser is. Codegen (suboptimal but functional) is even easier. But yeah, write the vertical slice and implement a print builtin are good tips.
armchairhacker 3y ago
My first language, I spent months on the parser and then gave up. As I learned more I got better until now, I can write a hand-rolled parser for a simple language in under a day. There's a technique to making syntax which is easy to read/write and easy to parse with a look-ahead parser. For more complicated languages, there are tools like tree-sitter and ANTLR4. You can even extend an existing tree-sitter grammar to augment a language without having to re-write the base language parser.
musicale 3y ago
> Consider including "print" as a built-in statement I really wish Python 3 had taken that advice. ;-)
KRAKRISMOTT 3y ago
I have a better idea. Do the code gen while writing out the parser. It won't be the fastest language and the codebase would be a mess and you will most likely have correctness problems but it's the fastest way to production. The classic 3 stage compiler design (tokenizing/lexing/parsing -> AST -> optional lowering/bytecode etc. -> JIT/code gen) is only important if you want to semantic analysis and optimization. They are kinda redundant for prototyping (plus cranelift and LLVM exist, you can focus on the backend after you get your language design in order). Do a quick peephole optimization pass on the generated instructions if things get too slow. If we are being honest here, most people just want their own better Python/Go/Rust. I doubt your average engineer is truly interested in the subtleties of compiler design and memory layout optimization. A single pass compiler, built like the olden day of yore, is the best option for most. https://en.m.wikipedia.org/wiki/One-pass_compiler https://en.m.wikipedia.org/wiki/One-pass_compiler Leave the 3 stage design for homework (or if you are being paid to do so).
brabel 3y ago
> rather than trying to write the entire parser for a fully-baked language before proceeding. I see programmers, even experienced ones, making this mistake all the time when implementing larger changes. They try to get the whole thing implemented at once even when it's obvious from the outset that there are multiple sub-features that can be more easily implemented separately, so one can focus all their efforts on it, make sure it's well tested and clean, before moving on to the next sub-feature.
kevingadd 3y ago
Yep, it was tremendously correct to start with (iirc) 'assert_eq' and 'print' as the two first statements in my language when building out the compiler. It meant I could start writing a test suite and assemble a larger and larger set of functionality as I went. On the other hand I do really hate the task of writing parsers, so I can relate to people who think it's the worst/most difficult part. But I think other parts are probably harder, like getting type system stuff right.

tester756 3y ago

Parser for "4-lulz language" or something that will also be used in IDEs, so has robust error recovery, can perform partial updates, etc? I feel it's just that it is possible to say that work on the parser side is completed meanwhile optimizations? you can probably endlessly improve stuff

electroly 3y ago
Talking about the same thing as the article: hobby/learning languages. If you click the links to the author's projects you can see they have yet to finish a compiler or really even come close. Definitely not talking about sophisticated production-grade languages here; OP is trying to complete their first compiler at all. I think writing compilers is a really neat and useful learning project but you have to be smart about not biting off more than you can chew. I would absolutely recommend not supporting incremental compilation or error recovery in your first compiler. Just stop everything at the first error. Save that for your second hobby compiler, or better yet, the first commercial compiler that you get paid to work on.
- 082349872349872 3y ago
  > writing compilers is a really neat and useful learning project but you have to be smart about not biting off more than you can chew. Writing compilers is an especially useful project when you learn roughly how much you can chew. (Algol 68 is a nice example of a bunch of smart people, with relevant domain expertise, biting off way too much for the machines of the time.)
paulddraper 3y ago
I think you're reaching higher

smasher164 3y ago

> I don't think removing the parser will take you across the finish line if you otherwise were running out of steam. Everyone's different. The problem with parsing isn't the difficulty, but rather the potential for endless bikeshedding. You're having to make a ton of opinionated decisions that in turn produce more questions about your syntax. If your personality is like mine in that it's a bit obsessive "completing" a phase, then parsing feels like an endless quagmire. In comparison, AST -> type inference -> codegen feels more structured and straightforward.

rewmie 3y ago
> The problem with parsing isn't the difficulty, but rather the potential for endless bikeshedding. You're having to make a ton of opinionated decisions that in turn produce more questions about your syntax. I fail to see your point. OP was commenting on the technical difficulty of actually rolling out a parser, but you're not arguing the technical side. You're arguing the project management sides where "endless bike shedding" results in code churn. That's not the parser's fault, but the person's fault. You're bound to get hung up on details regardless of what part of the project you're working on, because it's not the parser compelling you to get hung up on details.
- convolvatron 3y ago
  but it is. language syntax design is a total sinkhole. you give me a spec - I'll crap out a parser for you. tell me to design a textual language and chances are pretty good I won't get back to you. and if I do, chances are really high that you're gonna want me to do something different.
  - rewmie 3y ago
    > language syntax design is a total sinkhole. Sintax design, not implementing a parser. You're arguing that you might not make up your mind and feel compelled to churn parts of your implementation due to your indecisiveness. That's not the parser's fault, and I fail to see how anyone avoids this indecisiveness by replacing a proper parser with a bunch of ad-hoc hacks. > chances are really high that you're gonna want me to do something different. So what? Just work on it. How many parser generator tools are there? It's as if people have been doing this for decades.

throwaway894345 3y ago

I disagree. Parsers are tedious to write and obscure corner cases crop up over the course of development that make us completely change our approach to parsing or the grammar. You have to be pretty expert to get the grammar and parsing to work right the first time. Type inference is tough, but it’s not difficult the way writing a parser is. Codegen (suboptimal but functional) is even easier. But yeah, write the vertical slice and implement a print builtin are good tips.

armchairhacker 3y ago

My first language, I spent months on the parser and then gave up. As I learned more I got better until now, I can write a hand-rolled parser for a simple language in under a day. There's a technique to making syntax which is easy to read/write and easy to parse with a look-ahead parser. For more complicated languages, there are tools like tree-sitter and ANTLR4. You can even extend an existing tree-sitter grammar to augment a language without having to re-write the base language parser.

musicale 3y ago

> Consider including "print" as a built-in statement I really wish Python 3 had taken that advice. ;-)

KRAKRISMOTT 3y ago

I have a better idea. Do the code gen while writing out the parser. It won't be the fastest language and the codebase would be a mess and you will most likely have correctness problems but it's the fastest way to production. The classic 3 stage compiler design (tokenizing/lexing/parsing -> AST -> optional lowering/bytecode etc. -> JIT/code gen) is only important if you want to semantic analysis and optimization. They are kinda redundant for prototyping (plus cranelift and LLVM exist, you can focus on the backend after you get your language design in order). Do a quick peephole optimization pass on the generated instructions if things get too slow. If we are being honest here, most people just want their own better Python/Go/Rust. I doubt your average engineer is truly interested in the subtleties of compiler design and memory layout optimization. A single pass compiler, built like the olden day of yore, is the best option for most. https://en.m.wikipedia.org/wiki/One-pass_compiler https://en.m.wikipedia.org/wiki/One-pass_compiler Leave the 3 stage design for homework (or if you are being paid to do so).

brabel 3y ago

> rather than trying to write the entire parser for a fully-baked language before proceeding. I see programmers, even experienced ones, making this mistake all the time when implementing larger changes. They try to get the whole thing implemented at once even when it's obvious from the outset that there are multiple sub-features that can be more easily implemented separately, so one can focus all their efforts on it, make sure it's well tested and clean, before moving on to the next sub-feature.

kevingadd 3y ago

Yep, it was tremendously correct to start with (iirc) 'assert_eq' and 'print' as the two first statements in my language when building out the compiler. It meant I could start writing a test suite and assemble a larger and larger set of functionality as I went. On the other hand I do really hate the task of writing parsers, so I can relate to people who think it's the worst/most difficult part. But I think other parts are probably harder, like getting type system stuff right.