5 ms·
Let's write a compiler, part 5: A code generator
- p4bl0 5y agoThis series of blogposts is amusing but it is kind of frustrating too: the end result is merely a compiler. It is a source to source translator from a source language which is very very similar to a subset of the target language. Well I guess you can call that a compiler, but it really doesn't teach the readers what an actual compiler looks like. Exaggerating a little, I would say that it feels like a few calls to sed could do the same thing using regexp.
- chrisseaton 5y agoLots of compiler tutorials are like this - there's very little out there to explain how compilers really work. This is my effort - trying to show genuine data structures and processes. https://github.com/chrisseaton/rhizome https://github.com/chrisseaton/rhizome
- divtiwari 5y agoHey, I just skimmed through this, it seems to be brilliant!! Just one question, can one go through this tutorial without much knowledge about Ruby? Should one read it with 'Ruby Under a Microscope'? I have a decent enough knowledge about compilers, have gone through Thorsten Ball's, 'Writing an Interpreter in Go'.
- chrisseaton 5y agoOnly need to know basic Ruby programming - no libraries or deep knowledge. But the project is unfinished I’m afraid - you will find dead-ends.
- divtiwari 5y agoOh I see, thanks for the info!! I'll see what I can gain out of it. Thanks once again for the info!!
- junon 5y agoIt could not. Unless your language is regular class, then regular expressions cannot parse the language accurately.
- im3w1l 5y agoYou can build a parser around regexes though, where most of the code is regexes and then you have a little bit of code to deal with the irregularity. For instance consider arithmethic expressions consisting of constants, -, +, *, /, and parentheses. You could evaluate that using something like (expression to parse a numeral is left as exercise to reader). while expression is not a numeral replace all "\((NUMERAL)\)" with first group if find first "(NUMERAL)([*/])(NUMERAL)" replace with result else if find first "(NUMERAL)([+-])(NUMERAL)" replace with result
- ashton314 5y agoWhat you are doing there is conflating the lexing phase and the parsing phase. Regexes are perfect for recognizing tokens, but for a language with nested parentheses, you must have a push-down automaton to process it. Otherwise, you will not be able to verify that your delimiters are matched.
- im3w1l 5y agoThat's one way of looking at it I suppose. But it does require stretching the definition of token a little bit if you are eg using regexes to identify arbitrarily long multiplications. I think my prefered perspective is that you are using regexes to parse regular sublanguages and then something more powerful (e.g. a push down automaton, but there are even more powerful parsers than that) to bind those sub-parsers together.
- p4bl0 5y agoThat's why I said "if feels like". But nonetheless even if regexps would not be able to validate syntax, it does not mean that the source-to-source transformation could not be (mostly) achieved using them, because the source and target language are quite similar. Imagine a new language exactly like C but every semicolon is replaced by a duck "\_o<". You could not parse it using regexps, but regexp could mostly "compile" it to C as it only requires to look for all "\_o<" to replace them with ";".
- peepholeoptim 5y agoC is his hammer, but tenure is tenure.
- ofiryanai 5y agoHi all, I have a bit off topic question but seems related. I'm trying to write sort of a SQL compiler. The current goal is to analyze queries and find similarities, later maybe to translate between sql dialects. I found Uber's QueryParser[1] but it's in haskell, so I started wrapping the python sqlparse[2] library and implement a Visitor to traverse their weird AST. 1. How close is it to implementing a compiler? 2. Is there theory you can suggest further reading for that matter? 3. Would you use a different language/library then I picked? Thanks :) [1] https://github.com/uber/queryparser https://github.com/uber/queryparser [2] https://github.com/andialbrecht/sqlparse https://github.com/andialbrecht/sqlparse
- codr7 5y agoDepends on the complexity of your queries, but if you have a narrow subset that you're interested in, implementing a recursive descent parser for just those parts of the syntax that fits your problem like a glove could be a better solution. https://github.com/codr7/swifties https://github.com/codr7/swifties
- ofiryanai 5y agoI'm aiming to analyze all the BI queries in my organization, some of them are quite complicated and most of them are hundreds lines of sql. Thanks for the direction, I'll dive into it
- refneb 5y agoI’ve been playing with ANTLR[1] and pretty happy with the generated parser. You can find sql grammar on GitHub[2] [1] https://www.antlr.org/ https://www.antlr.org/ [2] https://github.com/antlr/grammars-v4 https://github.com/antlr/grammars-v4
- setheron 5y agoLookup Apache Calcite that does most of what you describe. You can contribute to the project.
- deleted 5y ago
- ModernMech 5y agoThe ontology of compilation seems to be the only thing people are engaging with regarding this blog post series here on HN, so I don't see why this series is on the front page every day when the discussion is so stale. If you want to talk about how compiling to C is or isn't compiling, save yourself the trouble, everything has been said already in the past week (including this thread): https://news.ycombinator.com/item?id=28184187 https://news.ycombinator.com/item?id=28184187 https://news.ycombinator.com/item?id=28199971 https://news.ycombinator.com/item?id=28199971 https://news.ycombinator.com/item?id=28209950 https://news.ycombinator.com/item?id=28209950 Alternatively, congrats to the author, for managing to get his blog on the front page 4 out of the last 5 days in a row. Has that ever been done before on HN, I wonder? It's gotta be some sort of record.
- chrisseaton 5y ago> so I don't see why this series is on the front page every day when the discussion is so stale People submit it and upvote it.
- ingve 5y agoCrafting Interpreters [0] just came out in print and was widely discussed here [1], there's a Lang Jam [2] event happening this weekend which also spawned a useful HN discussion [3], so I just think a lot of readers here are interested in simple and fun language development tutorials and inspiration. But it is curious, part 2 of Brian Callahan's series [4] failed to get any traction here, maybe people aren't that into lexing? [0] https://craftinginterpreters.com/ https://craftinginterpreters.com/ [1] https://news.ycombinator.com/item?id=27997167 https://news.ycombinator.com/item?id=27997167 [2] https://github.com/langjam/langjam https://github.com/langjam/langjam [3] https://news.ycombinator.com/item?id=28021161 https://news.ycombinator.com/item?id=28021161 [4] https://briancallahan.net/blog/20210815.html https://briancallahan.net/blog/20210815.html
- imvetri 5y agoCode generators are fun !
- Joker_vD 5y agoAlright, let's write a comment about the actual implementation described in the post. So: why make the symbol table a linked list that has its head fixed at "main" and grows and shrinks at its tail? It is used in a stack-like fashion, so why not reverse the order: put "main" at its tail and grow and shrink it at its head? That simplifies adding symbols: you keep checking for duplicates until the depth of the symbols you see drops below your depth, and actual insertion doesn't need the pointer to the list's last element, you insert at the head. That simplifies destroying symbols: you keep popping the head until you meet a TOK_PROCEDURE at which point you stop. That simplifies looking symbols up: you search until the first name match and return it immediately, instead of remembering the last seen matching symbol while traversing the full list. I am not even talking about efficiency (although of course this implementation is also more efficient in all 3 use cases), it's about code simplicity: the less context you need to maintain during a list traversal, the easier it is to understand what this traversal looks for.