4 ms·

Yes exactly :), I meant that it does everything from lexing the source code to emitting bytecode instructions (and provides the VM to run this bytecode). It is

by p4bl0 4y ago

Yes exactly :), I meant that it does everything from lexing the source code to emitting bytecode instructions (and provides the VM to run this bytecode). It is not a part of a compiler, it is a tiny, simple, stupid, but complete, compiler.

kragen 4y ago
I want to preface this comment with the fact that I'm terrible at teaching and know nothing about it, so it's entirely possible that your example compiler is much better for pedagogical purposes than the ones I'm going to suggest here. But I think they all have a significantly higher "power density", which at least from my point of view makes them more interesting and inspiring. I think mostly people are complaining that the language it compiles isn't a programming language, so it omits a lot of things that compilers for programming languages have to handle. And so it feels a bit like much ado about nothing: because there are no variables, loops, or values beyond single-bit values, the program it compiles can only ever produce a constant, predetermined output, and that output can only ever be 1 or 0. So the compiler is just sort of a particularly awkward and inefficient way to output that 1 or 0. A compiler for Brainf*** or SK-combinators or the untyped λ-calculus would be "more real" in that sense. More practical, and maybe also fitting into 150 lines, might be a language for something like a programmable calculator (untested sketch): program ::= stmt | stmt ";" program. stmt ::= "print" numexpr | variable ":=" numexpr | "while" numexpr comparator numexpr "{" program "}". comparator ::= "<" | ">" | "==" | "!=" | "<=" | ">=". numexpr ::= term | numexpr "+" term | numexpr "-" term. term ::= factor | term "*" factor | term "/" factor | term "%" factor. factor ::= atom | "-" atom | "+" atom | atom "^" factor. atom ::= "(" numexpr ")" | numeric_constant | variable. That's powerful enough to conveniently write, for example, a numerical root finding program for an arbitrary arithmetic expression. Maybe for pedagogical purposes (which, as I said, I am totally ignorant of) it would be useful to leave out one of the levels of precedence and some of the operators and have the students add them as an exercise. But I think that within a complexity budget of 150 lines of code you can maybe be even more ambitious than that. Darius Bacon's example compiler in https://github.com/darius/parson/blob/master/eg_calc_compile.py https://github.com/darius/parson/blob/master/eg_calc_compile... is a bit more stripped down than that, but in its 32 lines of code it compiles arithmetic assignment statements to a three-address RISC-like code (though using an unbounded number of registers). https://github.com/darius/parson/blob/master/eg_calc_to_rpn.py https://github.com/darius/parson/blob/master/eg_calc_to_rpn.... is a 16-line version that compiles the same language to a stack machine like your tutorial example: from parson import Grammar, alter g = Grammar(r""" stmt* :end. stmt : ident '=' exp0 ';' :assign. exp0 : exp1 ('+' exp1 :'add')*. exp1 : exp2 ('*' exp2 :'mul')*. exp2 : '(' exp0 ')' | /(\d+)/ | ident :'fetch'. ident : /([A-Za-z]+)/. FNORD ~: /\s*/. """)(assign=alter(lambda name, *rpn: rpn + (name, 'store'))) ## print ' '.join(g('v = 42 * (5+3) + 2*2; v = v + 1;')) #. 42 5 3 add mul 2 2 mul add v store v fetch 1 add v store Of course that still contains no control flow, so it's still not a very convincing example of why you'd take the extra trouble to write a compiler rather than an interpreter. In 66 lines of code in https://github.com/kragen/peg-bootstrap/blob/master/peg.md https://github.com/kragen/peg-bootstrap/blob/master/peg.md I wrote an example compiler which compiles a PEG grammar into a JavaScript parser for that grammar. Admittedly those 66 lines do not include an implementation of JavaScript to run the code on. It compiles the language it's written in. In 132 lines of code in https://github.com/kragen/stoneknifeforth/blob/master/tinyboot1.tbf1 https://github.com/kragen/stoneknifeforth/blob/master/tinybo... I wrote an example compiler which compiles a crippled Forth dialect into i386 machine code, including an ELF header so you can run the result. It also compiles the language it's written in. It also doesn't include an i386 emulator to run it on. In 83 lines of code in http://canonical.org/~kragen/sw/dev3/neelcompiler.ml http://canonical.org/~kragen/sw/dev3/neelcompiler.ml Neel Krishnaswami wrote a compiler from the untyped λ-calculus to a simple assembly language for a register machine. It also doesn't include an implementation of the assembly language. In 18 lines of code in http://canonical.org/~kragen/sw/dev3/meta5ix.m5 http://canonical.org/~kragen/sw/dev3/meta5ix.m5, a simplification of Val Schorre's META-II, I wrote a compiler from grammar descriptions to an assembly code for a parsing-oriented virtual machine. It compiles the language it's written in. A Python interpreter for the machine is in http://canonical.org/~kragen/sw/dev3/meta5ixrun.py http://canonical.org/~kragen/sw/dev3/meta5ixrun.py (109 lines of code) and a precompiled version of the compiler-compiler for bootstrapping is in http://canonical.org/~kragen/sw/dev3/meta5ix.generated.m5asm http://canonical.org/~kragen/sw/dev3/meta5ix.generated.m5asm. It does recursive descent with only a single token of backtracking; a minimal summary of the most eccentric aspects of its syntax is that "" encloses text to expect on input, {} encloses text to output, <<>> encloses a token to parse from the input that might be copied to the output with $it (or unindented with @it), commas separate alternatives, and [x] is zero or more repetitions of x. - program: ["-" name @it ":" terms {return}, "#" [:notin ""]] - terms: term ["," {continue $choice} term] @choice - term: (factor {else $seq}, output) [factor {assert}, output] @seq - factor: string {literal $it} , "(" terms ")" , "[" @many terms {continue $many} "]" , name {call $it} , "<<" {begin} terms ">>" {end} , ":fnord" {fnord} factor , ":notin" string {notin $it} , ":between" string {between $it} - output: (quasiquote, "@" {dedent} (var, quasiquote)) {writeline} - quasiquote: "{" [(<<ch [ch]>>, "\" <<:notin "">>) {say "$it"}, "$" var] "}" - ch: :notin "$}\" - var: "it" {copyinput}, name {gen $it} - string: :fnord <<'"' [:notin '"'] '"', "'" [:notin "'"] "'">> - name: :fnord <<letter [letter, :between "09"]>> - letter: :between "az", :between "AZ", :between "__" A slightly incompatible variant of Meta5ix which instead compiles itself to C is in http://canonical.org/~kragen/sw/dev3/meta5ix2c.m5 http://canonical.org/~kragen/sw/dev3/meta5ix2c.m5 (133 lines of code, depending on how you count). (No C compiler is included.) The precompiled C output for bootstrapping is in http://canonical.org/~kragen/sw/dev3/meta5ix2c.c http://canonical.org/~kragen/sw/dev3/meta5ix2c.c. Meta5ix is extremely weak and limited, really only enough for a compiler front-end. Dave Long described META-II as "a field-improvised lever", which I think is still apt for Meta5ix. Now, again, I am totally ignorant about pedagogy, so I could be wrong about this, but I think that probably self-compiling compiler-compilers like most of my examples here are not the best tutorial examples, because they refer to themselves in complex and confusing ways. But I feel like they represent pretty good evidence that you can do something a lot more interesting in a 150-line-of-code compiler than evaluating a Boolean circuit.
- p4bl0 4y ago
  As I wrote elsewhere, I use this tiny compiler as an introduction to the course, which goes into much more details and of course covers how to compile realistic programming languages (with multiple types of data, variables, conditional and loop control structures, functions, pointers, list and arbitrary structures, etc.) down to assembly code. Thanks for all your interesting links :).
  - kragen 4y ago
    Oh, of course I didn't mean to imply that that was all you were teaching! I was just thinking that maybe a compiler that did something a little more impressive (for the effort required to understand it) might arouse more interest and provoke less No-True-Scotsman criticism. Though, honestly, stupid criticism is the vast majority of what HN is capable of, and you shouldn't take the uninformed criticism of HN commenters too seriously. Of course the compilers you teach later on with variables, conditionals, loops, multiple types, pointers, etc., are more capable and thus more impressive. But presumably they also require even more effort to understand than the "tiny" compiler you're presenting in this gist. I hope you find many things you enjoy in the links! Even if they aren't directly useful as pedagogical examples, they do demonstrate, for example, that a plain recursive-descent parser is already "two-pass" enough to handle assignment to variables, contra the statement in your gist, "In a more complex language (e.g., adding support for assignment to variables), we would need an additional front-end pass after the parser." Indeed, StoneKnifeForth is a single-pass compiler in less code than your tiny compiler that supports not only assignment to variables, but also arrays, I/O, conditionals, loops, defining and calling subroutines with arguments, pointers, ELF output, and i386 machine code generation with peephole optimization. And it's not because it's written in a more powerful language than OCaml; it's written in the profoundly substandard dialect of Forth that it implements, which lacks Forth's compile-time metaprogramming facilities and in which only one character is significant in identifiers. I was wondering if maybe my memory of StoneKnifeForth was wrong and it wasn't actually 132 lines of code, because that sure sounds like a lot of functionality in 132 lines of code. It turns out it's 117 lines of code, not 132. (The build process builds a version with comments stripped out, called trimmed.tbf1, and verifies that it compiles identically, to find this out.) So maybe you can learn some things from them that will enable you to write better compilers!

by p4bl0 4y ago

kragen 4y ago

I want to preface this comment with the fact that I'm terrible at teaching and know nothing about it, so it's entirely possible that your example compiler is much better for pedagogical purposes than the ones I'm going to suggest here. But I think they all have a significantly higher "power density", which at least from my point of view makes them more interesting and inspiring. I think mostly people are complaining that the language it compiles isn't a programming language, so it omits a lot of things that compilers for programming languages have to handle. And so it feels a bit like much ado about nothing: because there are no variables, loops, or values beyond single-bit values, the program it compiles can only ever produce a constant, predetermined output, and that output can only ever be 1 or 0. So the compiler is just sort of a particularly awkward and inefficient way to output that 1 or 0. A compiler for Brainf*** or SK-combinators or the untyped λ-calculus would be "more real" in that sense. More practical, and maybe also fitting into 150 lines, might be a language for something like a programmable calculator (untested sketch): program ::= stmt | stmt ";" program. stmt ::= "print" numexpr | variable ":=" numexpr | "while" numexpr comparator numexpr "{" program "}". comparator ::= "<" | ">" | "==" | "!=" | "<=" | ">=". numexpr ::= term | numexpr "+" term | numexpr "-" term. term ::= factor | term "*" factor | term "/" factor | term "%" factor. factor ::= atom | "-" atom | "+" atom | atom "^" factor. atom ::= "(" numexpr ")" | numeric_constant | variable. That's powerful enough to conveniently write, for example, a numerical root finding program for an arbitrary arithmetic expression. Maybe for pedagogical purposes (which, as I said, I am totally ignorant of) it would be useful to leave out one of the levels of precedence and some of the operators and have the students add them as an exercise. But I think that within a complexity budget of 150 lines of code you can maybe be even more ambitious than that. Darius Bacon's example compiler in https://github.com/darius/parson/blob/master/eg_calc_compile.py https://github.com/darius/parson/blob/master/eg_calc_compile... is a bit more stripped down than that, but in its 32 lines of code it compiles arithmetic assignment statements to a three-address RISC-like code (though using an unbounded number of registers). https://github.com/darius/parson/blob/master/eg_calc_to_rpn.py https://github.com/darius/parson/blob/master/eg_calc_to_rpn.... is a 16-line version that compiles the same language to a stack machine like your tutorial example: from parson import Grammar, alter g = Grammar(r""" stmt* :end. stmt : ident '=' exp0 ';' :assign. exp0 : exp1 ('+' exp1 :'add')*. exp1 : exp2 ('*' exp2 :'mul')*. exp2 : '(' exp0 ')' | /(\d+)/ | ident :'fetch'. ident : /([A-Za-z]+)/. FNORD ~: /\s*/. """)(assign=alter(lambda name, *rpn: rpn + (name, 'store'))) ## print ' '.join(g('v = 42 * (5+3) + 2*2; v = v + 1;')) #. 42 5 3 add mul 2 2 mul add v store v fetch 1 add v store Of course that still contains no control flow, so it's still not a very convincing example of why you'd take the extra trouble to write a compiler rather than an interpreter. In 66 lines of code in https://github.com/kragen/peg-bootstrap/blob/master/peg.md https://github.com/kragen/peg-bootstrap/blob/master/peg.md I wrote an example compiler which compiles a PEG grammar into a JavaScript parser for that grammar. Admittedly those 66 lines do not include an implementation of JavaScript to run the code on. It compiles the language it's written in. In 132 lines of code in https://github.com/kragen/stoneknifeforth/blob/master/tinyboot1.tbf1 https://github.com/kragen/stoneknifeforth/blob/master/tinybo... I wrote an example compiler which compiles a crippled Forth dialect into i386 machine code, including an ELF header so you can run the result. It also compiles the language it's written in. It also doesn't include an i386 emulator to run it on. In 83 lines of code in http://canonical.org/~kragen/sw/dev3/neelcompiler.ml http://canonical.org/~kragen/sw/dev3/neelcompiler.ml Neel Krishnaswami wrote a compiler from the untyped λ-calculus to a simple assembly language for a register machine. It also doesn't include an implementation of the assembly language. In 18 lines of code in http://canonical.org/~kragen/sw/dev3/meta5ix.m5 http://canonical.org/~kragen/sw/dev3/meta5ix.m5, a simplification of Val Schorre's META-II, I wrote a compiler from grammar descriptions to an assembly code for a parsing-oriented virtual machine. It compiles the language it's written in. A Python interpreter for the machine is in http://canonical.org/~kragen/sw/dev3/meta5ixrun.py http://canonical.org/~kragen/sw/dev3/meta5ixrun.py (109 lines of code) and a precompiled version of the compiler-compiler for bootstrapping is in http://canonical.org/~kragen/sw/dev3/meta5ix.generated.m5asm http://canonical.org/~kragen/sw/dev3/meta5ix.generated.m5asm. It does recursive descent with only a single token of backtracking; a minimal summary of the most eccentric aspects of its syntax is that "" encloses text to expect on input, {} encloses text to output, <<>> encloses a token to parse from the input that might be copied to the output with $it (or unindented with @it), commas separate alternatives, and [x] is zero or more repetitions of x. - program: ["-" name @it ":" terms {return}, "#" [:notin ""]] - terms: term ["," {continue $choice} term] @choice - term: (factor {else $seq}, output) [factor {assert}, output] @seq - factor: string {literal $it} , "(" terms ")" , "[" @many terms {continue $many} "]" , name {call $it} , "<<" {begin} terms ">>" {end} , ":fnord" {fnord} factor , ":notin" string {notin $it} , ":between" string {between $it} - output: (quasiquote, "@" {dedent} (var, quasiquote)) {writeline} - quasiquote: "{" [(<<ch [ch]>>, "\" <<:notin "">>) {say "$it"}, "$" var] "}" - ch: :notin "$}\" - var: "it" {copyinput}, name {gen $it} - string: :fnord <<'"' [:notin '"'] '"', "'" [:notin "'"] "'">> - name: :fnord <<letter [letter, :between "09"]>> - letter: :between "az", :between "AZ", :between "__" A slightly incompatible variant of Meta5ix which instead compiles itself to C is in http://canonical.org/~kragen/sw/dev3/meta5ix2c.m5 http://canonical.org/~kragen/sw/dev3/meta5ix2c.m5 (133 lines of code, depending on how you count). (No C compiler is included.) The precompiled C output for bootstrapping is in http://canonical.org/~kragen/sw/dev3/meta5ix2c.c http://canonical.org/~kragen/sw/dev3/meta5ix2c.c. Meta5ix is extremely weak and limited, really only enough for a compiler front-end. Dave Long described META-II as "a field-improvised lever", which I think is still apt for Meta5ix. Now, again, I am totally ignorant about pedagogy, so I could be wrong about this, but I think that probably self-compiling compiler-compilers like most of my examples here are not the best tutorial examples, because they refer to themselves in complex and confusing ways. But I feel like they represent pretty good evidence that you can do something a lot more interesting in a 150-line-of-code compiler than evaluating a Boolean circuit.

p4bl0 4y ago
As I wrote elsewhere, I use this tiny compiler as an introduction to the course, which goes into much more details and of course covers how to compile realistic programming languages (with multiple types of data, variables, conditional and loop control structures, functions, pointers, list and arbitrary structures, etc.) down to assembly code. Thanks for all your interesting links :).
- kragen 4y ago
  Oh, of course I didn't mean to imply that that was all you were teaching! I was just thinking that maybe a compiler that did something a little more impressive (for the effort required to understand it) might arouse more interest and provoke less No-True-Scotsman criticism. Though, honestly, stupid criticism is the vast majority of what HN is capable of, and you shouldn't take the uninformed criticism of HN commenters too seriously. Of course the compilers you teach later on with variables, conditionals, loops, multiple types, pointers, etc., are more capable and thus more impressive. But presumably they also require even more effort to understand than the "tiny" compiler you're presenting in this gist. I hope you find many things you enjoy in the links! Even if they aren't directly useful as pedagogical examples, they do demonstrate, for example, that a plain recursive-descent parser is already "two-pass" enough to handle assignment to variables, contra the statement in your gist, "In a more complex language (e.g., adding support for assignment to variables), we would need an additional front-end pass after the parser." Indeed, StoneKnifeForth is a single-pass compiler in less code than your tiny compiler that supports not only assignment to variables, but also arrays, I/O, conditionals, loops, defining and calling subroutines with arguments, pointers, ELF output, and i386 machine code generation with peephole optimization. And it's not because it's written in a more powerful language than OCaml; it's written in the profoundly substandard dialect of Forth that it implements, which lacks Forth's compile-time metaprogramming facilities and in which only one character is significant in identifiers. I was wondering if maybe my memory of StoneKnifeForth was wrong and it wasn't actually 132 lines of code, because that sure sounds like a lot of functionality in 132 lines of code. It turns out it's 117 lines of code, not 132. (The build process builds a version with comments stripped out, called trimmed.tbf1, and verifies that it compiles identically, to find this out.) So maybe you can learn some things from them that will enable you to write better compilers!