13 ms·

Rich Programmer Food (2007)

92 points by rspivak 10y ago

PaulHoule 10y ago
I've come to the same conclusion, almost, except the way I see it is that everybody who takes CS takes compiler construction, other people read the literature about it, and it is still a technology that largely sits in the box. Lately there has been a lot of work towards languages that improve the state of systems programming (Rust) but the deeper problems of better languages for applications programming are largely ignored. My belief is that the content of those courses and most of the literature on compiler construction, as well as the tooling most generally available, is not at all suited towards getting people to apply it in new ways.
- qwertyuiop924 10y ago
  Yeah, and most of the tooling that isn't written in C (Lex, YACC, and MPC come to mind) has documentation that assumes you already understand the API. I mean, what?
  - steveklabnik 10y ago
    This is an extremely common problem with documentation, it's not limited to these kinds of tools. The fundamental problem is: you understand how something works, and you are trying to communicate with someone who doesn't. In a conversation, your partner can ask clarifying questions when you accidentally say something that's not clear, but in text, you're separated by time and space. So this can't happen.
    - PaulHoule 10y ago
      It can happen. Ever hear of a wiki, or an issue tracking system? Literate programming systems have failed so far because a "code-document" bundle is not the way. The way is to attach whatever documentation exists to a scaffolding that includes the code. For instance, stackoverflow contains a large amount of documentation which is poorly organized and is only findable (sorta kinda) because Google is (still) a good search engine.
      - steveklabnik 10y ago
        > Ever hear of a wiki, or an issue tracking system? ... yes. I didn't say that it was impossible, just that it's a very common issue, more broad than just compiler tooling.
        qwertyuiop924 10y ago
        Well, yeah. but compiler tooling has it far worse than most things. And I was reading through the docs to some compiler tooling written in scheme. Shudders. It has a Lex/YACC like grammar DSL, but it doesn't explain the syntax, just provides a fairly bad example. And this was (chibi parse), which was written by Alex Shinn, who usually does pretty GOOD documentation (irregex in particular is excellent, assuming you already know regex).
  - PaulHoule 10y ago
    On top of that there are a lot of other practical problems with those tools. It's hard to learn to use them to do simple things, and even harder to use them to do moderately hard things. For instance in a lot of cases what you really want is to convert a grammar to some reasonable abstract syntax tree and I've found often you have to write a lot of code into those frameworks to do that, whereas you ought to be able to write something declarative that generates the AST objects and the translation code. Another problem is a lack of composability. Particularly in DSL creation it would be desirable to "cut and paste" grammars into each other; it's very possible to embed SQL into a language like C, Java, or such -- you ought to be able to embed one parser in another but generally it is not easy. Also there is the problem of error message generation, which is bad enough when you do are doing things that are "easy with yacc" but when you are doing hardcore transpilation of some kind, the error messages are often completely incomprehensible. This is a big factor in any kind of "rules engine" built around an existing language; error handling is disastrously bad in Drools, for instance -- you might think homoiconicity would improve matters with Clara in Clojure but it really doesn't.
    - sklogic 10y ago
      If you want this kind of composability, use PEG-based tools. Same for the error messages and error recovery.
      - qwertyuiop924 10y ago
        Wait a sec. Since MPC is a Parser Combinator library, wouldn't it have the composability?
        sklogic 10y ago
        Yes, you're right - did not even notice MPC in the list, was commenting on Yacc and alike.
        qwertyuiop924 10y ago
        Yeah. I wasn't commenting on specifically YACC and Lex, just compiler toolkits with good documentations. And MPC has some of the best documentation out there. Also, Build Your Own Lisp is amazing.
- beat 10y ago
  Taking a compiler theory class did wonders for me as a programmer, even though I am in no way capable of actually writing a compiler without a lot of learning curve. The real falling-down place for most programmers, I think, is that they have no clue how parsing works. The widespread prevalence of XML/JSON, with convenient libraries, has made data stream construction/parsing a thing of the past (which is good, because most of the data stream protocols I saw were awful, filled with shift/reduce and reduce/reduce ambiguities, basically just giant hand-rolled case statements).
- sklogic 10y ago
  Correct. I am tirelessly ranting that the major culprit here is the infamous "Dragon Book". It somehow managed to turn an easy and fun subject into something intimidating and thoroughly boring.
- deleted 10y ago
  [deleted]
why-el 10y ago
I got excited for a second because I thought Steve is back. :)
weinzierl 10y ago
> If you don't know how compilers work, then you don't know how computers work. This works both ways: If you don't know how your hardware works you will never understand the compiler.
- matthewmacleod 10y ago
  And if you don't understand how physics works, you will never understand how your hardware works… ad infinitum. It's a weak argument for me, if I'm honest. Learning how a compiler works is interesting and provides useful insight into how other parts of the stack work. But it seems like a falsehood to suggest that it's not possible to understand one level of abstraction without knowing the full stack; isn't that why we have abstractions in the first place?
  - khedoros 10y ago
    > But it seems like a falsehood to suggest that it's not possible to understand one level of abstraction without knowing the full stack; isn't that why we have abstractions in the first place? Any non-trivial abstraction will have some leaks. You can memorize the problem cases and usefully use the abstraction without understanding the source of the leaks. Partial understanding can be sufficient. Fully understanding an abstraction probably requires you to understand all the levels of the stack. Is it actually useful to know the stack top-to-bottom like that? Not always. As you note, that's why we cover up the details in the first place, so that we don't always have to keep the underlying implementation in mind.
- mywittyname 10y ago
  I believe you're missing the point of the article with this statement. Compilers contain basic patterns that are repeated everywhere in programming. So, to be ignorant of how compilers work is to be pretty ignorant about programming in general. What I think Yegge was getting at is that having a comprehensive understanding of how compilers work will build a foundation for being a good programmer in general. The problems we solve everyday are fundamentally solved the smarter people that built the compilers we all use. His example was a problem involving text processing is more elegantly solved by someone with a knowledge of lexical parsing and ASTs than by someone who blindly throws regex at it. Compilers are about effectively translating input into the desired program output. The "hardware" is just one type of structured output, but there are many instances where a programmer would output a structure that is entirely unrelated to a processor or machine code.
JaRail 10y ago
I haven't laughed this hard in weeks. This is a top-shelf article!
paloaltokid 10y ago
I'm a big Steve Yegge fan and I like most of his articles. I like this one too. But, some of them, like this one, leave me feeling a bit disheartened. Okay, compilers are important, I get it. I would love to learn more about how they work. But - I never got a CS degree and I'm 10 years into my career. I've learned a lot of valuable stuff but I'm not going to go back to school just to make Steve happy. So what do I do? What's the best resource out there for me to learn & grok something about compilers without working through 3 textbooks and watching a bunch of lectures? I like being able to learn something by applying it to my current work. That's how I learned about IP networking - I did sign up for a class, but I was a Jr. DevOps person at the time and so everything I learned could be immediately applied at work the next day or next week.
- danpalmer 10y ago
  I recommend "Understanding Computation" from O'Reilly - it's pretty straightforward, teaches through examples and exercises, and uses Ruby, to build toy compilers from first principles. It's not going to give you the mathematical basis that you'd get in a uni course, (it was actually 3 separate classes over 2 years at my uni), but it should help develop the practical _feel_ for when these things might help solve real engineering problems, which I think is what matters.
  - paloaltokid 10y ago
    Thanks for the recommendation!
- rspivak 10y ago
  Sorry for the shameless plug, but about a year ago I started a series of articles on how to write a simple interpreter called "Let's Build A Simple Interpreter." Take a look, it might be something that you may like: Part 1: https://ruslanspivak.com/lsbasi-part1/ https://ruslanspivak.com/lsbasi-part1/ ... Part 9: https://ruslanspivak.com/lsbasi-part9/ https://ruslanspivak.com/lsbasi-part9/
  - sklogic 10y ago
falsedan 10y ago
Executive summary: If you don't know how compilers work, then you don't know how computers work. If you're not 100% sure whether you know how compilers work, then you don't know how they work. Why does he say that? I don't have the time to read 5495 myself to find out.
- laxatives 10y ago
  He doesn't really elaborate on that point, except that 95% of developers, including PhD's don't really understand compilers (even though his summary isn't that far off from the explanations he hears). This is going to be unpopular, but I don't think you're missing much. IMO its a long, drunken rant fueled by a massive ego.
  - dionidium 10y ago
    IMO its a long, drunken rant fueled by a massive ego. I get the opposite feeling from Steve. I think he's a guy who, like most of us, realized that he didn't know something that it's pretty important to know and, like most of us, that realization made him feel pretty bad about himself. Unlike most of us, he recognized that the way to make that bad feeling go away is to get out there and learn the damn thing you feel bad for not knowing. If that's ego, then we're all fucked.
- jaggederest 10y ago
  To TL;DR the post, any time you transform a stream of symbols into another stream of symbols, it's compilation, and we do that all the time in computing, so if you don't understand how that process works, then you don't understand the fundamental systems of computing. Unix pipeline commands, for example. If you've ever used sed, it's apparently a compiler by his definition.
- xigency 10y ago
  Well, it's not actually true. Imagine a world where all software were written in assembly language, or even better, machine code. There would be no use for compilers. 'Portable code' would be machine code that was frantically re-written (by hand) from one platform to another platform. And we wouldn't have a number of JIT technologies and other languages (by definition) as well. Here, without compilers, there would be many people who understand computers. The fact is in the world we live in, code is on a higher plane than hardware instructions, and it's something that impacts software engineers' lives proportionally. Practical computers work with machine instructions, but idealized computers operate on programs which can be abstracted as code in another language. The practical constraint is that program code must be transformed to be used, so the compiler becomes an essential tool. The fact that compilers as we know them are so complex leads to the assertion that "if you're not 100% sure you know how they work, then you don't know how they work." Because face it, if you're working on a monolithic, complex piece of software that automates a large volume of menial work, and there is no certainty that it performs as expected or does the right task, then it won't function at all. You don't need to know how computers work (in this semantic argument) to be a computer programmer, but you absolutely must know how computers work to be a compiler writer. And like the transition from machine instructions to programs written in high level languages, the journey from programmer to compiler writer is a quantum leap, but one that gives someone advantages as a programmer.
gh-lfneu28 10y ago
This post is from 2007. Yegge is not back :( OP or mods, add (2007) to the title svp.
krallja 10y ago
(2007)
sklogic 10y ago
Compilers matter because of DSLs. The rest is of a marginal importance.
fish2000 10y ago
Wow this takes me back – 2007, those were the days. Back before so many flagship projects had frankensteined libLLVM into their innards; before every single JavaScript framework was required by law to include at least one CLT whose executable name ended in 'c'; back when GCC on Apple was just a matter of /usr/bin, and not a parade of endless Homebrew formulae PRs… srsly dogg this is an excellent article, a product of its time but a bonafide bulwark, presciently on the precipice of paradigm. I +1'ed.
phamilton 10y ago
I highly recommend Make-a-lisp for anyone interested in a superficial understanding of writing a compiler. It's a very guided tour, but touches on a lot of very interesting parts of parsing, code generation, and optimization. https://github.com/kanaka/mal/blob/master/process/guide.md https://github.com/kanaka/mal/blob/master/process/guide.md

92 points by rspivak 10y ago

I've come to the same conclusion, almost, except the way I see it is that everybody who takes CS takes compiler construction, other people read the literature about it, and it is still a technology that largely sits in the box. Lately there has been a lot of work towards languages that improve the state of systems programming (Rust) but the deeper problems of better languages for applications programming are largely ignored. My belief is that the content of those courses and most of the literature on compiler construction, as well as the tooling most generally available, is not at all suited towards getting people to apply it in new ways.

qwertyuiop924 10y ago
Yeah, and most of the tooling that isn't written in C (Lex, YACC, and MPC come to mind) has documentation that assumes you already understand the API. I mean, what?
- steveklabnik 10y ago
  This is an extremely common problem with documentation, it's not limited to these kinds of tools. The fundamental problem is: you understand how something works, and you are trying to communicate with someone who doesn't. In a conversation, your partner can ask clarifying questions when you accidentally say something that's not clear, but in text, you're separated by time and space. So this can't happen.
  - PaulHoule 10y ago
    It can happen. Ever hear of a wiki, or an issue tracking system? Literate programming systems have failed so far because a "code-document" bundle is not the way. The way is to attach whatever documentation exists to a scaffolding that includes the code. For instance, stackoverflow contains a large amount of documentation which is poorly organized and is only findable (sorta kinda) because Google is (still) a good search engine.
    - steveklabnik 10y ago
      > Ever hear of a wiki, or an issue tracking system? ... yes. I didn't say that it was impossible, just that it's a very common issue, more broad than just compiler tooling.
      - qwertyuiop924 10y ago
        Well, yeah. but compiler tooling has it far worse than most things. And I was reading through the docs to some compiler tooling written in scheme. Shudders. It has a Lex/YACC like grammar DSL, but it doesn't explain the syntax, just provides a fairly bad example. And this was (chibi parse), which was written by Alex Shinn, who usually does pretty GOOD documentation (irregex in particular is excellent, assuming you already know regex).
- PaulHoule 10y ago
  On top of that there are a lot of other practical problems with those tools. It's hard to learn to use them to do simple things, and even harder to use them to do moderately hard things. For instance in a lot of cases what you really want is to convert a grammar to some reasonable abstract syntax tree and I've found often you have to write a lot of code into those frameworks to do that, whereas you ought to be able to write something declarative that generates the AST objects and the translation code. Another problem is a lack of composability. Particularly in DSL creation it would be desirable to "cut and paste" grammars into each other; it's very possible to embed SQL into a language like C, Java, or such -- you ought to be able to embed one parser in another but generally it is not easy. Also there is the problem of error message generation, which is bad enough when you do are doing things that are "easy with yacc" but when you are doing hardcore transpilation of some kind, the error messages are often completely incomprehensible. This is a big factor in any kind of "rules engine" built around an existing language; error handling is disastrously bad in Drools, for instance -- you might think homoiconicity would improve matters with Clara in Clojure but it really doesn't.
  - sklogic 10y ago
    If you want this kind of composability, use PEG-based tools. Same for the error messages and error recovery.
    - qwertyuiop924 10y ago
      Wait a sec. Since MPC is a Parser Combinator library, wouldn't it have the composability?
      - sklogic 10y ago
        Yes, you're right - did not even notice MPC in the list, was commenting on Yacc and alike.
        qwertyuiop924 10y ago
        Yeah. I wasn't commenting on specifically YACC and Lex, just compiler toolkits with good documentations. And MPC has some of the best documentation out there. Also, Build Your Own Lisp is amazing.
beat 10y ago
Taking a compiler theory class did wonders for me as a programmer, even though I am in no way capable of actually writing a compiler without a lot of learning curve. The real falling-down place for most programmers, I think, is that they have no clue how parsing works. The widespread prevalence of XML/JSON, with convenient libraries, has made data stream construction/parsing a thing of the past (which is good, because most of the data stream protocols I saw were awful, filled with shift/reduce and reduce/reduce ambiguities, basically just giant hand-rolled case statements).
sklogic 10y ago
Correct. I am tirelessly ranting that the major culprit here is the infamous "Dragon Book". It somehow managed to turn an easy and fun subject into something intimidating and thoroughly boring.
deleted 10y ago
[deleted]

why-el 10y ago

I got excited for a second because I thought Steve is back. :)

weinzierl 10y ago

> If you don't know how compilers work, then you don't know how computers work. This works both ways: If you don't know how your hardware works you will never understand the compiler.

matthewmacleod 10y ago
And if you don't understand how physics works, you will never understand how your hardware works… ad infinitum. It's a weak argument for me, if I'm honest. Learning how a compiler works is interesting and provides useful insight into how other parts of the stack work. But it seems like a falsehood to suggest that it's not possible to understand one level of abstraction without knowing the full stack; isn't that why we have abstractions in the first place?
- khedoros 10y ago
  > But it seems like a falsehood to suggest that it's not possible to understand one level of abstraction without knowing the full stack; isn't that why we have abstractions in the first place? Any non-trivial abstraction will have some leaks. You can memorize the problem cases and usefully use the abstraction without understanding the source of the leaks. Partial understanding can be sufficient. Fully understanding an abstraction probably requires you to understand all the levels of the stack. Is it actually useful to know the stack top-to-bottom like that? Not always. As you note, that's why we cover up the details in the first place, so that we don't always have to keep the underlying implementation in mind.
mywittyname 10y ago
I believe you're missing the point of the article with this statement. Compilers contain basic patterns that are repeated everywhere in programming. So, to be ignorant of how compilers work is to be pretty ignorant about programming in general. What I think Yegge was getting at is that having a comprehensive understanding of how compilers work will build a foundation for being a good programmer in general. The problems we solve everyday are fundamentally solved the smarter people that built the compilers we all use. His example was a problem involving text processing is more elegantly solved by someone with a knowledge of lexical parsing and ASTs than by someone who blindly throws regex at it. Compilers are about effectively translating input into the desired program output. The "hardware" is just one type of structured output, but there are many instances where a programmer would output a structure that is entirely unrelated to a processor or machine code.

JaRail 10y ago

I haven't laughed this hard in weeks. This is a top-shelf article!

paloaltokid 10y ago

I'm a big Steve Yegge fan and I like most of his articles. I like this one too. But, some of them, like this one, leave me feeling a bit disheartened. Okay, compilers are important, I get it. I would love to learn more about how they work. But - I never got a CS degree and I'm 10 years into my career. I've learned a lot of valuable stuff but I'm not going to go back to school just to make Steve happy. So what do I do? What's the best resource out there for me to learn & grok something about compilers without working through 3 textbooks and watching a bunch of lectures? I like being able to learn something by applying it to my current work. That's how I learned about IP networking - I did sign up for a class, but I was a Jr. DevOps person at the time and so everything I learned could be immediately applied at work the next day or next week.

danpalmer 10y ago
I recommend "Understanding Computation" from O'Reilly - it's pretty straightforward, teaches through examples and exercises, and uses Ruby, to build toy compilers from first principles. It's not going to give you the mathematical basis that you'd get in a uni course, (it was actually 3 separate classes over 2 years at my uni), but it should help develop the practical _feel_ for when these things might help solve real engineering problems, which I think is what matters.
- paloaltokid 10y ago
  Thanks for the recommendation!
rspivak 10y ago
Sorry for the shameless plug, but about a year ago I started a series of articles on how to write a simple interpreter called "Let's Build A Simple Interpreter." Take a look, it might be something that you may like: Part 1: https://ruslanspivak.com/lsbasi-part1/ https://ruslanspivak.com/lsbasi-part1/ ... Part 9: https://ruslanspivak.com/lsbasi-part9/ https://ruslanspivak.com/lsbasi-part9/
- sklogic 10y ago

falsedan 10y ago

Executive summary: If you don't know how compilers work, then you don't know how computers work. If you're not 100% sure whether you know how compilers work, then you don't know how they work. Why does he say that? I don't have the time to read 5495 myself to find out.

laxatives 10y ago
He doesn't really elaborate on that point, except that 95% of developers, including PhD's don't really understand compilers (even though his summary isn't that far off from the explanations he hears). This is going to be unpopular, but I don't think you're missing much. IMO its a long, drunken rant fueled by a massive ego.
- dionidium 10y ago
  IMO its a long, drunken rant fueled by a massive ego. I get the opposite feeling from Steve. I think he's a guy who, like most of us, realized that he didn't know something that it's pretty important to know and, like most of us, that realization made him feel pretty bad about himself. Unlike most of us, he recognized that the way to make that bad feeling go away is to get out there and learn the damn thing you feel bad for not knowing. If that's ego, then we're all fucked.
jaggederest 10y ago
To TL;DR the post, any time you transform a stream of symbols into another stream of symbols, it's compilation, and we do that all the time in computing, so if you don't understand how that process works, then you don't understand the fundamental systems of computing. Unix pipeline commands, for example. If you've ever used sed, it's apparently a compiler by his definition.
xigency 10y ago
Well, it's not actually true. Imagine a world where all software were written in assembly language, or even better, machine code. There would be no use for compilers. 'Portable code' would be machine code that was frantically re-written (by hand) from one platform to another platform. And we wouldn't have a number of JIT technologies and other languages (by definition) as well. Here, without compilers, there would be many people who understand computers. The fact is in the world we live in, code is on a higher plane than hardware instructions, and it's something that impacts software engineers' lives proportionally. Practical computers work with machine instructions, but idealized computers operate on programs which can be abstracted as code in another language. The practical constraint is that program code must be transformed to be used, so the compiler becomes an essential tool. The fact that compilers as we know them are so complex leads to the assertion that "if you're not 100% sure you know how they work, then you don't know how they work." Because face it, if you're working on a monolithic, complex piece of software that automates a large volume of menial work, and there is no certainty that it performs as expected or does the right task, then it won't function at all. You don't need to know how computers work (in this semantic argument) to be a computer programmer, but you absolutely must know how computers work to be a compiler writer. And like the transition from machine instructions to programs written in high level languages, the journey from programmer to compiler writer is a quantum leap, but one that gives someone advantages as a programmer.

gh-lfneu28 10y ago

This post is from 2007. Yegge is not back :( OP or mods, add (2007) to the title svp.

krallja 10y ago

(2007)

sklogic 10y ago

Compilers matter because of DSLs. The rest is of a marginal importance.

fish2000 10y ago

Wow this takes me back – 2007, those were the days. Back before so many flagship projects had frankensteined libLLVM into their innards; before every single JavaScript framework was required by law to include at least one CLT whose executable name ended in 'c'; back when GCC on Apple was just a matter of /usr/bin, and not a parade of endless Homebrew formulae PRs… srsly dogg this is an excellent article, a product of its time but a bonafide bulwark, presciently on the precipice of paradigm. I +1'ed.

phamilton 10y ago

I highly recommend Make-a-lisp for anyone interested in a superficial understanding of writing a compiler. It's a very guided tour, but touches on a lot of very interesting parts of parsing, code generation, and optimization. https://github.com/kanaka/mal/blob/master/process/guide.md https://github.com/kanaka/mal/blob/master/process/guide.md