3 ms·

Jeff Dean is well known as a superstar within Google, although not so much to the outside world. With the exception of PageRank he created or had a hand in almo

by neilk 17y ago

Jeff Dean is well known as a superstar within Google, although not so much to the outside world. With the exception of PageRank he created or had a hand in almost all the technologies you've heard of as the major Google innovations. I have a Googler friend (genius coder in his own right) who sometimes wondered if he wouldn't be more productive by just devoting his workday to ensuring that Jeff Dean was properly caffeinated. I find this kind of fascinating because Jeff Dean's academic background was in compiler optimization research. Not the obvious choice to build infrastructure for a large website. But perhaps compiler people know how every nanosecond counts, and can see the network as just another high latency part of a big computation.

strlen 17y ago
Designing and implementing a compiler involves many aspects of computer science -- as does distributed systems development. If you've never written a compiler from scratch, I highly suggest doing so(1): there are data structures (symbol table -- a hash table, parse tree, tries), algorithms (converting an NDFSM to a DFSM(2), parsing, filling registers and many others), computer organization / architecture (outputting assembly, optimizing the code with pipelining and CPU cache in mind). Not to mention it's also a substantial project which teaches you a lot about software engineering: it's not something you can write in a single session, it's something that has to be frequently extended and tested (I can't think of a cleaner candidate for test driven development: you specify the source code you want to input and test if the generated IR/assembly code is what you're looking for). It's also really fun :-) Not to mention custom compilers/languages are very useful in distributed systems: look at Thrift/Protocol Buffers ( languages for describing serialization of data structures for RPC). Google also has custom language which is compiled into state machines that they used to implement their Chubby service (and probably other distributed systems) -- see the "Paxos Live" paper. At an employer I've been at, which (when I was there) was deploying distributed systems with tens of thousands of nodes, we've created a custom language for describing/querying groups of machines: this language and systems built on top of it (for monitoring, configuration, provisioning and deployment), helped reduce admin:machine ratio by orders of magnitude (think a team of three handling ~10,000 machines). (1) I suggest picking up an older (not the newest) version of the Dragon Book and targeting a x86/amd64/MIPS: while for a practical compiler I'd almost certainly suggest targeting LLVM or JVM/CLR, you'll learn more targeting bare metal. (2) Incidentally, Lamport's landmark paper suggested treating processes in distributed systems as state machines.
- sandGorgon 17y ago
  that is a fascinating account - could you elaborate on how you leveraged a compiler/custom language for your 10K machine management. I am not able to wrap my head around what was it specifically that you needed a language for - rather than, say building it on top of perl/python/lua - and whether you used the yacc/bison toolchain to build it.
  - strlen 17y ago
    First, I didn't build the language/compiler myself (it was written in my group before I arrived). It would be embedded in other languages, much as SQL is: you execute queries (first against data on a local disk distributed via version control, later against a web service). The queries you'd make would be like: "Give me difference of (union of ( all machines running 64 bit Linux , all machine in cluster "match", all machines in wast coast datacenter )) machines starting "box10" and then show me the vlans these machines are on". Of course this would be in a much more terse (but very easy to learn) language. Perl/Python/C code (and through a CLI tool, shell scripts) would then do operations on returned values (i.e. parallel ssh to execute a command, parallel TCP call).
snprbob86 17y ago
The guy is definitely incredible: http://research.google.com/people/jeff/index.html http://research.google.com/people/jeff/index.html http://news.ycombinator.com/item?id=220031 http://news.ycombinator.com/item?id=220031
- nostrademons 17y ago
  He certainly is, but I suspect the YCombinator thread you linked to actually got the wrong person. All the facts the post cites are true, but the Steve Yegge blog he's responding to was talking about the engineering culture. While Jeff Dean's had a hand in basically every piece of engineering infrastructure at Google, another name comes to mind for someone who's had a hand in basically every piece of engineering culture.

by neilk 17y ago

strlen 17y ago

Designing and implementing a compiler involves many aspects of computer science -- as does distributed systems development. If you've never written a compiler from scratch, I highly suggest doing so(1): there are data structures (symbol table -- a hash table, parse tree, tries), algorithms (converting an NDFSM to a DFSM(2), parsing, filling registers and many others), computer organization / architecture (outputting assembly, optimizing the code with pipelining and CPU cache in mind). Not to mention it's also a substantial project which teaches you a lot about software engineering: it's not something you can write in a single session, it's something that has to be frequently extended and tested (I can't think of a cleaner candidate for test driven development: you specify the source code you want to input and test if the generated IR/assembly code is what you're looking for). It's also really fun :-) Not to mention custom compilers/languages are very useful in distributed systems: look at Thrift/Protocol Buffers ( languages for describing serialization of data structures for RPC). Google also has custom language which is compiled into state machines that they used to implement their Chubby service (and probably other distributed systems) -- see the "Paxos Live" paper. At an employer I've been at, which (when I was there) was deploying distributed systems with tens of thousands of nodes, we've created a custom language for describing/querying groups of machines: this language and systems built on top of it (for monitoring, configuration, provisioning and deployment), helped reduce admin:machine ratio by orders of magnitude (think a team of three handling ~10,000 machines). (1) I suggest picking up an older (not the newest) version of the Dragon Book and targeting a x86/amd64/MIPS: while for a practical compiler I'd almost certainly suggest targeting LLVM or JVM/CLR, you'll learn more targeting bare metal. (2) Incidentally, Lamport's landmark paper suggested treating processes in distributed systems as state machines.

sandGorgon 17y ago
that is a fascinating account - could you elaborate on how you leveraged a compiler/custom language for your 10K machine management. I am not able to wrap my head around what was it specifically that you needed a language for - rather than, say building it on top of perl/python/lua - and whether you used the yacc/bison toolchain to build it.
- strlen 17y ago
  First, I didn't build the language/compiler myself (it was written in my group before I arrived). It would be embedded in other languages, much as SQL is: you execute queries (first against data on a local disk distributed via version control, later against a web service). The queries you'd make would be like: "Give me difference of (union of ( all machines running 64 bit Linux , all machine in cluster "match", all machines in wast coast datacenter )) machines starting "box10" and then show me the vlans these machines are on". Of course this would be in a much more terse (but very easy to learn) language. Perl/Python/C code (and through a CLI tool, shell scripts) would then do operations on returned values (i.e. parallel ssh to execute a command, parallel TCP call).

snprbob86 17y ago

The guy is definitely incredible: http://research.google.com/people/jeff/index.html http://research.google.com/people/jeff/index.html http://news.ycombinator.com/item?id=220031 http://news.ycombinator.com/item?id=220031

nostrademons 17y ago
He certainly is, but I suspect the YCombinator thread you linked to actually got the wrong person. All the facts the post cites are true, but the Steve Yegge blog he's responding to was talking about the engineering culture. While Jeff Dean's had a hand in basically every piece of engineering infrastructure at Google, another name comes to mind for someone who's had a hand in basically every piece of engineering culture.