11 ms·
The Friendship That Made Google Huge
- jdpedrie 8y agoSite's down for me. Archive link: http://archive.is/w4oWL http://archive.is/w4oWL
- bigtones 8y agoThis is a wonderfully told story of two very senior engineers who developed a lifelong friendship and wrote some of the core of the Google search engine I use countless times per day - I really enjoyed this piece.
- VikingCoder 8y ago"Jeff Dean once shifted a bit so hard it ended up on another computer."
- shafte 8y agoI'm so glad this this article (unlike so many other representations of Jeff and Sanjay's work) does not overlook Sanjay's contributions. I don't know what it is; maybe Sanjay is a weird (read: non-English) name, maybe it's that computing culture's obsession with lone hackers leaves no room for a partnership like Jeff and Sanjay's. Anyway, kudos to the New Yorker for not falling into that trap.
- Itaxpica 8y agoI think a big part of it is something the article touches on: Jeff is an extrovert, Sanjay is an introvert. Jeff is the much more visible of the two; the one who people are more likely to have heard speak personally or to have dealt with. It sounds like this is how they both prefer it, but it definitely means that Sanjay's contributions get passed over in the common discourse. I was also glad to see the article avoid that trap.
- ugh123 8y agoYou mean like Steve and Woz? j/k :)
- nlh 8y agoDitto on the sentiment. I’m not a Googler but I’ve worked closely with a few, and Jeff was revered accordingly (along with the Chuck Norris-esque jokes). I’d honestly never heard Sanjay mentioned (and didn’t know who he was) until I read this article. I’m glad he was given his due and I’m glad to have learned about his contributions.
- AceJohnny2 8y ago> (along with the Chuck Norris-esque jokes) Funny thing, that was started by kentonv [0], who worked on open-sourcing protobufs [1], then CapnProto [2], then (with Jade Wang) Sandstorm [3]. He lurks around here :) [0] https://plus.google.com/+KentonVarda/posts/TSDhe5CvaFe https://plus.google.com/+KentonVarda/posts/TSDhe5CvaFe [1] https://github.com/protocolbuffers/protobuf https://github.com/protocolbuffers/protobuf [2] https://capnproto.org https://capnproto.org [3] https://sandstorm.io https://sandstorm.io
- kentonv 8y agoHello... Indeed, I wrote the internal web site (built on a pre-release prototype of AppEngine!) that aggregated these jokes, and released it on April 1, 2007. I was not funny enough to come up with the jokes myself; I enlisted some friends to seed the site before "launch". In retrospect, I regret that Jeff Dean Facts gave people the impression that Jeff is more notable than Sanjay. The only reason it was "Jeff Dean Facts" and not "Sanjay Ghemawat Facts" is because Jeff seemed "funnier". But I dunno, that might just be some form of subtle racism that I failed to recognize at the time. :( PS. Your list doesn't include my current project, Cloudflare Workers... :) https://blog.cloudflare.com/cloud-computing-without-containers/ https://blog.cloudflare.com/cloud-computing-without-containe...
- Jyaif 8y ago“In terms of sexiness, compilers are pretty much as boring as it gets,” That bit surprised me. For me it doesn't get much sexier than compilers.
- ineedasername 8y agoYeah. Maybe that's the outside view, or view from the surface level folks concerned with the next shiny framework, but I think the best engineers are always at least a little enamored with deeper nuts and bolts.
- AlexCoventry 8y agoI think he meant the business side isn't going to give a damn about a compiler.
- ineedasername 8y agoThat makes sense. At least, outside of massive companies like Google, who would see the value in a compiler that optimized code for their use even a fraction of a percent better. Then compilers become the "shiny object" for them.
- amelius 8y agoWriting a modern concurrent garbage collector is as complicated as writing a compiler, to a great degree because of their "real time" constraints and concurrency. But to call them sexy is another thing ...
- strikelaserclaw 8y agothese are the kind of articles i like reading, well written, interesting subject and good content length. Too many times i read bold headlines on HN, "Why X is bringing america down", then go to read it and just when they get to the crux of the matter i find the article has ended smh.
- woodhull 8y agoThe New Yorker does this very consistently every week at an extraordinary level of quality. They will even mail you a copy in exchange for a subscription. It's magical.
- strikelaserclaw 8y agoi will look into it
- forgot-my-pw 8y agoOnly $6 for 12 weeks for print + digital. Subscribed!
- fspacef 8y agoGood stuff. Every sport has great partnerships (Jordon - Pippen, Brady - Gronk) and I always wondered if such a thing would even be possible in SE as I've felt overly self conscious and inefficient while pair programming. Though what really makes this my favorite long read of the year is how it follows their story over the course of Google's rise and the subtle juxtapositions between early Google of "doors across sawhorses" and today with "Level 0-11". Given all the recent news with engineer walkouts and protests it makes you realize that Google have probably done the best job of handling the scaling challenges of being a unicorn and making the necessary cultural adjustments as they grew in size.
- warent 8y agoDon't forget Sam and Frodo
- ineedasername 8y agoYeah, teams are good but every once in a while you need that lone crazy Tom Bombadil genius to come along and save your butt from disaster.
- kadal 8y agoAnd sometimes you need eagles
- ineedasername 8y agoThe really bad times are when your entire universe screaches to a halt until Brandon Sanderson helps it limp across the finish line.
- gowld 8y agohttp://cycle-gap.blogspot.com/2007/09/extreme-pair-programming-guy-steele-and.html http://cycle-gap.blogspot.com/2007/09/extreme-pair-programmi... Extreme Pair Programming - Guy Steele and Richard Stallman
- 8y ago
- NelsonMinar 8y agoWhat a lovely profile. I worked at Google early enough that I got to know Jeff and Sanjay, although never worked directly with them (to my regret). Every aspect of this article rings true to me, the way they are, the way they work together. I always admired how incredibly collaborative they were. They never competed with each other. Or really with anyone else, either, they just quietly did excellent work and helped others do their work too.
- glenrivard 8y agoGreat article. Glad to see Sanjay get the credit he deserves.
- deleted 8y ago[deleted]
- skybrian 8y agoI wonder if more couldn't be done to encourage pair programming and collaboration in general? What are some other great partnerships?
- Daktest 8y agoWould highly recommend checking out `The Undoing Project` by Michael Lewis. It's a biography (of sorts) about Amos Tversky and Daniel Kahneman that has excellent stories about their fruitful collaboration in behavioral economics.
- enneff 8y agoIf you want to see an example of Sanjay's elegant code, take a look at his Go stream package: https://github.com/ghemawat/stream https://github.com/ghemawat/stream
- stdplaceholder 8y agoSanjay's "experimental" directory in Google's repo -- a place where users can commit unreviewed code that are just ideas, or park some code that isn't ripe yet -- is full of gems.
- puzzle 8y agoHoly moly, a mention of Bogdan! He was a legend of his own. Some of the most consequential changes, like elevating your service to higher QoS levels, which something like e.g. Gmail would require, were gated on his approval and his only. Someone made him a SW Easter egg, but the story already on the Internet gets some basic facts wrong.
- lowercased 8y ago“I don’t know why more people don’t do it,” Sanjay said, of programming with a partner. ======= Maybe because most employers won't pay for that. Two people? For one level of output? You'd have to prove yourself a level 11 googler before most places would give you the luxury of working as a long-term pair.
- skybrian 8y agoI don't think that's it. It's typical to have a team of programmers working on a large project, and many managers are willing to try things. However, pair programming does require a lot of commitment to make it work. I've only seen it work on a team where it was an XP project to start with, so everyone hired was already willing to give it a solid try. Your average team isn't self-selected in this way and won't have the same level of enthusiasm for a new management initiative.
- NelsonMinar 8y agoI think from the very beginning it was understood they were producing more than one level of output together.
- jogjayr 8y agoPivotal Labs has had across-the-board pair programming for years[1]. Although the folks there are undoubtedly talented, I'm fairly certain most aren't at Jeff and Sanjay's level. I don't know why companies don't do it, but it's not cost alone. Here are some of my hypotheses: 1. It seems like it would be more expensive - which might deter companies from thinking about it 2. If you're not already doing it, it's hard to convince all or even a large fraction of your engineers to do it. It's a completely alien way of working to most people 3. It's not obvious to employees who haven't done it that it's clearly better. And it might not be - I honestly don't know 4. For 100% pairing, you need 2 programmers' schedules to line up exactly, 5 days/week - so goodbye flextime, work from home, remote working and all the other arrangements that people structure their lives around 5. It must be exhausting to sit with, and talk to, another person constantly your entire working day 1. https://pivotal.io/careers https://pivotal.io/careers
- 8y ago
- tareqak 8y agoAfter reading this article, I'm surprised that Google doesn't encourage pair programming / developing developer relationships at the level that Jeff and Sanjay have here. One might even argue that the manner in which interviewing and hiring is currently done in the tech industry in general might actively work against building the sorts of relationships where the output is greater than the sum of the inputs. Edit: added a missing "the".
- why_only_15 8y agoTheir specific choices seem hard to replicate. Very strong friends, both very strong programmers, both willing to work at the same company. I'm sure that parts of their successes can be applied elsewhere, but from the article a lot of it is about having a strong relationship.
- sdenton4 8y agoSee also Andrew and Brad: https://m.youtube.com/watch?v=yG-UaBJXZ80 https://m.youtube.com/watch?v=yG-UaBJXZ80
- skybrian 8y agoIn a way, Google does encourage it by requiring all code to be reviewed before check in. If you're pair programming, you already have a reviewer and can check in right away. One-person projects can be more frustrating due to the delay. It doesn't seem to be enough to encourage widespread pair programming, though.
- jogjayr 8y agoEvery company I've worked at as a software engineer, even my first where we emailed patches to each other like cavepeople (and no we were not using git) has had code reviews before check in. It's not at all the same thing as pair programming though.
- Pica_soO 8y agoThey do not discourage is, but there promotion system encourages "authorship" and pair-programming does not produce that.
- stirner 8y ago> Most of the fragile insights that laid the foundation of a new vision emerged not when the whole group was together, and not when members worked alone, but when they collaborated and responded to one another in pairs The article gives several examples of this dynamic, but one that came to mind for me was how Monty Python members created sketches for Flying Circus: > Writing started at 9 am and finished at 5 pm. Typically, Cleese and Chapman worked as one pair isolated from the others, as did Jones and Palin, while Idle wrote alone. After a few days, they would join together with Gilliam, critique their scripts, and exchange ideas. Their approach to writing was democratic. If the majority found an idea humorous, it was included in the show. https://en.m.wikipedia.org/wiki/Monty_Python https://en.m.wikipedia.org/wiki/Monty_Python
- jamestimmins 8y agoAnyone know where folks like Peter Norvig and Urs Hölzle fit on the L1-L11 scale? I assumed that all of the early employees/now legends were at the top level, so I was surprised to see the article state otherwise.
- stdplaceholder 8y agoUrs doesn't need a level on the individual contributor scale; he's a Senior Vice President.
- joshuamorton 8y agoUrs however does have one, he's listed as a Google fellow (L10) if you go looking. As a general comment, to compare, a director is L8 equivalent, and VP L9, (but on the management ladder) although at VP+ it gets tricky because there are subdivisions. Also worth noting that the management ladder goes up more. But the total number of people who are >L11 is probably <11, so it's not like there's any data on how exactly that's broken up.
- Shebanator 8y agoPeter is a director, and Urs is a senior VP. As such, they are now both on the management ladder. I think the article was discussing only people who were on the eng ladder.
- pk2200 8y agoI can't remember what Senior VP equates to on the tech ladder, but in terms of his overall impact on the company, Urs is definitely an 11.
- warunsl 8y agoWas surprised to hear the audio version of the story has a different title - "Binary stars". I think the editorial board went with a more click friendly title.
- panopticon 8y agoThat was also the title of the print version. The New Yorker tends to re-title online "reprints" with less vague/terse/artsy titles.
- timdellinger 8y agoDoes anyone have a reference for "In 1966, researchers at the System Development Corporation discovered that the best programmers were more than ten times as effective as the worst." ? I'm seeing the year 1968 at http://thecomputerboys.com/?tag=black-art#fn-156-1 http://thecomputerboys.com/?tag=black-art#fn-156-1 Is there a 1966 reference out there somewhere? Or perhaps the actual study was done in '66 and didn't make it into press until '68? I didn't pull the original paper. (The fact checkers at the New Yorker are typically very sharp.) edit: I pulled the original paper (10.1145/362851.362858). The studies were done in 1966! Interesting to see the genesis of the myth of the 10x programmer.
- azhenley 8y agoIt is awesome that you were able to find this. This very well may be the genesis! For anyone else, that DOI points to: https://dl.acm.org/citation.cfm?doid=362851.362858 https://dl.acm.org/citation.cfm?doid=362851.362858 "Exploratory experimental studies comparing online and offline programming performance" from CACM 1968.
- akhilcacharya 8y agoI'm not so sure it's a "Myth" when Jeff and Sanjay are out and about. Honestly, I'm beginning to think it's inherent and can't be taught.
- thwy12321 8y ago"Interesting to see the genesis of the myth of the 10x programmer." How could it be a myth? Ever see a professional sports player compete against a non professional player? Why wouldnt this dynamic exist in programming?
- superhuzza 8y agoMaybe I'm unimaginative, but in sports where performance is objectively comparable, I can't think of any where professionals 10x amateurs. Amateur marathons are about 4 hours, professional are about 2 hours. Amateur deadlifts are about 500lb, professional are about 1000lb. Amateur long jump might be 6m, professional almost 9m.
- bertmuthalaly 8y agoI was so delighted by this piece and then I got to the byline - James Somers strikes again! You might know his writing from “You’re Probably Using the Wrong Dictionary” or his amazing article about New York’s subway system told through the lens of the countdown clock: http://www.theatlantic.com/technology/archive/2015/11/why-dont-we-know-where-all-the-trains-are/415152/ http://www.theatlantic.com/technology/archive/2015/11/why-do... Highly recommend all his work: http://jsomers.net/ http://jsomers.net/
- misiti3780 8y agoseems to be a awesome engineer too.
- sjg007 8y agoThis article will queue up a slew of startups that pair program.. perhaps a good thing!
- partycoder 8y agoIt is refreshing to see an article celebrating collaboration.
- sytelus 8y agoInteresting tidbits: To solve problems at scale, paradoxically, you have to know the smallest details -Alan Eustace Today, Google’s engineers exist in a Great Chain of Being that begins at Level 1. At the bottom are the I.T. support staff. Level 2s are fresh out of college; Level 3s often have master’s degrees. Getting to Level 4 takes several years, or a Ph.D. Most progression stops at Level 5. Level 6 engineers—the top ten per cent—are so capable that they could be said to be the reason a project succeeds; Level 7s are Level 6s with a long track record. Principal Engineers, the Level 8s, are associated with a major product or piece of infrastructure. Distinguished Engineers, the Level 9s, are spoken of with reverence. To become a Google Fellow, a Level 10, is to win an honor that will follow you for life. Google Fellows are usually the world’s leading experts in their fields. Jeff and Sanjay are Google Senior Fellows—the company’s first and only Level 11s.
- akhilcacharya 8y ago> Level 2s are fresh out of college; Level 3s often have master’s degrees. Getting to Level 4 takes several years, or a Ph.D. I..don't think this is still accurate. New grad SWEs are L3's.
- habosa 8y agoYeah it's mostly accurate but drop level 1. Lowest I've seen for technical employee of any kind is 2. New grad is 3. 4 is about 2 years later. 5 is the point where you have "no expectation if advancement" and is the most common level for mid-career people.
- sandGorgon 8y agoDoes pair programming only work with friends?
- wpietri 8y agoNah. You can do it with anybody. I've been part of long-lived teams that pair. But there we rotated pairing partners once or twice a day, so that over the course of the week you'd pair with everybody. I think that's much more sustainable for most. Pairing with the same person every day is sort of like sharing a snowbound cabin with them. There are very few people I get along with so well that they wouldn't grate after a few weeks.
- AlexCoventry 8y agoEnemies are the best to pair with.
- the_arun 8y agoThere was an earlier thread in HN on them here - https://news.ycombinator.com/item?id=16744353 https://news.ycombinator.com/item?id=16744353
- continuations 8y ago> The world’s most robust computer systems... used special hardware that could tolerate single bit-flips. But Google... bought cheaper computers that lacked that feature. > Together, Jeff and Sanjay wrote code to compensate for the offending machines. I'm guessing the article is referring to Google buying servers with non-ECC RAM. How do you write code to compensate for non-ECC RAM?
- mav3rick 8y agoBy pushing this complexity up the stack and also having redundancy in storage. What's not correct can be an error and result in retries from the client that may be routed to good storage.
- continuations 8y agoBut how do you even tell something's not correct? For example in the article they had to manually examine the index and find out some words were missing. How do you detect that not on the hardware level but up on the software level?
- perfmode 8y agoChecksums
- cududa 8y agoJust had a fun flashback to checksuming new builds of Windows media after manually burning/ before installing the build!
- nostrademons 8y agohttps://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_cor...
- quicklime 8y ago> When a car goes around a turn, more ground must be covered by the outside wheels; likewise, the outer edge of a spinning hard disk moves faster than the inner one. Google had moved the most frequently accessed data to the outside, so that bits could flow faster under the read-head, but had left the inner half empty; Jeff and Sanjay used the space to store preprocessed data for common search queries So Google was a company that was too cheap to buy ECC RAM, but willing to let half of their hard disks go to waste. I'm struggling to understand the logic behind this; can anyone explain?
- hn_throwaway_99 8y agoWow, this article is pretty amazing to me. I have a joke (err, kind of a joke) that I like to say, "It's never a cosmic ray." What I mean by that is that when you are debugging and getting incredibly frustrated, right when you feel that you want to give up because you can't find the bug, that you shouldn't blame "cosmic rays" because 99.999% of the time it's a logic problem. Here was a case where it actually was cosmic rays (at least, not a logic problem), and these guys found it. I also agree with the title and intro of the article - I think it's very possible that if these 2 exact people hadn't been at Google at the time than Google's path could have been very different. Makes me also appreciate how much luck is involved in any successful indeavor. Not luck that they found the bug (that was hard work), but luck that these 2 guys found themselves working at Google right when Google needed them.
- true_tuna 8y agoIt’s never a cosmic ray but it’s often bad ram. You don’t have to inspect the binary to diagnose bad ram. The system would be behaving erratically in other ways due to regular memory corruption. It’s an amazing story but by no means the only way it could have played out.
- nyc111 8y ago> On Sanjay’s monitor, a thick column of 1s and 0s appeared, each row representing an indexed word. Sanjay pointed: a digit that should have been a 0 was a 1. I was curious how he can see "a digit that should have been a 0 was a 1" in a "thick column of 1s and 0s"? Can you give a specific example. I'm not questioning that he did but I'd like to know a specific example.
- stdplaceholder 8y agoFor example the high bit in blocks of ASCII text will always be zero. In a binary dump it would appear as a column of zeros.
- nyc111 8y agoOk. So you don't need to actually "read" and understand the binary code like you are reading a programming language. You look for out of place numbers.
- techs 8y agoGreat article! Two powerful minds working on analyzing and solving on same thread of thought. Kudos to the writer, very well presented.
- hi41 8y agoIt is such a beautiful story of friendship. Even married couple don’t this level of match in mind. Only a few a luckily like that. Their partnership has resulted monumental contributions.
- pumaNpl 8y agoAwesome article
- gt2 8y agoSerious question - is this an ad? For the company, showing they have a solution to new growth problems which will keep them a growth company? (custom chip, architected for AI) And how they solved the problem before with this team?