28 ms·

Software Engineering at Google

442 points by asnt 10y ago

sjakobi 10y ago
Why aren't OKR scores used as an input to performance appraisals?
- fooker 10y ago
  You can give yourself full marks.
- npelly 10y ago
  They would be far too easy to game.
- bskap 10y ago
  Because they aren't measures of how good you are at developing, testing, and releasing software, they're just measures of how good you are at estimating the amount of work it'll take to complete a project 3 months in advance.
__strisk 10y ago
Is the practice of shoving all disparate pieces of proprietary software (or individual projects) in the same repo a common occurrence? I have found that pulling unrelated changes just so that I can push my changes is an inconvenience. Furthermore, tracking the history of a particular project is confounded due to unrelated commits. I am sure that their vcs (piper?) makes this a feasible task, but for git, it seems like it would suck. The article posted by kyrra, mentions this. Given the value gained from the existing tools Google has built and the many advantages of the monolithic codebase structure, it is clear that moving to more and smaller repositories would not make sense for Google's main repository. The alternative of moving to Git or any other DVCS that would require repository splitting is not compelling for Google. It seems like they have just too much invested in this "shove it in the same repo" style. Or is this the more appropriate way to do things in a large organization?
- tehlike 10y ago
  It actually works quite nicely. Most of the google software is built internally. Ensuring everyone is running at head is a blessing (rarely a curse), because when you make a change you immediately see if it breaks something. You can also be sure everything gets all the bug fixes in their new release. This is good for various reasons, including the fact that it makes security audits significantly simpler. Some google orgs, like android, don't use the existing infra, and they kind of struggle and have to reinvent most wheels, because of that. Edit: added second paragraph
- throwawayGogEng 10y ago
  Coming from companies that use reasonably-sized git repos, I absolutely hated Google's VCS. Here's some of my painpoints with it: * No branches. If you want to make a temporary code branch, you create a CL (Google's version of a pull request), but never submit it. This means nobody else can collaborate on it with you, and it must be manually updated to HEAD. * No CL collaboration. Unlike Git branches, CLs can only contain changes from one user. * No stable branch. Since everything is essentially on one long branch, it's a real hassle when a project is broken at HEAD. Sure, integration tests should ideally prevent this. In practice, HEAD is often broken. Teams have created bash scripts and mailing lists to determine 'stable' old versions that can be checked out for development. * Single versions of libraries. Any library that is used is also checked into the VCS. However, only one version of the library can exist in the codebase, which is rarely updated. However, there are exceptions to this. At one point, Sergey mentioned bringing Google "up to industry standards" regarding VCS's. However, that would be a monumental task and I doubt it will happen.
  - jmillikin 10y ago
    Everything you posted is wrong, which makes me believe you know everything you posted is wrong. Nobody writes things that inaccurate by accident. For the benefit of YC, here are corrections: > No branches. If you want to make a temporary code branch, > you create a CL (Google's version of a pull request), but > never submit it. This means nobody else can collaborate on > it with you, and it must be manually updated to HEAD. Piper has branches, which can be committed to as normal by any number of engineers. It's common to have both branches for developing certain features ("dev branches"), and branches pinned to a stable base version with cherry-picked bug fixes ("release branches"). > No CL collaboration. Unlike Git branches, CLs can only > contain changes from one user. CLs are equivalent to Git commits. Collaboration is expected to occur via a series of CLs, just like Git-based projects have large changes made via a series of smaller commits. > No stable branch. Since everything is essentially on one > long branch, it's a real hassle when a project is broken > at HEAD. Sure, integration tests should ideally prevent > this. In practice, HEAD is often broken. Teams have > created bash scripts and mailing lists to determine > 'stable' old versions that can be checked out for > development. There is a global testing system for the entire repository, which is used to decide whether a particular project's tests pass. Commits on which all relevant tests pass are the branch point for releases. This is similar to the Linux kernel's dev model, where stable releases are cut at known-healthy points in an evolving codebase. Important libraries define more rigorous releases, similar to Git labels, which are updated automatically every day or two. These both reduce the amount of tests that need to run, and reduce chances of errors in low-level code affecting many teams. > Single versions of libraries. Any library that is used is > also checked into the VCS. However, only one version of > the library can exist in the codebase, which is rarely > updated. However, there are exceptions to this. Many third-party open-source libraries have multiple versions, and new upstream releases are added when there's either a security/bug fix, or someone wants a new feature. Three of the four languages most often used at Google (Python, C++, Go) do not allow multiple versions of a library to be linked into a single process due to symbol conflicts. This is a limitation of those languages, not of the Google repository, and they affect any company that allows use of third-party code. The standard recommendation at Google is to avoid dependency hell by sharding large binaries and using RPCs to communicate. This development model has many advantages that have been documented elsewhere.
anonsockpuppet 10y ago
> hunger is never a reason to leave But good luck finding a free restroom at 10am. Is this an issue at other companies?
- menssen 10y ago
  Nobody drives in New York because the traffic is so bad.
- RhodesianHunter 10y ago
  Literally everywhere. It's the post first coffee crush.
aanm1988 10y ago
They have a billion files in their repo, 9 million are source files. What the heck is the other 991000000? I skimmed this. Mostly just stuff any competent company would/should be doing. it's google though, so they act like it's super awesome.
- tehlike 10y ago
  Translation files, xml files, some data files, images, and lots of other things. Ps: goog employee
- deleted 10y ago
  [deleted]
- gumby 10y ago
  > What the heck is the other 991000000? Says right in the article: various config and dependency files, presumably both as caches (where everyone would generate the same product) or as a record of where things stood on at time t. For example: > In some cases, notably Go programs, build files can be generated (and updated) automatically, since the dependency information in the BUILD files is (often) an abstraction of the dependency information in the source files. But they are nevertheless checked in to the repository.
  - pzh 10y ago
    So basically somebody can write a script to put this Build file in gitignore, save the company millions of dollars, and get promoted for it?
    - gsylvie 10y ago
      People rarely get promoted for saving money.
      - williadc 10y ago
        Finance people do.
deleted 10y ago
[deleted]
harryjo 10y ago
Is section 3.1 "20%" time still true?
- raphlinus 10y ago
  It is for me.
- dontreact 10y ago
  In my experience, yes. Most people find it courteous to inform their manager but I did a lot of 20% time that led to getting hired with Google Brain, and it was never any doubt in my mind that my manager would approve. That being said, there were times when I was really excited about the project so I would work 80% time plus 40% time and stay late.
  - nlpleb 10y ago
    Can you tell us about what exactly you did that led you to getting hired with GB?
    - dontreact 10y ago
      I don't want to get too specific but I did a 20% project that led to a couple of conference abstracts with one subteam. This built relationships which helped when I applied internally to work on a different subteam. 20% time aiding in team transfer is very common I think.
- mamon 10y ago
  You guys seem to think that Google is benevolent by giving engineers that 20%, while in fact they do that for very selfish reasons: it is all about copyrights. They hire the smartest Software Engineers in the world, so it is only a matter of time that some of them will create new, disruptive product. If they weren't given that 20% of time at Google they would do that anyway, on weekends, but since they do it in company-sponsored time then Google owns the copyrights to all of their work.
  - 10y ago
l1feh4ck 10y ago
>>Engineers are permitte d to spend up to 20% of their time working on any project of their choice, without needing approval from their manager or anyone else. Can someone tell more about what they did and the things are that permitted (even though without needing approval.
- wutbrodo 10y ago
  It's all kinda of things: I did mine on another team I was thinking of transferring to (as a sort of pilot program), a friend of mine in ads worked on a cloud robotics team (this was like 5 years ago), another friend spent some time on a research team for the stuff he had done his thesis on, etc
- oscardelben 10y ago
  You definitely need to tell your manager though. EDIT: one personal example is at some point I took some external classes and used 20℅ time for it. But most cases are for developing some sort of tools or product.
- Itaxpica 10y ago
  In theory basically anything software related is possible, and I've never heard of a legitimately suggested 20% project being rejected, but these days its most common for people to 20% building a feature that interests them for another project, rather than an entirely new project.
OneMoreGoogle 10y ago
I spent ~18 months at Google, and one of the annoying aspects was the diversity of build systems. I built with Ninja, emerge, Blaze, and Android's Rube-Goldberg-shell-script system.
- 762236 10y ago
  > Android's Rube-Goldberg-shell-script system Perfect! We need a t-shirt.
mck- 10y ago
A lot of the things they had to build in-house for repo/build/test/deploy as described in chapter two, all of us are fortunate enough to get for (almost) free with all the tools these days. It's a good time to be a founder :)
fenollp 10y ago
> has released over 500,000 lines of open-source code Interesting metric. I wonder why no "social" VCSs report that metric?
Matthias247 10y ago
Interesting, compared to what's common in the automotive industry it doesn't even mention the terms "requirements", "specifications", "estimations", "project plan", "tracability", "UML", etc...
- hueving 10y ago
  Right, if it mentioned those, search would still be the version we saw in 2007. None of the other products would exist yet because they would still be conducting user studies based on wire frame mock ups of workflows designed by committees of behavioral psychologists. That's a bit dramatic, but when your product development has a fast turnaround for fixes (git push vs 100 million dollar recall) and it won't kill people when it breaks, you should immediately throw most of that process shit out the window. You can't be competitive in consumer SaaS if you get bogged down in 'real engineering' processes.
  - kuschku 10y ago
    Except, this is showing very much in reliability of Google, which is horrible. Just look at Google Nest and their massive outage in December 2015, that’s usually enough to kill a company, and if Nest was independent, it’d have hurt them massively. Their reliability is nice compared to competing websites, but compared to other infrastructure, it’s quite bad. Hell, I’ve seen 2 orders of magnitudes more Google outages than power outages in my entire life (combined 29min power outage vs. several days Google outage since '96)
  - deleted 10y ago
    [deleted]
  - nickpsecurity 10y ago
    A clear statement of what you're going to do, some constraints on the design a la Design-by-Contract, and languages/libraries that mitigate errors by design are so easy to do that small shops do them on a regular basis. Ada/SPARK, Eiffel, Ocaml, and Haskell are examples with steady business in industry with last three used on relatively-fast-moving projects. Add in static analysis, spec-based generation of tests, and/or fuzzers to get lots of reliability for free. Guess what? This method also scales excellently if the company has access to a huge pile of servers and engineers whose build system can automate all the checks with every submitted change. Your idea that it has to be as ridiculous as process junkies is a strawman. A strawman that happens in a lot of places for sure but doesn't have to. Google can just take the few, easy-to-apply practices from high-integrity systems to get tons of benefit. It's the 80/20 rule for increasing assurance.
hueving 10y ago
Google has gotten so large so quickly in the past 3 years, that I wonder how much damage has been done to their engineering culture. A lot of "less than stellar people" have joined in these recent years according to several of my friends that work there (in infrastructure and some ML groups). It seems the push to golang is entirely to sustain large projects with average engineers. Maybe somewhere high up they decided that it's better to just have a massive engineering workforce rather than only hiring top talent? At what point does brain drain start as the best people get sick of dealing with mediocrity?
- throwaway40483 10y ago
  As some point you have to hire someone to take out the trash. They can't all be Einsteins.
  - DBBKS 10y ago
    Funnily when I got to visit the Google office in London people just left their trash on the floor of the cafeteria for the cleaners..
    - benp84 10y ago
      I think his comment was mostly figurative.
      - varelse 10y ago
        Except no, I saw stuff like that too when I was there along with childish crap like throwing gum in the urinals and googlers acting like entitled jerks towards the cleaning staff. I met some great people at Google, but I'd be remiss not to mention I also met too many #IAmGoogle sorts (who were all dudes BTW).
- jboggan 10y ago
  I've only been there 13 months but I've found the quality of previous and new hires are ridiculously high. Given the number of talented people applying every day I think Google could increase hiring by 100% with no appreciable drop in quality. Google has a lot of imperfections addressed elsewhere in the comments here but the engineering culture is extremely strong and one of the best parts about being there. I think the overall "Googley culture" is suffering but it isn't for technical reasons. Then again maybe I'm one of those C players who snuck in the past year.
hueving 10y ago
>Engineers are permitted to spend up to 20% of their time working on any project of their choice, without needing approval from their manager or anyone else. From what I understand talking to current employees, this is now bullshit. You could spend 20% on other stuff, but the culture in many of the groups is such that you are putting your peer review at risk by doing so because of your reduced output. Any current Googlers that spend 1 day of every week working on something completely unrelated to their main job want to comment?
- throwaway40483 10y ago
  Even when 20% was in existence, it was (mockingly) called 120%. I will leave you to guess as to why.
- Itaxpica 10y ago
  This (like a lot of things at Google) varies wildly across PA and even among managers within a PA. All my managers so far have been extremely supportive of 20% time, and I've personally worked on three different 20% projects over my last four years (including two which have been open-sourced). That being said my engagement model has generally been less "one solid day a week" and more "a few days in a row once a month or two" in quieter times for my 80% gig, or "an hour or two every day" in busier ones. My understanding is that while I'm a little bit of an outlier, in general 20% time at Google is nowhere near as dead as people on the internet tend to claim (at least for engineers).
  - pyb 10y ago
    What is a PA ?
    - Itaxpica 10y ago
      Product Area (so Ads, Cloud, Search, etc).
- beder 10y ago
  I'm not currently, but I was doing that for the last couple years with no ill consequences. I'm only not doing that now because I don't have a 20% idea that's particularly exciting to me.
benzewdu 10y ago
Although there is a well-defined process for launch approvals, Google does not have a well-defined process for project approval or cancellation. Despite having been at Google for nearly 10 years, and now having become a manager myself, I still don’t fully understand how such decisions are made. The reason why I quit Google.
- dajohnson89 10y ago
  Customers feel it too -- eg reader.
- kyrra 10y ago
  The rest of the paragraph you quote is fairly important as it goes on to explain cases and how a project can be cancelled. The rest of the quote: In part this is because the approach to this is not uniform across the company. Managers at every level are responsible and accountable for what projects their teams work on, and exercise their discretion as they see fit. In some cases, this means that such decisions are made in a quite bottom-up fashion, with engineers being given freedom to choose which projects to work on, within their team’s scope. In other cases, such decisions are made in a much more top-down fashion, with executives or managers making decisions about which projects will go ahead, which will get additional resources, and which will get cancelled.
  - benzewdu 10y ago
    The question isn't how, it's why. I wish i could share the details.
- wingless 10y ago
  And on the other end of the spectrum we had Jobs era Apple, with One True God overseeing all demigods who oversee projects. IMO the things that made Apple products so successful was this structure which left no room for confusion and that the company was run by QA people. My impression of Jobs is that he was a QA guy himself.
  - qu1j0t3 10y ago
Chasmo 10y ago
> Software engineers at Google are strongly encouraged to program in one of four officially-approved programming languages at Google: C++, Java, Python, or Go. I wonder which of these languages they use to develop the google front page or any other frontend when no Javascript is allowed...
- mining 10y ago
  http://www.gwtproject.org/ http://www.gwtproject.org/ Slightly tongue in cheek (since Google obviously uses javascript), but Java :)
- Itaxpica 10y ago
  Those are the primary, general four languages, but other languages are used as necessary for specific domains (e.g. Javascript, Objective-C/Swift), they're just not guaranteed to get the same level of internal tooling and infrastructure support (though in practice they do).
- DocSavage 10y ago
  Clearly there is some use of Dart and JavaScript (vanilla, Angular and Polymer). It would be interesting to hear percentages across Google development.
- fergus_google 10y ago
  That's actually an error in the paper -- JavaScript is now an officially approved programming language at Google.
- user5994461 10y ago
  The only 4 languages I known. Awesome!
NumberSix 10y ago
Google is highly non-representative of businesses and technology businesses in particular. It has a near monopoly on the search business and has enormous amounts of money from a single source -- advertising. Google Fiscal Year 2015 Revenues: $74.54 Billion (source: Google) Advertising Revenue: $67.39 Billion (source: https://www.statista.com/statistics/266249/advertising-revenue-of-google/ https://www.statista.com/statistics/266249/advertising-reven...) Profits: $23.4 Billion Market Capitalization: $570 Billion Google Not a Startup/Has Not Been a Startup for Many Years Founded: September 4, 1998 (19 years ago) IPO: August 19, 2004 (13 years ago) Number of Employees (2015): 57,100 Revenues per Employee: $1.3 Million Profits per Employee: $409,000 The issue is that Google has so much money and is so successful that it can do all sorts of things that are extremely inefficient, even very harmful and still do fine, unlike smaller companies or startups in particular. For example the arxiv article states: 2.11. Frequent rewrites Most software at Google gets rewritten every few years. This may seem incredibly costly. Indeed, it does consume a large fraction of Google’s resources. Google has the money to do this. As the article argues, it may work for Google. On the other hand, Google has so much money and such a dominant market position, it probably can keep succeeding even if continual code rewriting is actively harmful to Google. In orthodox software engineering theory, competent software engineers, let alone the best of the best that Google claims to hire, should write modular highly reusable code that does not need to be rewritten. Many rewrites and "refactorings" are justified by claiming the "legacy" code is "bad code" that is not reusable and must be written by "real software engineers" to be reusable/maintainable/scalable etc. One rewrite ought then to be enough. Even highly successful businesses, for example $1 Billion dollars in revenues with $50 million in profits and 5000 employees (revenues per employee of $200,000), have nowhere near these vast resources --- either in total dollars or per employee or product unit shipped. Blindly copying highly expensive software development processes from Google or other super-unicorn companies like Apple or Facebook is likely a prescription for failure.
- nickpsecurity 10y ago
  I agree. Many companies, including mine, run critical stuff on applications that are over a decade old that do just fine. Worst thing you can say about them is that some weren't designed well enough for easy extensions or integration with newer apps. They do their job, though. The backbone of ours is mainframe code running on mainframes and AS/400's with simple, terminal interfaces. Terminal stuff is ultra-fast & reliable but sometimes ugly. Some have GUI apps that basically hide the terminal details they interact with to be a bit easier to use. Those terminal apps, probably 20+ years old, still work and get periodically updated. New people learn them easily, too, since interface was well-designed for the time. Can't goof off on the thin clients either as there's no web browser or native apps. ;) I've seen Google do some rewrites that make sense when one has the money for them. The shift from eventual to stronger consistency in their databases via F1 RDBMS was impressive. Worth some rewrites across the apps to use it to knock out major problems that could affect them once and for all. After developing Go, they also might want to rewrite performance-critical apps in, say, Python to it. There's definitely benefits on such rewrites. A lot of the other stuff I'm betting they could've done more long-term esp if using and extending FOSS solutions.
supremesaboteur 10y ago
> The maintenance of operational systems is done by software engineering teams, rather than traditional sysadmin types, but the hiring requirements for software engineering skills for the SRE are slightly lower than the requirements for the Software Engineering position. but from https://www.youtube.com/watch?v=H4vMcD7zKM0&feature=youtu.be&t=424 https://www.youtube.com/watch?v=H4vMcD7zKM0&feature=youtu.be... "All of SREs have to pass a full software interview to get hired"
- jmillikin 10y ago
  I'm an SRE. Both the OP article and the YouTube video are making generalizations that aren't really accurate. There's two positions called "SREs", SRE-SWE and SRE-SA. SRE-SWE has to pass a full software interview, and are equivalent to SWEs working for other departments. SRE-SA are not expected to have Google-level software engineering skills, but make up for that in other knowledge areas.
varelse 10y ago
Not one word about what I consider the most toxic aspect of working at Google: blind allocation. Unless one absolutely does not care what one wishes to work on, joining Google is throwing your future into the Hogwarts hat of an ill-defined cabal of billionaires to pick your job at Google. Sometimes that works out, but most of the time you are allocated to whatever mission-critical project is currently leaking buttcheeks. In my brief time there, I was placed on a 7-person team that lost one person per month. That is the single worst retention rate I have ever seen. I left after 4 months for that and the realization that the powers-that-be were in denial about the coming impact of an emerging technology that they have since embraced a year or so after my departure. That said, the SCCS and the build process were top-notch. But there are good reasons why despite all the perks and smart people, Google's overall retention rate is barely a month longer than Amazon's. http://www.slate.com/blogs/business_insider/2013/07/28/turnover_rates_by_company_how_amazon_google_and_others_stack_up.html http://www.slate.com/blogs/business_insider/2013/07/28/turno...
- fhoffa 10y ago
  Blind allocation? When I joined Google I was given 4 teams in 2 different locations to choose. I had lunch with each of the teams, and chose the one that better suited my style. To be fair, your experience at Google will depend a lot on what team you land in. Not everyone's given a choice, but now I know better: Talk to your recruiter and make them clear what you want.
  - varelse 10y ago
    Sure, that would be great, but as a result of leaving, I have been blacklisted by HR from ever returning. And the one guy that tried to bring me back in a year ago met in the lobby of his building and then walked me out to the marsh behind the GooglePlex before saying a word to me. I then asked if I could use said technology for which I am a recognized expert, he said no, and that was the end of that. I have been tempted to print myself a "Bad Cultural Fit - Google HR" T-shirt in response, but I have better things to do.
    - 10y ago
Upvoter33 10y ago
So much snark and negativity on this thread, when the paper is just a factual description of a pretty impressive feat of software engineering (still way better than most companies I have seen the inside of).
- varelse 10y ago
  I really like the first part of the paper and it describes a lot of their best practices, but the last 1/3 or so of the paper is more about Google's culture, no? IMO that opens them up for the snark here. Google has some amazing perks and it's a wonderful overall employer, but I really wonder how much cash is left on the table by not addressing its shockingly low retention rate or at least explaining why it's not a problem. There are a lot of articles estimating the cost of a departure is anywhere from 6-12 months of the lost employee's salary. Does that seem like something that should be ignored like they seemingly do? Why do shareholders even tolerate this?
  - Upvoter33 10y ago
    Fair point - I was mostly focused on the build system. It is quite something and a marvel of engineering. The culture is actually good in parts of the company -- like core systems infra -- but yes it is hard to spread that throughout, it seems. A separate article about culture and what's good and bad would be interesting, but probably not likely to be seen in public, particularly if it's honest...
dj-wonk 10y ago
I read most of the paper. For the most part, it struck a nice tone as being mostly descriptive and not too promotional. However, the final paragraph in the conclusion section differs: > "For those in other organizations who are advocating for the use of a particular practice that happens to be described in this paper, perhaps it will help to say “it’s good enough for Google”. In my opinion, this style of writing doesn't fit nor belong. I would leave that paragraph out. Instead, let's judge on the merits and applicability of an engineering practice based on thinking, reasoning, and experimentation. That said, as I've read various comments about Google's processes, I'm struck by the cognitive dissonance. On one hand, I see bandwagoning; e.g. "monolithic source control is nuts; we don't do that; no one I know does that". There is also some appeal to authority; e.g. "well, Google is the best, they do X, so we should too." I'm glad to see different argumentation fallacies colliding here.
- benashford 10y ago
  With one-or-two exceptions, what the paper describes is very familiar and sounds like most software teams, but most teams don't achieve Google-like performance/stability/success. The differentiation is in details that the paper doesn't explore. I agree, this paper will just lead to more monolithic Git repos "because it's good enough for Google", without the appreciation of Google's other tooling and processes.
jondubois 10y ago
The single-repo is quite surprising. How do they manage to store that much data in a central place? I guess they're using some sort of distributed network file system - This seems overly complex though. It would be interesting to know if this was intentional (if there is a reason for this) or things just evolved like this out of habit. I think that engineers in most large software companies don't actually accomplish much on a day-to-day basis. I'm saying this after having worked in both corporations and startups. Large companies are laughably inefficient - Engineers tend to focus all of their energy on very narrow (often tedious) problems at great depth. These huge companies never try to reign-in the complexity because they don't really need to - Soon enough, the employees at these companies get used to the enormous complexity and start thinking that it's normal - And when they switch jobs to a different company; they contaminate other companies with that attitude. That's why I think startups will always have an advantage over corporations when it comes to productivity. In these big companies, the most clever, complex solutions tend to win.
- kyrra 10y ago
  See my reply elsewhere in this post to the ACM article. It's custom in-house based on their own distributed databases. The front-end UI is based on perforce, but there is a Git front-end (but logically it behaves a lot like perforce). As the ACM paper calls out, files are loaded asynchronously like on access, so you only have the files you need on your desktop. From a developer's point of view, you can see all the files in the entire repo at all times. And with how CITC (client in the cloud) works, I can move from a laptop to a desktop without any effort (all of the files I have in a working state on one are immediately available everywhere I can access CITC). As far as complexity... how would a large company solve these problems without these large and complex systems? Google is probably one of the more efficient ways I've seen large companies work. When you want all your devs to share code, thing get hard at scale.
- gaius 10y ago
  The single-repo is quite surprising It means you can modify the interface to a service, and all the clients that use it, in an atomic commit.
mlinksva 10y ago
"Most software at Google gets rewritten every few years" at incredible expense. That sounds crazy, but the article claims it has benefits including keeping up with product requirements, eliminating complexity, and transferring ownership. Would be interesting to see some kind of metric indicating how much of an outlier Google really is here, and what measures it takes to make sure rewrites aren't worse (second system).
wmu 10y ago
Things I'm envy for: 1) documenting changes, 2) compulsory code review, 3) post-failure reports. I wish my company introduced at least one of those.

442 points by asnt 10y ago

sjakobi 10y ago

Why aren't OKR scores used as an input to performance appraisals?

fooker 10y ago
You can give yourself full marks.
npelly 10y ago
They would be far too easy to game.
bskap 10y ago
Because they aren't measures of how good you are at developing, testing, and releasing software, they're just measures of how good you are at estimating the amount of work it'll take to complete a project 3 months in advance.

__strisk 10y ago

Is the practice of shoving all disparate pieces of proprietary software (or individual projects) in the same repo a common occurrence? I have found that pulling unrelated changes just so that I can push my changes is an inconvenience. Furthermore, tracking the history of a particular project is confounded due to unrelated commits. I am sure that their vcs (piper?) makes this a feasible task, but for git, it seems like it would suck. The article posted by kyrra, mentions this. Given the value gained from the existing tools Google has built and the many advantages of the monolithic codebase structure, it is clear that moving to more and smaller repositories would not make sense for Google's main repository. The alternative of moving to Git or any other DVCS that would require repository splitting is not compelling for Google. It seems like they have just too much invested in this "shove it in the same repo" style. Or is this the more appropriate way to do things in a large organization?

tehlike 10y ago
It actually works quite nicely. Most of the google software is built internally. Ensuring everyone is running at head is a blessing (rarely a curse), because when you make a change you immediately see if it breaks something. You can also be sure everything gets all the bug fixes in their new release. This is good for various reasons, including the fact that it makes security audits significantly simpler. Some google orgs, like android, don't use the existing infra, and they kind of struggle and have to reinvent most wheels, because of that. Edit: added second paragraph
throwawayGogEng 10y ago
Coming from companies that use reasonably-sized git repos, I absolutely hated Google's VCS. Here's some of my painpoints with it: * No branches. If you want to make a temporary code branch, you create a CL (Google's version of a pull request), but never submit it. This means nobody else can collaborate on it with you, and it must be manually updated to HEAD. * No CL collaboration. Unlike Git branches, CLs can only contain changes from one user. * No stable branch. Since everything is essentially on one long branch, it's a real hassle when a project is broken at HEAD. Sure, integration tests should ideally prevent this. In practice, HEAD is often broken. Teams have created bash scripts and mailing lists to determine 'stable' old versions that can be checked out for development. * Single versions of libraries. Any library that is used is also checked into the VCS. However, only one version of the library can exist in the codebase, which is rarely updated. However, there are exceptions to this. At one point, Sergey mentioned bringing Google "up to industry standards" regarding VCS's. However, that would be a monumental task and I doubt it will happen.
- jmillikin 10y ago
  Everything you posted is wrong, which makes me believe you know everything you posted is wrong. Nobody writes things that inaccurate by accident. For the benefit of YC, here are corrections: > No branches. If you want to make a temporary code branch, > you create a CL (Google's version of a pull request), but > never submit it. This means nobody else can collaborate on > it with you, and it must be manually updated to HEAD. Piper has branches, which can be committed to as normal by any number of engineers. It's common to have both branches for developing certain features ("dev branches"), and branches pinned to a stable base version with cherry-picked bug fixes ("release branches"). > No CL collaboration. Unlike Git branches, CLs can only > contain changes from one user. CLs are equivalent to Git commits. Collaboration is expected to occur via a series of CLs, just like Git-based projects have large changes made via a series of smaller commits. > No stable branch. Since everything is essentially on one > long branch, it's a real hassle when a project is broken > at HEAD. Sure, integration tests should ideally prevent > this. In practice, HEAD is often broken. Teams have > created bash scripts and mailing lists to determine > 'stable' old versions that can be checked out for > development. There is a global testing system for the entire repository, which is used to decide whether a particular project's tests pass. Commits on which all relevant tests pass are the branch point for releases. This is similar to the Linux kernel's dev model, where stable releases are cut at known-healthy points in an evolving codebase. Important libraries define more rigorous releases, similar to Git labels, which are updated automatically every day or two. These both reduce the amount of tests that need to run, and reduce chances of errors in low-level code affecting many teams. > Single versions of libraries. Any library that is used is > also checked into the VCS. However, only one version of > the library can exist in the codebase, which is rarely > updated. However, there are exceptions to this. Many third-party open-source libraries have multiple versions, and new upstream releases are added when there's either a security/bug fix, or someone wants a new feature. Three of the four languages most often used at Google (Python, C++, Go) do not allow multiple versions of a library to be linked into a single process due to symbol conflicts. This is a limitation of those languages, not of the Google repository, and they affect any company that allows use of third-party code. The standard recommendation at Google is to avoid dependency hell by sharding large binaries and using RPCs to communicate. This development model has many advantages that have been documented elsewhere.

anonsockpuppet 10y ago

> hunger is never a reason to leave But good luck finding a free restroom at 10am. Is this an issue at other companies?

menssen 10y ago
Nobody drives in New York because the traffic is so bad.
RhodesianHunter 10y ago
Literally everywhere. It's the post first coffee crush.

aanm1988 10y ago

They have a billion files in their repo, 9 million are source files. What the heck is the other 991000000? I skimmed this. Mostly just stuff any competent company would/should be doing. it's google though, so they act like it's super awesome.

tehlike 10y ago
Translation files, xml files, some data files, images, and lots of other things. Ps: goog employee
deleted 10y ago
[deleted]
gumby 10y ago
> What the heck is the other 991000000? Says right in the article: various config and dependency files, presumably both as caches (where everyone would generate the same product) or as a record of where things stood on at time t. For example: > In some cases, notably Go programs, build files can be generated (and updated) automatically, since the dependency information in the BUILD files is (often) an abstraction of the dependency information in the source files. But they are nevertheless checked in to the repository.
- pzh 10y ago
  So basically somebody can write a script to put this Build file in gitignore, save the company millions of dollars, and get promoted for it?
  - gsylvie 10y ago
    People rarely get promoted for saving money.
    - williadc 10y ago
      Finance people do.

deleted 10y ago

[deleted]

harryjo 10y ago

Is section 3.1 "20%" time still true?

raphlinus 10y ago
It is for me.
dontreact 10y ago
In my experience, yes. Most people find it courteous to inform their manager but I did a lot of 20% time that led to getting hired with Google Brain, and it was never any doubt in my mind that my manager would approve. That being said, there were times when I was really excited about the project so I would work 80% time plus 40% time and stay late.
- nlpleb 10y ago
  Can you tell us about what exactly you did that led you to getting hired with GB?
  - dontreact 10y ago
    I don't want to get too specific but I did a 20% project that led to a couple of conference abstracts with one subteam. This built relationships which helped when I applied internally to work on a different subteam. 20% time aiding in team transfer is very common I think.
mamon 10y ago
You guys seem to think that Google is benevolent by giving engineers that 20%, while in fact they do that for very selfish reasons: it is all about copyrights. They hire the smartest Software Engineers in the world, so it is only a matter of time that some of them will create new, disruptive product. If they weren't given that 20% of time at Google they would do that anyway, on weekends, but since they do it in company-sponsored time then Google owns the copyrights to all of their work.
- 10y ago

l1feh4ck 10y ago

>>Engineers are permitte d to spend up to 20% of their time working on any project of their choice, without needing approval from their manager or anyone else. Can someone tell more about what they did and the things are that permitted (even though without needing approval.

wutbrodo 10y ago
It's all kinda of things: I did mine on another team I was thinking of transferring to (as a sort of pilot program), a friend of mine in ads worked on a cloud robotics team (this was like 5 years ago), another friend spent some time on a research team for the stuff he had done his thesis on, etc
oscardelben 10y ago
You definitely need to tell your manager though. EDIT: one personal example is at some point I took some external classes and used 20℅ time for it. But most cases are for developing some sort of tools or product.
Itaxpica 10y ago
In theory basically anything software related is possible, and I've never heard of a legitimately suggested 20% project being rejected, but these days its most common for people to 20% building a feature that interests them for another project, rather than an entirely new project.

OneMoreGoogle 10y ago

I spent ~18 months at Google, and one of the annoying aspects was the diversity of build systems. I built with Ninja, emerge, Blaze, and Android's Rube-Goldberg-shell-script system.

762236 10y ago
> Android's Rube-Goldberg-shell-script system Perfect! We need a t-shirt.

mck- 10y ago

A lot of the things they had to build in-house for repo/build/test/deploy as described in chapter two, all of us are fortunate enough to get for (almost) free with all the tools these days. It's a good time to be a founder :)

fenollp 10y ago

> has released over 500,000 lines of open-source code Interesting metric. I wonder why no "social" VCSs report that metric?

Matthias247 10y ago

Interesting, compared to what's common in the automotive industry it doesn't even mention the terms "requirements", "specifications", "estimations", "project plan", "tracability", "UML", etc...

hueving 10y ago
Right, if it mentioned those, search would still be the version we saw in 2007. None of the other products would exist yet because they would still be conducting user studies based on wire frame mock ups of workflows designed by committees of behavioral psychologists. That's a bit dramatic, but when your product development has a fast turnaround for fixes (git push vs 100 million dollar recall) and it won't kill people when it breaks, you should immediately throw most of that process shit out the window. You can't be competitive in consumer SaaS if you get bogged down in 'real engineering' processes.
- kuschku 10y ago
  Except, this is showing very much in reliability of Google, which is horrible. Just look at Google Nest and their massive outage in December 2015, that’s usually enough to kill a company, and if Nest was independent, it’d have hurt them massively. Their reliability is nice compared to competing websites, but compared to other infrastructure, it’s quite bad. Hell, I’ve seen 2 orders of magnitudes more Google outages than power outages in my entire life (combined 29min power outage vs. several days Google outage since '96)
- deleted 10y ago
  [deleted]
- nickpsecurity 10y ago
  A clear statement of what you're going to do, some constraints on the design a la Design-by-Contract, and languages/libraries that mitigate errors by design are so easy to do that small shops do them on a regular basis. Ada/SPARK, Eiffel, Ocaml, and Haskell are examples with steady business in industry with last three used on relatively-fast-moving projects. Add in static analysis, spec-based generation of tests, and/or fuzzers to get lots of reliability for free. Guess what? This method also scales excellently if the company has access to a huge pile of servers and engineers whose build system can automate all the checks with every submitted change. Your idea that it has to be as ridiculous as process junkies is a strawman. A strawman that happens in a lot of places for sure but doesn't have to. Google can just take the few, easy-to-apply practices from high-integrity systems to get tons of benefit. It's the 80/20 rule for increasing assurance.

hueving 10y ago

Google has gotten so large so quickly in the past 3 years, that I wonder how much damage has been done to their engineering culture. A lot of "less than stellar people" have joined in these recent years according to several of my friends that work there (in infrastructure and some ML groups). It seems the push to golang is entirely to sustain large projects with average engineers. Maybe somewhere high up they decided that it's better to just have a massive engineering workforce rather than only hiring top talent? At what point does brain drain start as the best people get sick of dealing with mediocrity?

throwaway40483 10y ago
As some point you have to hire someone to take out the trash. They can't all be Einsteins.
- DBBKS 10y ago
  Funnily when I got to visit the Google office in London people just left their trash on the floor of the cafeteria for the cleaners..
  - benp84 10y ago
    I think his comment was mostly figurative.
    - varelse 10y ago
      Except no, I saw stuff like that too when I was there along with childish crap like throwing gum in the urinals and googlers acting like entitled jerks towards the cleaning staff. I met some great people at Google, but I'd be remiss not to mention I also met too many #IAmGoogle sorts (who were all dudes BTW).
jboggan 10y ago
I've only been there 13 months but I've found the quality of previous and new hires are ridiculously high. Given the number of talented people applying every day I think Google could increase hiring by 100% with no appreciable drop in quality. Google has a lot of imperfections addressed elsewhere in the comments here but the engineering culture is extremely strong and one of the best parts about being there. I think the overall "Googley culture" is suffering but it isn't for technical reasons. Then again maybe I'm one of those C players who snuck in the past year.

hueving 10y ago

>Engineers are permitted to spend up to 20% of their time working on any project of their choice, without needing approval from their manager or anyone else. From what I understand talking to current employees, this is now bullshit. You could spend 20% on other stuff, but the culture in many of the groups is such that you are putting your peer review at risk by doing so because of your reduced output. Any current Googlers that spend 1 day of every week working on something completely unrelated to their main job want to comment?

throwaway40483 10y ago
Even when 20% was in existence, it was (mockingly) called 120%. I will leave you to guess as to why.
Itaxpica 10y ago
This (like a lot of things at Google) varies wildly across PA and even among managers within a PA. All my managers so far have been extremely supportive of 20% time, and I've personally worked on three different 20% projects over my last four years (including two which have been open-sourced). That being said my engagement model has generally been less "one solid day a week" and more "a few days in a row once a month or two" in quieter times for my 80% gig, or "an hour or two every day" in busier ones. My understanding is that while I'm a little bit of an outlier, in general 20% time at Google is nowhere near as dead as people on the internet tend to claim (at least for engineers).
- pyb 10y ago
  What is a PA ?
  - Itaxpica 10y ago
    Product Area (so Ads, Cloud, Search, etc).
beder 10y ago
I'm not currently, but I was doing that for the last couple years with no ill consequences. I'm only not doing that now because I don't have a 20% idea that's particularly exciting to me.

benzewdu 10y ago

Although there is a well-defined process for launch approvals, Google does not have a well-defined process for project approval or cancellation. Despite having been at Google for nearly 10 years, and now having become a manager myself, I still don’t fully understand how such decisions are made. The reason why I quit Google.

dajohnson89 10y ago
Customers feel it too -- eg reader.
kyrra 10y ago
The rest of the paragraph you quote is fairly important as it goes on to explain cases and how a project can be cancelled. The rest of the quote: In part this is because the approach to this is not uniform across the company. Managers at every level are responsible and accountable for what projects their teams work on, and exercise their discretion as they see fit. In some cases, this means that such decisions are made in a quite bottom-up fashion, with engineers being given freedom to choose which projects to work on, within their team’s scope. In other cases, such decisions are made in a much more top-down fashion, with executives or managers making decisions about which projects will go ahead, which will get additional resources, and which will get cancelled.
- benzewdu 10y ago
  The question isn't how, it's why. I wish i could share the details.
wingless 10y ago
And on the other end of the spectrum we had Jobs era Apple, with One True God overseeing all demigods who oversee projects. IMO the things that made Apple products so successful was this structure which left no room for confusion and that the company was run by QA people. My impression of Jobs is that he was a QA guy himself.
- qu1j0t3 10y ago

Chasmo 10y ago

> Software engineers at Google are strongly encouraged to program in one of four officially-approved programming languages at Google: C++, Java, Python, or Go. I wonder which of these languages they use to develop the google front page or any other frontend when no Javascript is allowed...

mining 10y ago
http://www.gwtproject.org/ http://www.gwtproject.org/ Slightly tongue in cheek (since Google obviously uses javascript), but Java :)
Itaxpica 10y ago
Those are the primary, general four languages, but other languages are used as necessary for specific domains (e.g. Javascript, Objective-C/Swift), they're just not guaranteed to get the same level of internal tooling and infrastructure support (though in practice they do).
DocSavage 10y ago
Clearly there is some use of Dart and JavaScript (vanilla, Angular and Polymer). It would be interesting to hear percentages across Google development.
fergus_google 10y ago
That's actually an error in the paper -- JavaScript is now an officially approved programming language at Google.
user5994461 10y ago
The only 4 languages I known. Awesome!

NumberSix 10y ago

Google is highly non-representative of businesses and technology businesses in particular. It has a near monopoly on the search business and has enormous amounts of money from a single source -- advertising. Google Fiscal Year 2015 Revenues: $74.54 Billion (source: Google) Advertising Revenue: $67.39 Billion (source: https://www.statista.com/statistics/266249/advertising-revenue-of-google/ https://www.statista.com/statistics/266249/advertising-reven...) Profits: $23.4 Billion Market Capitalization: $570 Billion Google Not a Startup/Has Not Been a Startup for Many Years Founded: September 4, 1998 (19 years ago) IPO: August 19, 2004 (13 years ago) Number of Employees (2015): 57,100 Revenues per Employee: $1.3 Million Profits per Employee: $409,000 The issue is that Google has so much money and is so successful that it can do all sorts of things that are extremely inefficient, even very harmful and still do fine, unlike smaller companies or startups in particular. For example the arxiv article states: 2.11. Frequent rewrites Most software at Google gets rewritten every few years. This may seem incredibly costly. Indeed, it does consume a large fraction of Google’s resources. Google has the money to do this. As the article argues, it may work for Google. On the other hand, Google has so much money and such a dominant market position, it probably can keep succeeding even if continual code rewriting is actively harmful to Google. In orthodox software engineering theory, competent software engineers, let alone the best of the best that Google claims to hire, should write modular highly reusable code that does not need to be rewritten. Many rewrites and "refactorings" are justified by claiming the "legacy" code is "bad code" that is not reusable and must be written by "real software engineers" to be reusable/maintainable/scalable etc. One rewrite ought then to be enough. Even highly successful businesses, for example $1 Billion dollars in revenues with $50 million in profits and 5000 employees (revenues per employee of $200,000), have nowhere near these vast resources --- either in total dollars or per employee or product unit shipped. Blindly copying highly expensive software development processes from Google or other super-unicorn companies like Apple or Facebook is likely a prescription for failure.

nickpsecurity 10y ago
I agree. Many companies, including mine, run critical stuff on applications that are over a decade old that do just fine. Worst thing you can say about them is that some weren't designed well enough for easy extensions or integration with newer apps. They do their job, though. The backbone of ours is mainframe code running on mainframes and AS/400's with simple, terminal interfaces. Terminal stuff is ultra-fast & reliable but sometimes ugly. Some have GUI apps that basically hide the terminal details they interact with to be a bit easier to use. Those terminal apps, probably 20+ years old, still work and get periodically updated. New people learn them easily, too, since interface was well-designed for the time. Can't goof off on the thin clients either as there's no web browser or native apps. ;) I've seen Google do some rewrites that make sense when one has the money for them. The shift from eventual to stronger consistency in their databases via F1 RDBMS was impressive. Worth some rewrites across the apps to use it to knock out major problems that could affect them once and for all. After developing Go, they also might want to rewrite performance-critical apps in, say, Python to it. There's definitely benefits on such rewrites. A lot of the other stuff I'm betting they could've done more long-term esp if using and extending FOSS solutions.

supremesaboteur 10y ago

> The maintenance of operational systems is done by software engineering teams, rather than traditional sysadmin types, but the hiring requirements for software engineering skills for the SRE are slightly lower than the requirements for the Software Engineering position. but from https://www.youtube.com/watch?v=H4vMcD7zKM0&feature=youtu.be&t=424 https://www.youtube.com/watch?v=H4vMcD7zKM0&feature=youtu.be... "All of SREs have to pass a full software interview to get hired"

jmillikin 10y ago
I'm an SRE. Both the OP article and the YouTube video are making generalizations that aren't really accurate. There's two positions called "SREs", SRE-SWE and SRE-SA. SRE-SWE has to pass a full software interview, and are equivalent to SWEs working for other departments. SRE-SA are not expected to have Google-level software engineering skills, but make up for that in other knowledge areas.

varelse 10y ago

Not one word about what I consider the most toxic aspect of working at Google: blind allocation. Unless one absolutely does not care what one wishes to work on, joining Google is throwing your future into the Hogwarts hat of an ill-defined cabal of billionaires to pick your job at Google. Sometimes that works out, but most of the time you are allocated to whatever mission-critical project is currently leaking buttcheeks. In my brief time there, I was placed on a 7-person team that lost one person per month. That is the single worst retention rate I have ever seen. I left after 4 months for that and the realization that the powers-that-be were in denial about the coming impact of an emerging technology that they have since embraced a year or so after my departure. That said, the SCCS and the build process were top-notch. But there are good reasons why despite all the perks and smart people, Google's overall retention rate is barely a month longer than Amazon's. http://www.slate.com/blogs/business_insider/2013/07/28/turnover_rates_by_company_how_amazon_google_and_others_stack_up.html http://www.slate.com/blogs/business_insider/2013/07/28/turno...

fhoffa 10y ago
Blind allocation? When I joined Google I was given 4 teams in 2 different locations to choose. I had lunch with each of the teams, and chose the one that better suited my style. To be fair, your experience at Google will depend a lot on what team you land in. Not everyone's given a choice, but now I know better: Talk to your recruiter and make them clear what you want.
- varelse 10y ago
  Sure, that would be great, but as a result of leaving, I have been blacklisted by HR from ever returning. And the one guy that tried to bring me back in a year ago met in the lobby of his building and then walked me out to the marsh behind the GooglePlex before saying a word to me. I then asked if I could use said technology for which I am a recognized expert, he said no, and that was the end of that. I have been tempted to print myself a "Bad Cultural Fit - Google HR" T-shirt in response, but I have better things to do.
  - 10y ago

Upvoter33 10y ago

So much snark and negativity on this thread, when the paper is just a factual description of a pretty impressive feat of software engineering (still way better than most companies I have seen the inside of).

varelse 10y ago
I really like the first part of the paper and it describes a lot of their best practices, but the last 1/3 or so of the paper is more about Google's culture, no? IMO that opens them up for the snark here. Google has some amazing perks and it's a wonderful overall employer, but I really wonder how much cash is left on the table by not addressing its shockingly low retention rate or at least explaining why it's not a problem. There are a lot of articles estimating the cost of a departure is anywhere from 6-12 months of the lost employee's salary. Does that seem like something that should be ignored like they seemingly do? Why do shareholders even tolerate this?
- Upvoter33 10y ago
  Fair point - I was mostly focused on the build system. It is quite something and a marvel of engineering. The culture is actually good in parts of the company -- like core systems infra -- but yes it is hard to spread that throughout, it seems. A separate article about culture and what's good and bad would be interesting, but probably not likely to be seen in public, particularly if it's honest...

dj-wonk 10y ago

I read most of the paper. For the most part, it struck a nice tone as being mostly descriptive and not too promotional. However, the final paragraph in the conclusion section differs: > "For those in other organizations who are advocating for the use of a particular practice that happens to be described in this paper, perhaps it will help to say “it’s good enough for Google”. In my opinion, this style of writing doesn't fit nor belong. I would leave that paragraph out. Instead, let's judge on the merits and applicability of an engineering practice based on thinking, reasoning, and experimentation. That said, as I've read various comments about Google's processes, I'm struck by the cognitive dissonance. On one hand, I see bandwagoning; e.g. "monolithic source control is nuts; we don't do that; no one I know does that". There is also some appeal to authority; e.g. "well, Google is the best, they do X, so we should too." I'm glad to see different argumentation fallacies colliding here.

benashford 10y ago
With one-or-two exceptions, what the paper describes is very familiar and sounds like most software teams, but most teams don't achieve Google-like performance/stability/success. The differentiation is in details that the paper doesn't explore. I agree, this paper will just lead to more monolithic Git repos "because it's good enough for Google", without the appreciation of Google's other tooling and processes.

jondubois 10y ago

The single-repo is quite surprising. How do they manage to store that much data in a central place? I guess they're using some sort of distributed network file system - This seems overly complex though. It would be interesting to know if this was intentional (if there is a reason for this) or things just evolved like this out of habit. I think that engineers in most large software companies don't actually accomplish much on a day-to-day basis. I'm saying this after having worked in both corporations and startups. Large companies are laughably inefficient - Engineers tend to focus all of their energy on very narrow (often tedious) problems at great depth. These huge companies never try to reign-in the complexity because they don't really need to - Soon enough, the employees at these companies get used to the enormous complexity and start thinking that it's normal - And when they switch jobs to a different company; they contaminate other companies with that attitude. That's why I think startups will always have an advantage over corporations when it comes to productivity. In these big companies, the most clever, complex solutions tend to win.

kyrra 10y ago
See my reply elsewhere in this post to the ACM article. It's custom in-house based on their own distributed databases. The front-end UI is based on perforce, but there is a Git front-end (but logically it behaves a lot like perforce). As the ACM paper calls out, files are loaded asynchronously like on access, so you only have the files you need on your desktop. From a developer's point of view, you can see all the files in the entire repo at all times. And with how CITC (client in the cloud) works, I can move from a laptop to a desktop without any effort (all of the files I have in a working state on one are immediately available everywhere I can access CITC). As far as complexity... how would a large company solve these problems without these large and complex systems? Google is probably one of the more efficient ways I've seen large companies work. When you want all your devs to share code, thing get hard at scale.
gaius 10y ago
The single-repo is quite surprising It means you can modify the interface to a service, and all the clients that use it, in an atomic commit.

mlinksva 10y ago

"Most software at Google gets rewritten every few years" at incredible expense. That sounds crazy, but the article claims it has benefits including keeping up with product requirements, eliminating complexity, and transferring ownership. Would be interesting to see some kind of metric indicating how much of an outlier Google really is here, and what measures it takes to make sure rewrites aren't worse (second system).

wmu 10y ago

Things I'm envy for: 1) documenting changes, 2) compulsory code review, 3) post-failure reports. I wish my company introduced at least one of those.