13 ms·

Cool. Could someone, maybe an ex-googler, comment on which parts of these work well and which don't? A lot of other companies get into trouble trying to cargo-

Cool. Could someone, maybe an ex-googler, comment on which parts of these work well and which don't? A lot of other companies get into trouble trying to cargo-cult what Google does when they are operating in very different environments wherein those practices aren't optimal. E.g. different levels of scale. Additionally, critics of Google may point out that their engineering culture may not be great on its own terms -- every time Google launches a new feature, people post links to the Google product graveyard.

hiddencost 3y ago
Engineering culture has somewhat collapsed at Google. The things that made engineering great didn't really survive the last couple rounds of internal coups.
- n0zmer 3y ago
  I thought so too, but since then I moved over to Cloud and things are a LOT better.
  - jupblb 3y ago
    Nice bait
  - eddd-ddde 3y ago
    I always here about Cloud having the worst culture tho? Has that not been the case to you?
    - VirusNewbie 3y ago
      People work hard in cloud but there are no MBAs in sight. It’s all very technical work, often very bottom up driven. A lot of the overall goals of cloud are more ambitious than AWS offerings. Reliability is prized more than it is in other areas of Google as well, because customers are so technical and often notice. Not a place to coast, but I’d say most people do a solid 45 a week for those that want to get good reviews and get a fat bonus.
      - nonethewiser 3y ago
        > A lot of the overall goals of cloud are more ambitious than AWS offerings. In what sense?
    - opportune 3y ago
      Some teams in Cloud suck but the core engineering teams have some top talent and solve some very hard problems. Keep in mind Borg and Spanner are both “cloud”, but so are many field sales teams with an average tenure <2y
    - n0zmer 3y ago
      Not at all, it's a breath of fresh air. It's much less of a check buganizer, check email, write code, push CL cycle. Work is very project focused with high flexibility, my current team isn't even pushing to g3 and using different build systems entirely just because we wanted some more flexibility for example, and it doesn't matter as long as we are getting results. The problems are a lot more technical though and I don't see a lot of L3s being able to work in the environment as it requires a lot more intuition and experience. I usually work 45hours a week, but I don't mind it. Plus I'm 100% WFH here cause my management isn't dealing with in office bs.
  - deleted 3y ago
    [deleted]
- yowlingcat 3y ago
  Interesting -- having not had any experience inside Google I'm having difficulty painting a picture, could you give an example or two of some of these internal coups?
NoMoreNicksLeft 3y ago
> Additionally, critics of Google may point out that their engineering culture may not be great on its own terms -- every time Google launches a new feature, people post links to the Google product graveyard. It is personally scary when they develop new products. What if it is a brilliant idea, one I cannot live without? If Google develops it, then I am looking at this stillborn thing, mewling for life when I know its horrible fate. The trouble here is that Google employees (and perhaps even its upper management) want to believe that they are a company which is an inventor of things. But they are not this at all. They are an advertisement company. Advertisement companies should not and do not want to invent things... inventions are worse than burdens, inventions are these weird alien objects that appear valuable but are quite expensive and do not help to sell ads at all. So they hawk the inventions like they were freaks in some carnival sideshow to move traffic past their billboards. Until the traffic dwindles (or until they get tired of it). And then they take it out back behind the woodshed and put an end to it.
- outside1234 3y ago
  The inventions are to keep the talent stream coming ... to work on ads. The inventions are the small tax they pay to pretend to candidates that they could work on inventions when the vast majority of them will be "allocated" to ads.
  - fidotron 3y ago
    It's also Google licking the cookie. They maintain a moat around ads by doing just enough to threaten to destroy anyone that gets close to their ecosystem. Facebook survived because G+ product vision was so out of touch with reality and FB were not part of the anti-compete hiring nonsense so that they managed to poach a lot of good people.
  - NoMoreNicksLeft 3y ago
    You're telling me this stupid, bizarre thing: that Google's major innovation was an HR process. That's fucked-in-the-head just enough that you've made me wonder if it's true.
jeffbee 3y ago
In my ex-Google experience, here are the stages of denial about something that Google does which is good but the industry doesn't yet embrace. Stage 1: "We're not Google, we don't need [[whatever]]"; Stage 2: Foreseeable disaster for which [[whatever]] was intended to address happens; Stage 3: Giant circus of P0/SEV0 action items while everyone assiduously ignores the [[whatever]]; Stage 4: Quiet accretion, over several years, of the [[whatever]] by people who understand it. And the [[whatever]] ranges from things that are obviously beneficial like pre-commit code review to other clear winners like multi-tenant machines, unit testing, user data encryption, etc etc. It is an extremely strange industry that fails to study and adopt the ways of their extremely successful competitors.
- MarceColl 3y ago
  The tools, design and manpower needed to build a skyscraper are different from those needed to build a 1-story wood house. It's not that the ones that build the wood house are failing to study and adopt the ways of their extremely successful competitors. Now, some of the things you say like unit-testing and user data encryption are ones that I've never seen associated with the "We're not google" mindset, so maybe people have started using that phrase for anythingnow
  - jeffbee 3y ago
    I'm not really talking about artisanal 3-man software shops, I'm talking about mid-sized companies with thousands of engineers, who don't realize they are already larger than and facing the same problems as Google was when they started adopting these practices. And to be clear, rejecting something as proven as pre-commit code review is not only to reject the example of Google and many other very successful enterprises, but also to ignore decades of developer productivity knowledge before Google existed. It's almost like the fact that Google adopted a long-standing best practice makes modern engineers reflexively revolt against those best practices. This can only be seen as a structural advantage for Google.
    - dinvlad 3y ago
      Curious how pre-commit code review worked, could you please elaborate a bit?
drewg123 3y ago
"Readability" works terribly when your company is acquired and your team enters all at the same time. Google has (or had ~10 years ago), a thing called "readability" for each language, where in order to be allowed to commit code to the central "google3" repo, you needed to have written some large amount of code in that language, and needed to have a readability reviewer sign off on your code. The process is designed for slowly on-boarding junior people into a team, and introducing them to google coding style and practices. Eg, the senior, mentoring folks on the team do the reviews and bring the new person up to speed. I imagine it must work well in that context. However, this breaks down when your entire team is new. How do you find somebody to review the code? All several million lines of the product that was acquired? Especially when it is written in multiple languages. So we were basically locked out of the main corporate repo, unable to do anything productive. We finally figured out that there was a paved path with a git repo used by the kernel team (and android?) that had none of these hurdles, where we could put our code and get productive immediately.
- jefftk 3y ago
  The way it's supposed to work is that acquired teams get lots of support on integration, including readability. This helps your team get integrated into writing Google-style code. Not sure why that didn't work out in this case? (Left Google a year ago)
  - badlucklottery 3y ago
    This. Someone dropped the ball. There's a form to get your corner of the codebase exempted from readability temporarily. This gives your team a quarter or two to build up readability.
    - Espressosaurus 3y ago
      A quarter or two isn't going to be enough for a drastic realignment of a large codebase. It's a start, but only a start.
      - 3y ago
marcyb5st 3y ago
DISCLAIMER 1: Current Googler here, but opinions are my own. DISCLAIMER 2: I think from a hands-on-keyboard SWE there is a lot of useful stuff. What you mentioned about Google culture of killing products and such I am not gonna talk about. I recommend chapters about testing first and foremost. Among all the codebases I saw (both OS and proprietary) Google tests are the most comprehensive and reliable. However, If you are in a startup-like environment you should pick and choose and not try to follow every single principle listed as they could sink your velocity drastically in the short run. Other interesting points (IMHO) are Monorepo, Build System, and Code Reviews. For the Monorepo I discover being a huge lover although I was skeptical. The sad thing is that it's a rather niche practice and tools like Git don't play ball very well (i.e. each time you pull you have to retrieve changes for all the codebase, even files you never saw/heard of managed by another team). I think there's no nice off-the-shelf offering for running monorepos out there. However, not having to fight with git submodules, library versions, ... is great. If the change I am submitting breaks something else in the company you are immediately aware and so can act accordingly (e.g. keep the old implementation alongside the new one and mark it as deprecated so the other team will get a warning next time they do anything). The build system is a bit more controversial. I learned to love blaze/bazel, but admittedly, the OS version is a bit messy to set up. Additionally, being so rigorous about the build rules felt like a massive chore at the beginning, but now I appreciate it a lot. I can instantly know the contacts of all the teams that use a build rule I declared and hence can be contacted to warn them about bugs, ... . I can create something experimental and have private visibility so only my team can use it and only later expose it to the wider world with just a one liner. Finally, the code review AKA Critique. Google has the best review tool I had the joy to use hands down. It's clear about what happens, at which stage is the review of a particular section/file and is focused on discussion. The evolution of each change is easy to follow along. These are things I really miss when using GitHub/GitLab PR view. The tooling is incredibly confusing to me. Luckily (I am not affiliated in any way) an ex-Googler (I believe) is working on an alternative that works with GitHub (https://codeapprove.com/ https://codeapprove.com/).
- deleted 3y ago
  [deleted]
- nonethewiser 3y ago
UncleMeat 3y ago
In my opinion, the monorepo, global presubmit, testing culture and the beyonce rule (if you liked it then you should have put a test on it) are basically a superpower for infrastructure teams. Without these things it'd be utterly impossible for certain kinds of infra refactors to be done and many more would be very very painful. In the open source world I see a fair amount of "tests are always red, don't worry" and "we can never edit this interface because who knows who it breaks." These problems aren't intractable at Google. This approach does have its own set of challenges and I do suspect that the monorepo has contributed in some ways to Google's inability or refusal to maintain some older products. But holy cow the ability to do something like move everybody in the company to different vocabulary types is powerful.
- j2kun 3y ago
  On the other hand, most weeks someone else breaks my system and I have to track down the culprit. Google's emphasis has always been to make things easy for library developers, at the expense of library clients. For people who value backwards compatibility over long timespans, Google's practices could be better.
- nonethewiser 3y ago
  I dont think its fair to classify code review and test coverage as “the google way.” Should evaluate more by the unique things google does or the things they specifically invented (not code review and testing). And of course volunteers working on open source projects have lower standards. Lets instead compare Google to companies which say “we arent google.”
  - UncleMeat 3y ago
    What I am describing is not code review and test coverage. What I am describing is the ability to run all of the tests for the entire company in one go so you can safely make absolutely massive changes to the codebase.
    - nonethewiser 3y ago
izacus 3y ago
> Cool. Could someone, maybe an ex-googler, comment on which parts of these work well and which don't? TBH most of this stuff is transferrable and even "common sense" in most of the companies you've worked for. Similarly how Google's SRE book is actually a very good collection of battle won experience on how ops can keep systems more reliable and running. The book is written in a way that you can easily throw away advice that you don't think useful.
opportune 3y ago
The “policies don’t scale well” section is inaccurate. There are plenty of policies floating around that don’t scale well, and plenty of migrations that are still forced on internal users rather than handled magically by the migrating team. The reality is that Google is such a big company most of these fly under the radar of whichever person actually enforces these policies, and it becomes a whole thing to escalate up to whoever enforced them, and then there’s potentially a political battle between whatever director or VP is in charge of the bad actors and the enforcer (ideally they get away with not allocating HC to the internal migration and amortize it across all their users, so that HC can work on flashier stuff). I think one reason Google has a proliferation of bureaucracy and red tape is that they do not “review” postmortem action items very formally. They are only reviewed as part of the larger incident postmortem review process and the tooling is way overengineered such that performing that review beyond a perfunctory once over isn’t easy to do. So you end up in a situation where “we need to do something” and whichever person handled the incident has to suggest a way to make sure it doesn’t happen again - the easiest of which is to introduce some CYA process. The other reason is that non-coding EMs introduce processes to show some kind of impact on their team. Also, the existence of the monorepo, global test runs, forced migrations, etc makes it so maintaining a mothballed project incurs some inherent engineering costs - IMO it’s a non-negligible reason Google kills products that could instead simply exist without changes. It also makes it so Google doesn’t really “version” software generally speaking.
summerlight 3y ago
> comment on which parts of these work well and which don't? I don't think there is a visible distinction between those parts that work/doesn't work. In fact, most of the cases each practice has pretty strong rationales. The problem is, when you take everything as whole, its cumulative complexity and cognitive overhead tends to go wild and almost no one can understand the whole stack when its original writers/maintainers leave the team. In fact, this might play a certain role of the Google graveyard narrative; it's not because its engineering culture is bad, but sometime its standard is too high for many cases so it's nearly impossible to keep it up for newcomers, especially when you have external pressures that you cannot ignore. Even if you make an eng team of 3~4 people for a small product, they'll likely suffer from tens of migration/deprecation/mandates over years.
underdeserver 3y ago
I left Google around six months ago. I worked in medium and small companies, currently at a startup with ~30 devs. I would say the vast majority of it works well, some you just don't need until you hit scale (here, scale in the number of developers). For example, policies work if you have <20 engineers, probably don't really work otherwise. Blaze/Bazel I miss a lot. Just wrangling the dependencies between shared packages is a mess (though we might just suck at configuring Poetry - at any rate it's not intuitive). Building and deploying is much more involved. Another thing I miss is code review the Google way. Google asks that you review within 24 hours, reviews by (the equivalent of a) commit and not by PR, and strongly advises to keep commits small; The GitHub PR workflow is terrible in comparison: 1) it nudges you into batching commits into large PRs 2) Is the PR message informative? Is each commit's? What about squash and merge - how many people edit that message? At Google part of the code review is reviewing the commit message. When you squash and merge, that's post approval, so you can't even do that. 3) Hidden conversations? What the actual fudge 4) How many comments have I not addressed yet? For that matter, how many PRs are waiting for my attention and when were they sent?
- sa46 3y ago
  Of all the Google dev tools, I miss Critique the most. GitHub is terrible at giving enough context to efficiently review a PR on a second or third pass. I think coupling commits with review progress was a mistake.
  - codeapprove 3y ago
    Shameless plug but if you're missing Critique and working on GitHub, try CodeApprove (https://codeapprove.com https://codeapprove.com) which brings as much of the Critique magic to GitHub as possible.
    - limonkufu 3y ago
      This seems very simiilar to Gitlab's MR. Does anyone know all of them to highlight their strengths and drawbacks?
    - 3y ago
patmcguire 3y ago
Readability is hit and miss. Very nice to have everything written to the same standard, it makes it much easier to navigate through any project. Downside is it's pretty rough for more peripheral teams or teams working in a language that's a small component of their product. I remember for one of DeepMind's big launches the interface was all in files ending in .notjs, presumably since they didn't have anyone on the team with Javascript readability. This was 5+ years ago, though, so some of the downsides may have been mitigated.
AlbertCory 3y ago
Personal story from ex-Googler: after getting exasperated at yet one more internal tool launched with great fanfare and almost no testing, let alone documentation, I suggested to the Internal Tools group that we have a contest for BEST internal tool. Not "worst" since that would be too hurtful. The hope was to recognize excellence, motivate people to be better, and maybe shame the people whose tools received no votes. This suggestion was summarily dismissed. There were, indeed, some truly excellent tools: Dremel comes to mind. And lots of tools that were nearly unusable.
FpUser 3y ago
I could not care less about what Google practices. They operate on enormous scale and have vastly different goals and values.
biogene 3y ago
>A lot of other companies get into trouble trying to cargo-cult what Google does when they are operating in very different environments wherein those practices aren't optimal. E.g. different levels of scale. Any prominent examples?

hiddencost 3y ago

Engineering culture has somewhat collapsed at Google. The things that made engineering great didn't really survive the last couple rounds of internal coups.

n0zmer 3y ago
I thought so too, but since then I moved over to Cloud and things are a LOT better.
- jupblb 3y ago
  Nice bait
- eddd-ddde 3y ago
  I always here about Cloud having the worst culture tho? Has that not been the case to you?
  - VirusNewbie 3y ago
    People work hard in cloud but there are no MBAs in sight. It’s all very technical work, often very bottom up driven. A lot of the overall goals of cloud are more ambitious than AWS offerings. Reliability is prized more than it is in other areas of Google as well, because customers are so technical and often notice. Not a place to coast, but I’d say most people do a solid 45 a week for those that want to get good reviews and get a fat bonus.
    - nonethewiser 3y ago
      > A lot of the overall goals of cloud are more ambitious than AWS offerings. In what sense?
  - opportune 3y ago
    Some teams in Cloud suck but the core engineering teams have some top talent and solve some very hard problems. Keep in mind Borg and Spanner are both “cloud”, but so are many field sales teams with an average tenure <2y
  - n0zmer 3y ago
    Not at all, it's a breath of fresh air. It's much less of a check buganizer, check email, write code, push CL cycle. Work is very project focused with high flexibility, my current team isn't even pushing to g3 and using different build systems entirely just because we wanted some more flexibility for example, and it doesn't matter as long as we are getting results. The problems are a lot more technical though and I don't see a lot of L3s being able to work in the environment as it requires a lot more intuition and experience. I usually work 45hours a week, but I don't mind it. Plus I'm 100% WFH here cause my management isn't dealing with in office bs.
- deleted 3y ago
  [deleted]
yowlingcat 3y ago
Interesting -- having not had any experience inside Google I'm having difficulty painting a picture, could you give an example or two of some of these internal coups?

NoMoreNicksLeft 3y ago

> Additionally, critics of Google may point out that their engineering culture may not be great on its own terms -- every time Google launches a new feature, people post links to the Google product graveyard. It is personally scary when they develop new products. What if it is a brilliant idea, one I cannot live without? If Google develops it, then I am looking at this stillborn thing, mewling for life when I know its horrible fate. The trouble here is that Google employees (and perhaps even its upper management) want to believe that they are a company which is an inventor of things. But they are not this at all. They are an advertisement company. Advertisement companies should not and do not want to invent things... inventions are worse than burdens, inventions are these weird alien objects that appear valuable but are quite expensive and do not help to sell ads at all. So they hawk the inventions like they were freaks in some carnival sideshow to move traffic past their billboards. Until the traffic dwindles (or until they get tired of it). And then they take it out back behind the woodshed and put an end to it.

outside1234 3y ago
The inventions are to keep the talent stream coming ... to work on ads. The inventions are the small tax they pay to pretend to candidates that they could work on inventions when the vast majority of them will be "allocated" to ads.
- fidotron 3y ago
  It's also Google licking the cookie. They maintain a moat around ads by doing just enough to threaten to destroy anyone that gets close to their ecosystem. Facebook survived because G+ product vision was so out of touch with reality and FB were not part of the anti-compete hiring nonsense so that they managed to poach a lot of good people.
- NoMoreNicksLeft 3y ago
  You're telling me this stupid, bizarre thing: that Google's major innovation was an HR process. That's fucked-in-the-head just enough that you've made me wonder if it's true.

jeffbee 3y ago

In my ex-Google experience, here are the stages of denial about something that Google does which is good but the industry doesn't yet embrace. Stage 1: "We're not Google, we don't need [[whatever]]"; Stage 2: Foreseeable disaster for which [[whatever]] was intended to address happens; Stage 3: Giant circus of P0/SEV0 action items while everyone assiduously ignores the [[whatever]]; Stage 4: Quiet accretion, over several years, of the [[whatever]] by people who understand it. And the [[whatever]] ranges from things that are obviously beneficial like pre-commit code review to other clear winners like multi-tenant machines, unit testing, user data encryption, etc etc. It is an extremely strange industry that fails to study and adopt the ways of their extremely successful competitors.

MarceColl 3y ago
The tools, design and manpower needed to build a skyscraper are different from those needed to build a 1-story wood house. It's not that the ones that build the wood house are failing to study and adopt the ways of their extremely successful competitors. Now, some of the things you say like unit-testing and user data encryption are ones that I've never seen associated with the "We're not google" mindset, so maybe people have started using that phrase for anythingnow
- jeffbee 3y ago
  I'm not really talking about artisanal 3-man software shops, I'm talking about mid-sized companies with thousands of engineers, who don't realize they are already larger than and facing the same problems as Google was when they started adopting these practices. And to be clear, rejecting something as proven as pre-commit code review is not only to reject the example of Google and many other very successful enterprises, but also to ignore decades of developer productivity knowledge before Google existed. It's almost like the fact that Google adopted a long-standing best practice makes modern engineers reflexively revolt against those best practices. This can only be seen as a structural advantage for Google.
  - dinvlad 3y ago
    Curious how pre-commit code review worked, could you please elaborate a bit?

drewg123 3y ago

"Readability" works terribly when your company is acquired and your team enters all at the same time. Google has (or had ~10 years ago), a thing called "readability" for each language, where in order to be allowed to commit code to the central "google3" repo, you needed to have written some large amount of code in that language, and needed to have a readability reviewer sign off on your code. The process is designed for slowly on-boarding junior people into a team, and introducing them to google coding style and practices. Eg, the senior, mentoring folks on the team do the reviews and bring the new person up to speed. I imagine it must work well in that context. However, this breaks down when your entire team is new. How do you find somebody to review the code? All several million lines of the product that was acquired? Especially when it is written in multiple languages. So we were basically locked out of the main corporate repo, unable to do anything productive. We finally figured out that there was a paved path with a git repo used by the kernel team (and android?) that had none of these hurdles, where we could put our code and get productive immediately.

jefftk 3y ago
The way it's supposed to work is that acquired teams get lots of support on integration, including readability. This helps your team get integrated into writing Google-style code. Not sure why that didn't work out in this case? (Left Google a year ago)
- badlucklottery 3y ago
  This. Someone dropped the ball. There's a form to get your corner of the codebase exempted from readability temporarily. This gives your team a quarter or two to build up readability.
  - Espressosaurus 3y ago
    A quarter or two isn't going to be enough for a drastic realignment of a large codebase. It's a start, but only a start.
    - 3y ago

marcyb5st 3y ago

DISCLAIMER 1: Current Googler here, but opinions are my own. DISCLAIMER 2: I think from a hands-on-keyboard SWE there is a lot of useful stuff. What you mentioned about Google culture of killing products and such I am not gonna talk about. I recommend chapters about testing first and foremost. Among all the codebases I saw (both OS and proprietary) Google tests are the most comprehensive and reliable. However, If you are in a startup-like environment you should pick and choose and not try to follow every single principle listed as they could sink your velocity drastically in the short run. Other interesting points (IMHO) are Monorepo, Build System, and Code Reviews. For the Monorepo I discover being a huge lover although I was skeptical. The sad thing is that it's a rather niche practice and tools like Git don't play ball very well (i.e. each time you pull you have to retrieve changes for all the codebase, even files you never saw/heard of managed by another team). I think there's no nice off-the-shelf offering for running monorepos out there. However, not having to fight with git submodules, library versions, ... is great. If the change I am submitting breaks something else in the company you are immediately aware and so can act accordingly (e.g. keep the old implementation alongside the new one and mark it as deprecated so the other team will get a warning next time they do anything). The build system is a bit more controversial. I learned to love blaze/bazel, but admittedly, the OS version is a bit messy to set up. Additionally, being so rigorous about the build rules felt like a massive chore at the beginning, but now I appreciate it a lot. I can instantly know the contacts of all the teams that use a build rule I declared and hence can be contacted to warn them about bugs, ... . I can create something experimental and have private visibility so only my team can use it and only later expose it to the wider world with just a one liner. Finally, the code review AKA Critique. Google has the best review tool I had the joy to use hands down. It's clear about what happens, at which stage is the review of a particular section/file and is focused on discussion. The evolution of each change is easy to follow along. These are things I really miss when using GitHub/GitLab PR view. The tooling is incredibly confusing to me. Luckily (I am not affiliated in any way) an ex-Googler (I believe) is working on an alternative that works with GitHub (https://codeapprove.com/ https://codeapprove.com/).

deleted 3y ago
[deleted]
nonethewiser 3y ago

UncleMeat 3y ago

In my opinion, the monorepo, global presubmit, testing culture and the beyonce rule (if you liked it then you should have put a test on it) are basically a superpower for infrastructure teams. Without these things it'd be utterly impossible for certain kinds of infra refactors to be done and many more would be very very painful. In the open source world I see a fair amount of "tests are always red, don't worry" and "we can never edit this interface because who knows who it breaks." These problems aren't intractable at Google. This approach does have its own set of challenges and I do suspect that the monorepo has contributed in some ways to Google's inability or refusal to maintain some older products. But holy cow the ability to do something like move everybody in the company to different vocabulary types is powerful.

j2kun 3y ago
On the other hand, most weeks someone else breaks my system and I have to track down the culprit. Google's emphasis has always been to make things easy for library developers, at the expense of library clients. For people who value backwards compatibility over long timespans, Google's practices could be better.
nonethewiser 3y ago
I dont think its fair to classify code review and test coverage as “the google way.” Should evaluate more by the unique things google does or the things they specifically invented (not code review and testing). And of course volunteers working on open source projects have lower standards. Lets instead compare Google to companies which say “we arent google.”
- UncleMeat 3y ago
  What I am describing is not code review and test coverage. What I am describing is the ability to run all of the tests for the entire company in one go so you can safely make absolutely massive changes to the codebase.
  - nonethewiser 3y ago

izacus 3y ago

> Cool. Could someone, maybe an ex-googler, comment on which parts of these work well and which don't? TBH most of this stuff is transferrable and even "common sense" in most of the companies you've worked for. Similarly how Google's SRE book is actually a very good collection of battle won experience on how ops can keep systems more reliable and running. The book is written in a way that you can easily throw away advice that you don't think useful.

opportune 3y ago

The “policies don’t scale well” section is inaccurate. There are plenty of policies floating around that don’t scale well, and plenty of migrations that are still forced on internal users rather than handled magically by the migrating team. The reality is that Google is such a big company most of these fly under the radar of whichever person actually enforces these policies, and it becomes a whole thing to escalate up to whoever enforced them, and then there’s potentially a political battle between whatever director or VP is in charge of the bad actors and the enforcer (ideally they get away with not allocating HC to the internal migration and amortize it across all their users, so that HC can work on flashier stuff). I think one reason Google has a proliferation of bureaucracy and red tape is that they do not “review” postmortem action items very formally. They are only reviewed as part of the larger incident postmortem review process and the tooling is way overengineered such that performing that review beyond a perfunctory once over isn’t easy to do. So you end up in a situation where “we need to do something” and whichever person handled the incident has to suggest a way to make sure it doesn’t happen again - the easiest of which is to introduce some CYA process. The other reason is that non-coding EMs introduce processes to show some kind of impact on their team. Also, the existence of the monorepo, global test runs, forced migrations, etc makes it so maintaining a mothballed project incurs some inherent engineering costs - IMO it’s a non-negligible reason Google kills products that could instead simply exist without changes. It also makes it so Google doesn’t really “version” software generally speaking.

summerlight 3y ago

> comment on which parts of these work well and which don't? I don't think there is a visible distinction between those parts that work/doesn't work. In fact, most of the cases each practice has pretty strong rationales. The problem is, when you take everything as whole, its cumulative complexity and cognitive overhead tends to go wild and almost no one can understand the whole stack when its original writers/maintainers leave the team. In fact, this might play a certain role of the Google graveyard narrative; it's not because its engineering culture is bad, but sometime its standard is too high for many cases so it's nearly impossible to keep it up for newcomers, especially when you have external pressures that you cannot ignore. Even if you make an eng team of 3~4 people for a small product, they'll likely suffer from tens of migration/deprecation/mandates over years.

underdeserver 3y ago

I left Google around six months ago. I worked in medium and small companies, currently at a startup with ~30 devs. I would say the vast majority of it works well, some you just don't need until you hit scale (here, scale in the number of developers). For example, policies work if you have <20 engineers, probably don't really work otherwise. Blaze/Bazel I miss a lot. Just wrangling the dependencies between shared packages is a mess (though we might just suck at configuring Poetry - at any rate it's not intuitive). Building and deploying is much more involved. Another thing I miss is code review the Google way. Google asks that you review within 24 hours, reviews by (the equivalent of a) commit and not by PR, and strongly advises to keep commits small; The GitHub PR workflow is terrible in comparison: 1) it nudges you into batching commits into large PRs 2) Is the PR message informative? Is each commit's? What about squash and merge - how many people edit that message? At Google part of the code review is reviewing the commit message. When you squash and merge, that's post approval, so you can't even do that. 3) Hidden conversations? What the actual fudge 4) How many comments have I not addressed yet? For that matter, how many PRs are waiting for my attention and when were they sent?

sa46 3y ago
Of all the Google dev tools, I miss Critique the most. GitHub is terrible at giving enough context to efficiently review a PR on a second or third pass. I think coupling commits with review progress was a mistake.
- codeapprove 3y ago
  Shameless plug but if you're missing Critique and working on GitHub, try CodeApprove (https://codeapprove.com https://codeapprove.com) which brings as much of the Critique magic to GitHub as possible.
  - limonkufu 3y ago
    This seems very simiilar to Gitlab's MR. Does anyone know all of them to highlight their strengths and drawbacks?
  - 3y ago

patmcguire 3y ago

Readability is hit and miss. Very nice to have everything written to the same standard, it makes it much easier to navigate through any project. Downside is it's pretty rough for more peripheral teams or teams working in a language that's a small component of their product. I remember for one of DeepMind's big launches the interface was all in files ending in .notjs, presumably since they didn't have anyone on the team with Javascript readability. This was 5+ years ago, though, so some of the downsides may have been mitigated.

AlbertCory 3y ago

Personal story from ex-Googler: after getting exasperated at yet one more internal tool launched with great fanfare and almost no testing, let alone documentation, I suggested to the Internal Tools group that we have a contest for BEST internal tool. Not "worst" since that would be too hurtful. The hope was to recognize excellence, motivate people to be better, and maybe shame the people whose tools received no votes. This suggestion was summarily dismissed. There were, indeed, some truly excellent tools: Dremel comes to mind. And lots of tools that were nearly unusable.

FpUser 3y ago

I could not care less about what Google practices. They operate on enormous scale and have vastly different goals and values.

biogene 3y ago

>A lot of other companies get into trouble trying to cargo-cult what Google does when they are operating in very different environments wherein those practices aren't optimal. E.g. different levels of scale. Any prominent examples?