4 ms·
In my ex-Google experience, here are the stages of denial about something that Google does which is good but the industry doesn't yet embrace. Stage 1: "We're
by jeffbee 3y ago
In my ex-Google experience, here are the stages of denial about something that Google does which is good but the industry doesn't yet embrace.
Stage 1: "We're not Google, we don't need [[whatever]]";
Stage 2: Foreseeable disaster for which [[whatever]] was intended to address happens;
Stage 3: Giant circus of P0/SEV0 action items while everyone assiduously ignores the [[whatever]];
Stage 4: Quiet accretion, over several years, of the [[whatever]] by people who understand it.
And the [[whatever]] ranges from things that are obviously beneficial like pre-commit code review to other clear winners like multi-tenant machines, unit testing, user data encryption, etc etc. It is an extremely strange industry that fails to study and adopt the ways of their extremely successful competitors.
- MarceColl 3y agoThe tools, design and manpower needed to build a skyscraper are different from those needed to build a 1-story wood house. It's not that the ones that build the wood house are failing to study and adopt the ways of their extremely successful competitors. Now, some of the things you say like unit-testing and user data encryption are ones that I've never seen associated with the "We're not google" mindset, so maybe people have started using that phrase for anythingnow
- jeffbee 3y agoI'm not really talking about artisanal 3-man software shops, I'm talking about mid-sized companies with thousands of engineers, who don't realize they are already larger than and facing the same problems as Google was when they started adopting these practices. And to be clear, rejecting something as proven as pre-commit code review is not only to reject the example of Google and many other very successful enterprises, but also to ignore decades of developer productivity knowledge before Google existed. It's almost like the fact that Google adopted a long-standing best practice makes modern engineers reflexively revolt against those best practices. This can only be seen as a structural advantage for Google.
- dinvlad 3y agoCurious how pre-commit code review worked, could you please elaborate a bit?
- jeffbee 3y agoEvery change is reviewed by someone other than the author before it lands in the repo. At google they take this a bit further. Every change has to have been either written or reviewed by a designated owner of the code (designated by subdirectory) and one of the participants must be a qualified user of the languages used in the change ("readability"). And they have technical measures in place to ensure that programs running in production descend exclusively from reviewed and committed code. Pre-commit review is common but not universal in the industry. Some shops practice post-commit review or no review. Some believe review consists entirely of quibbling.
- dinvlad 3y agoOh, I see - by "pre-commit" it doesn't really imply it in "Git commit" sense - the change is still propagated to others (presumably by committing it as a sort of "draft"), it's just not committed to the mainline - is that correct? I'm very familiar with CR at other companies, but tbh since most use Git, I wouldn't call that "pre-commit" but "pre-merge", if you will - unless I misunderstood and it really is pre-commit at Google (i.e. the changes are not even _committed_ to the repository - and then I'm confused once again at what exactly that means..)
- joshuamorton 3y agoGoogle (for $reasons) doesn't do long lives code branches and doesn't use git, at least for the main repo. So in that context every commit is reviewed pre-commit, but you'd do the same workflow elsewhere with trunk-based development, small pull requests, and CI and automated and human review of all PRs before they're merged.
- 3y ago
- sumtechguy 3y ago"were not google" is usually good for things where people are using cargo cult. I saw at one company that went open floor plan because google did it. No one was happy about that. Retention became very low and everyone bailed out. Emulating google does not fix process and management issues. As what may be at google for a good reason may be an utter failure at another company. There are things all shops can adopt that google does that would help them. But many of the ones I have seen adopted were little more than showy garbage instead of the things that would actually help. Also sometimes you just need a simple tool to get something done. As engineers we like to build things so sometimes we make it way more complex than it really needs to be. For someone like google that may just be fine to do. For others a minimum viable product may be in order. Do not worry about optimizing for the 3 million user per day case when you have 10 total users a month. Add logging and keep an eye on it. Then worry if you need to scale. As building good scale takes time and thought. Many times you do not need that at all. As your company/group grows you will take on more and more of the things 'google does' because you will need to, or you will go nuts chasing everything. You could probably even make stages out of the different times to do/evaluate things. To do it early could actually harm what you are doing. You need to evaluate what you are doing and why. Just copying someone else does not always lead to a good outcome and you could be wasting effort when you could be making product.
- MarceColl 3y agoI think you wanted to reply to the same person I replied to? Since I'm saying basically the same thing you do I believe
- robertlagrant 3y ago> "were [sic] not google" is usually good for things where people are using cargo cult It's no different to cargo culting. That should not be the reason for not doing something, any more than the opposite should be a reason for doing it. Just see if the practice makes sense in your context and decide that way.
- marcosdumay 3y ago"We are not Google" is the answer to "I saw on that blog here that we should do X", "everybody is doing X, we should too", or "you have to follow this good practice here" where the practice is only "good" because it's hyped. Those kinds of demand happen exactly because they saw them at Google, and an outright refusal is exactly how they should be dealt with. Once the unreasonable person is cut out, you can look at your context and decide what's the best way to solve the problem.
- lesuorac 3y agoI just wish more places would adopt `third_party`; I would also love reproducible builds but I'll settle on third_party.
- raldi 3y agoOff-topic for this thread, but one of the most poignant quips I remember about Google culture was that the performance-review process was really good at rewarding hard, challenging work that didn't produce much value and not very good at recognizing work that produced lots of value but was not astoundingly difficult. I think you were the one who first noted this.
- jeffbee 3y agoI miss you too, raldi.
- H8crilA 3y agoThe performance review process has a small impact on salary. The promo process is not based on value or difficulty, but on the size of the organization that one is running. This is also true for higher level ICs, except they do not manage people, but rather manage/lead projects (which then have a certain amount of people involved). Here's a rough breakdown: - L4 -> 1 person - L5 -> 1-3 people - L6 -> ~7 people - L7 -> ~25 people - L8 -> ~70 people The approximate 3x difference between the levels is also found in other organizations, for example in armies: division ~= 3 brigades, brigade ~= 3 battalions, battalion ~= 3 companies, company ~= 3 platoons. Misunderstanding this is the source of almost all frustration with the promo process. This process is designed to build and expand the organization, not reward awesomeness. There are of course deviations from the simple schema I listed here, but this is the hard reference point.
- pclmulqdq 3y agoPerf feeds into promotions, which are the real way to raise your long-term salary (both inside and outside of Google).
- fcpk 3y agoThis is a terrible process unfortunatelly. Raising the salary should be related to the usefulness of the person to the company, and not the breadth/impact of their work. This leads to terrible things like gaming the system to get high impact/leadership projects to get raises which comes with huge side effects, like projects getting abandonned fast, being deprecated in favor of new shiny promo-bringing things. But this is not just a Google specific issue, and it is quite widespread in the industry. Google however suffered from this especially due to its obsessive culture of pay-for-perf and by ignoring simple facts: - inflation means that your salaries should raise regardless of performance. if you only tail the market by adjusting salaries only if the market changes, then you are 1 year late(at least). This isn't a problem in an economy with low inflation, but is a huge problem in one with much higher inflation. - there is a significant number of people needed to maintain projects that won't show large impact. Those people need to be at the very least recognized and compensated. - making new products is great, but it requires huge amounts of ressources to do at Google scale from the get-go. A wider strategy is much needed, which Google obviously lacked for almost a decade.
- claytongulick 3y agoOr, alternately phrased: As a company grows and matures, their software development processes evolve to meet the business needs.
- dustingetz 3y agolabel the tradeoffs?
- hn_throwaway_99 3y agoStrong disagree. In my experience, this is not commonly why competitors don't adopt Google's practices. The main reasons I've seen are: 1. Money. Google essentially has a giant, gargantuan, enormous, bottomless pit of money to build a lot of this tooling (and also to take the risk if something ends up not working out). I think you might be able to say that other companies are just being short sighted if they don't implement some of these things up front, and that may be true, but (a) that's pretty much human nature, and (b) given that very few other companies have a bottomless pit of money like Google, that may just end up being the right decision (i.e. survive now and deal with the pain later). 2. Talent. This is closely related to #1, but few other companies have the engineering talent that Google does. If there is one thing I've seen with my experience with ex-Googlers is that most of them are fast coders. So when you go to your boss and say "I'd like to implement engineering/tech-debt improvement XYZ", at other companies it's a harder decision if (on average) it would take 9 months to implement vs. 2 or 3. 3. Related to both of the above, but your 4th bullet point, "Quiet accretion, over several years, of the [[whatever]] by people who understand it.", is actually other companies just waiting for more evidence to see what "shakes out" as the industry-standard, optimal way to do things. 4. Finally, your stage 1, "We're not Google, we don't need [[whatever]]" is actually true in tons of cases. Many of Google's processes are there to handle enormous scale, both in terms of their application/data capacity, as well as the sheer number of engineers they need to coordinate. Very, very, very few companies will ever hit Google's scale.
- jeffbee 3y agoJust discussing your point #1, I hear this but what I see is that the companies I have direct experience with spend much more and move more slowly with their we-are-not-Google hacks. People move fast and break things into a corner where their entire project is a haunted graveyard with no test and no comments, that has never been reviewed, and at that point nobody is allowed to change anything.
- hn_throwaway_99 3y agoPerhaps, but to me everything you put in your comment above just sounds like bad engineering practices in general and not something particularly related to Google processes. E.g. things like "do feature work on a branch and then code review/run PR checks before merge", "have unit test coverage (being a hard balance to judge what is sufficient coverage)", "have useful comments" - absolutely none of these things I associate with "Google engineering practices", and many of them definitely predate things that were specifically done at Google. Things I think of when I think about Google practices are things like ensuring data is infinitely horizontally scalable, monorepos, etc. Those things are all scale-specific.
- yibg 3y agoThere is risk of selection bias here. The companies that runs into [[whatever]] are the ones that made it far enough to have run into it. What you're not seeing are all the companies that tried to do what google does at scale, built a complex code base that doesn't serve it's customers needs and can innovate fast enough and are now dead.
- jimmaswell 3y ago> pre-commit code review Unless you're referring to automated precommit hooks, this sounds baffling. What's wrong with reviewing pull requests? What if I want to push a WIP while I switch to another branch, I still need a review? Is the final PR reviewed again at the end?
- jeffbee 3y agoThat's a git user's perspective and Google doesn't use git or anything analogous. Under their system, and generally under Perforce, it is never necessary to "push a WIP" because your client just contains whatever edits it contains. You never need to manually checkpoint, stash, or commit. People with multiple changes in flight will usually use two different clients, one for each change, although that is not strictly mandatory and in the perforce model you can have disjoint sets of files in multiple changes in the same client. Anyway, TL;DR, the problems you suggest are git-specific and one solution to them is not using git.
- ericye16 3y agoWhat they mean is code review prior to merging into what you'd call main or trunk or master or release, not for committing your WIP changes or whatever (unless you want those merged at that stage).
- deanCommie 3y agoI'm sorry are you saying Google invented multi-tenant services, unit testing, or user data encryption? I'll give you "pushed the WEB industry to have transport-layer encryption for the entire industry by default". I'll even give you "code reviews". But not the first 3.
- cornel_io 3y agoReadability doesn't help Google or anyone else, it's a pure "inmates running the asylum" artifact.