5 ms·

I’ve worked in three separate companies that had each hired a ton of ex Google engineers in their early years, and the systems they have are all identical to ea

by mlthoughts2018 8y ago

I’ve worked in three separate companies that had each hired a ton of ex Google engineers in their early years, and the systems they have are all identical to each other and seem like a poorly recalled-from-memory version of these types of patterns. It’s like a strange simulacrum of some bazel-like build tools but not as mature as bazel but also can’t be swapped with real bazel due to tech debt issues, and same with canary deployments, monorepo layouts, etc. It’s become something that I actively seek out information about when I interview for new jobs so I can actively avoid all these Google-but-not-quite-with-enough-resources-to-pull-it-off antipatterns. Worse yet is that in many of these shops, those early employees who came from Google just stayed long enough to mutate the whole monorepo / homebrewed bazel clone / canary deploys mess into a pile of unmaintainable crap that couldn’t be migrated away from without huge business risks, and thenthey mostly all quit and a lot of them even went back to Google! I call it the Google Borg Syndrome. Same crap monorepo ideas, same crap canary ideas, same crap bazel-like build system ideas just spread like kudzu. It really shows how bad the ideas themselves are, and how if it wasn’t for a giant money and labor-hours faucet that Google can shoot at these systems, they would be unmasked as deeply poor ways of solving engineering problems.

londons_explore 8y ago
>they would be unmasked as deeply poor ways of solving engineering problems. The best technical solution to problems can depend a lot on company size. These people are using a big-company solution in a small-company. And it nearly works, but not quite. Big companies reinvent pretty much everything internally - You won't find Chef, Jenkins, Slack, Apache, Nginx, or any of that stuff at Google. Yet a small startup would be stupid not to use tooling developed and maintained by someone else almost everywhere. In turn, the best solution to a problem depends on which tools you have at your disposal.
- mlthoughts2018 8y ago
  I agree to a large extent. But specifically in the case of - using bazel instead of make - using a monorepo ever - using Google-style canary deployment ever I think they are just nearly totally objectively bad ways to solve problems whether at Google scale or otherwise, with obvious alternatives that are strictly dominating in the sense of being unilaterally better in all use cases. These techniques have to be kept afloat with the money faucet. I guess I can agree that if you have enough money that you can offer pay packages that cause people to be willing to endure the hardship of propping bad systems up, then you don’t have to care about what systems are effective. For example, you could shoot lots of holes in a boat and then hire a full time staff on $500k / year to scoop water out fast enough that the boat seems to operate just fine. Then get the ACM to write a big publicity piece about how hole-filled boats are the best design and anyone not shooting holes in their boat is Doing It Wrong. This is not even exaggeration to offer as an analogy to Google’s combo use of monorepos + bazel.
  - londons_explore 8y ago
    Bazel is designed for big projects and distributed compilation. Unless your company has >500 engineers, it isn't going to make sense. I like monorepos - and in my experience you don't need special tooling for them until you have thousands of engineers. I just don't really see the point in splitting all my business logic into 50 different repos all of which have 25 branches and then I'm having versioning, dependency, and incompatibility hell between them. One repo, one 'master' branch, lots of tests, and as soon as all tests pass, auto-deploy! The 'canary deployment server', complete with webUI I think is a mistake, even for Google. If they were using kubernetes, it would have a 'Deployment' object, and canarying should be a feature of that, just like rolling updates are. Actually how that works is an implementation detail of kubernetes and is abstracted away.
    - gerash 8y ago
      I've used Unix make for personal projects (during and after school) and bazel at work. "bazel" is more about type checking the build artifacts, caching and guarantees about hermetic and reproducible builds. The distributed compilation is just supported but not a requirement IIUC. idk if it's the best tool but definitely an upgrade over "make". Whereas "make" looks like an old pseudo scripting utility which has too many degrees of freedom with not much guarantee and is fragile as a result. I think those who learned its quirks love it but I don't. I've also liked mono repos at work as long as their operations are not sluggish. They immensely help global refactoring, static analysis and reduce the amount of merge conflicts.
  - berdario 8y ago
    - I'm not sold on Bazel (I'm more of a Nix person), but hermetic builds are obviously a good idea - You don't need the added complexity of having to manage tooling to track dependencies across multiple repos. A single repo is just fine. You'll obviously want to split off repositories when you want to contribute stuff as open source, and in other cases http://danluu.com/monorepo/ http://danluu.com/monorepo/ - what you describe as "Google-style canary" is not actually Google-style or recommended by Google See the SRE workbook: https://landing.google.com/sre/workbook/chapters/canarying-releases/#use-a-gradual-canary-for-better-metric-selection https://landing.google.com/sre/workbook/chapters/canarying-r... (obviously, Google is big, so different teams might do things differently)
hnzix 8y ago
> spread like kudzu. I see this cargo-culting all the time, it spreads via breathless blog posts as well. People applying patterns from FAANG totally out of context because it sounds cool. Google uses Kubernetes, therefore I should hand-roll my own K8s infrastructure to host my 1000 user CRUD app. Oooh and let's make it a SPA with 100 microservices because Facebook does that so it must be a good idea.

londons_explore 8y ago

>they would be unmasked as deeply poor ways of solving engineering problems. The best technical solution to problems can depend a lot on company size. These people are using a big-company solution in a small-company. And it nearly works, but not quite. Big companies reinvent pretty much everything internally - You won't find Chef, Jenkins, Slack, Apache, Nginx, or any of that stuff at Google. Yet a small startup would be stupid not to use tooling developed and maintained by someone else almost everywhere. In turn, the best solution to a problem depends on which tools you have at your disposal.

mlthoughts2018 8y ago
I agree to a large extent. But specifically in the case of - using bazel instead of make - using a monorepo ever - using Google-style canary deployment ever I think they are just nearly totally objectively bad ways to solve problems whether at Google scale or otherwise, with obvious alternatives that are strictly dominating in the sense of being unilaterally better in all use cases. These techniques have to be kept afloat with the money faucet. I guess I can agree that if you have enough money that you can offer pay packages that cause people to be willing to endure the hardship of propping bad systems up, then you don’t have to care about what systems are effective. For example, you could shoot lots of holes in a boat and then hire a full time staff on $500k / year to scoop water out fast enough that the boat seems to operate just fine. Then get the ACM to write a big publicity piece about how hole-filled boats are the best design and anyone not shooting holes in their boat is Doing It Wrong. This is not even exaggeration to offer as an analogy to Google’s combo use of monorepos + bazel.
- londons_explore 8y ago
  Bazel is designed for big projects and distributed compilation. Unless your company has >500 engineers, it isn't going to make sense. I like monorepos - and in my experience you don't need special tooling for them until you have thousands of engineers. I just don't really see the point in splitting all my business logic into 50 different repos all of which have 25 branches and then I'm having versioning, dependency, and incompatibility hell between them. One repo, one 'master' branch, lots of tests, and as soon as all tests pass, auto-deploy! The 'canary deployment server', complete with webUI I think is a mistake, even for Google. If they were using kubernetes, it would have a 'Deployment' object, and canarying should be a feature of that, just like rolling updates are. Actually how that works is an implementation detail of kubernetes and is abstracted away.
  - gerash 8y ago
    I've used Unix make for personal projects (during and after school) and bazel at work. "bazel" is more about type checking the build artifacts, caching and guarantees about hermetic and reproducible builds. The distributed compilation is just supported but not a requirement IIUC. idk if it's the best tool but definitely an upgrade over "make". Whereas "make" looks like an old pseudo scripting utility which has too many degrees of freedom with not much guarantee and is fragile as a result. I think those who learned its quirks love it but I don't. I've also liked mono repos at work as long as their operations are not sluggish. They immensely help global refactoring, static analysis and reduce the amount of merge conflicts.
- berdario 8y ago
  - I'm not sold on Bazel (I'm more of a Nix person), but hermetic builds are obviously a good idea - You don't need the added complexity of having to manage tooling to track dependencies across multiple repos. A single repo is just fine. You'll obviously want to split off repositories when you want to contribute stuff as open source, and in other cases http://danluu.com/monorepo/ http://danluu.com/monorepo/ - what you describe as "Google-style canary" is not actually Google-style or recommended by Google See the SRE workbook: https://landing.google.com/sre/workbook/chapters/canarying-releases/#use-a-gradual-canary-for-better-metric-selection https://landing.google.com/sre/workbook/chapters/canarying-r... (obviously, Google is big, so different teams might do things differently)

hnzix 8y ago

> spread like kudzu. I see this cargo-culting all the time, it spreads via breathless blog posts as well. People applying patterns from FAANG totally out of context because it sounds cool. Google uses Kubernetes, therefore I should hand-roll my own K8s infrastructure to host my 1000 user CRUD app. Oooh and let's make it a SPA with 100 microservices because Facebook does that so it must be a good idea.