3 ms·
Just discussing your point #1, I hear this but what I see is that the companies I have direct experience with spend much more and move more slowly with their we
by jeffbee 3y ago
Just discussing your point #1, I hear this but what I see is that the companies I have direct experience with spend much more and move more slowly with their we-are-not-Google hacks. People move fast and break things into a corner where their entire project is a haunted graveyard with no test and no comments, that has never been reviewed, and at that point nobody is allowed to change anything.
- hn_throwaway_99 3y agoPerhaps, but to me everything you put in your comment above just sounds like bad engineering practices in general and not something particularly related to Google processes. E.g. things like "do feature work on a branch and then code review/run PR checks before merge", "have unit test coverage (being a hard balance to judge what is sufficient coverage)", "have useful comments" - absolutely none of these things I associate with "Google engineering practices", and many of them definitely predate things that were specifically done at Google. Things I think of when I think about Google practices are things like ensuring data is infinitely horizontally scalable, monorepos, etc. Those things are all scale-specific.
- nostrademons 3y agoMonorepos actually work better at small scale than at Google scale. I think it's nuts that individual startup founders actually consider microservices; if you are validating out a software product idea, write a computer program, the very simplest one possible, to prove that you can do it and get the general shape of the architecture before you start dividing it into microservices. I usually see the pressure to split into microservices appear around 20 engineers, just as your single repo is starting to get unwieldy. Knowing that the big companies use a monorepo is pretty important information here, because it may prompt you to invest in tooling to make that one repo less unwieldy rather than splitting into many small repos that will be very difficult if not impossible to merge back together again. Google doesn't actually plan for infinite horizontal scalability in data. The framing I've found most useful is [Jeff] Dean's Law: "Plan for a 10x increase in scale, but accept that sometime before you reach 100x, you will have to rewrite the system with a different architecture." The reason for this is shifting bottlenecks: as the system gets larger, different aspects become the bottleneck to future scalability, and each time the bottleneck changes you usually need a different architecture. But by planning for an order of magnitude growth, you ensure that you're not artificially introducing bottlenecks, and that you have enough headroom to actually discover the new bottleneck.
- hn_throwaway_99 3y agoRe: monorepos, I think we're talking about 2 different things. I usually hear the term "monorepo" discussed in the context of how it is practiced at places like Google and Facebook: having the code for all the company's services (micro or not) stored in a single source control repository. A monorepo really doesn't have anything to do with how code components are deployed - your comment seemed to be contrasting a monolith architecture with a microservices one.
- nostrademons 3y agoI was fast & loose with terminology, but I'm thinking of the organizations where every binary and every library is its own Github repository, and you make copious use of git submodule to build anything. I think that's the same thing you're talking about, right? It's impractical (particularly when the project is young) for the same reason having separate binaries is impractical: it makes it very difficult to do refactorings that cut across repos while still keeping atomic checkouts and rollbacks.
- hn_throwaway_99 3y agoYep, we're talking about similar things but in slightly different ways: 1. When you're small enough, and only have a few teams and/or deployed binaries, you keep everything in a single repo. 2. As you grow with more teams and more products/binaries, often companies will split into having separate repos per team/product, and then use some sort of dependency management tooling (either something like the git submodules you discuss, or a private package repository, e.g GitHub's Package Registry, Nexus, or Artifactory). I totally agree with you that a lot of companies do this prematurely. 3. What distinguishes "monorepos", in my opinion, is that this was an innovation at either Google or Facebook I think (not sure who was first), where they realized exactly what you point out - it makes it really hard to do refactorings across lots of dependent projects in different repos. So they decided to keep everything in a single source control repository, with a single commit history. But in order to do that and have things be sane with lots of teams and thousands of developers, they needed to invest a ton in custom tooling to make these giant monorepos usable by individual developers and teams. E.g Facebook has Sapling, Google has Piper, and there are open source tools like Lerna for JS monorepos. So, in my experience, just having everything in a single repository but without any special tooling (because you're small enough to not need it yet) is just a repo. Monorepo IMO implies that you've grown to the point where it's difficult to keep everything in a single repo without special developer workflow support tools. All that said, I definitely agree with your main point - a lot of companies can just keep everything in a single repository a lot longer than they think they can, even without special tooling beyond some separate build targets.
- ryandrake 3y agoYea--I've found it surprisingly difficult to get plain vanilla medium-sized companies to adopt obvious, time-tested best practices. I'm not talking about "Google engineering practices" but basic table stakes practices like using source control and a bug tracker. So-called "Joel Test"[1] items. The most common excuses are: "We don't have time/money to do infrastructure/process, we need to write shipping code!" and the usual "We've always done it this way". 1: https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-steps-to-better-code/ https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...
- hn_throwaway_99 3y agoIt is honestly baffling, totally baffling to me that in this day and age there are companies that don't use source control or a bug tracker (but trust me, I believe you). With these companies I think it's just best to walk the other way. Some of them may state that "yeah, we know we need to improve our engineering practices, so we're willing to learn" but they usually just have so much of a different mindset of what it takes to actually run a software company that it's just a waste of time. There are many more companies that have a modicum of understanding about what they're doing.
- OkayPhysicist 3y agoThese rarely are "software companies". They're companies in other industries, that happen to need some software. Sometimes it's a pretty plumb gig: Good, but not great, pay, often in relatively lower cost-of-living areas, a relatively light workload, a good amount of autonomy as one of a maybe a handful of software devs, and in their blindness to good practices (like source control), they're also untouched by common bad practices in software, like whatever bastardized version of agile/scrum that your bosses heard through an extremely lossy telephone game. But there's also the bad: Software isn't the company's focus, so you aren't the company's focus. That means no "Senior FAANG" salaries, no best practices to keep things sane, and often you find yourself working on a codebase that was originally hacked together in a week by a chemist who may or may not have been deliberating huffing reagent fumes.
- vbezhenar 3y agoTests, comments and code reviews is not something unique to Google. It's a commonly accepted practice. There might be some dark corners, just like there are people who perform version controlling with ctrl+c, ctrl+v technique, but it's not a norm. I don't think that many people would argue against basic software development rules. However being Google is much more than writing tests and doing code reviews. Being Google means having a team which writes source control management system for you.
- majesticglue 3y agoclearly there is a missing link in your experience of companies where you don't have control over a lot of the things you think is needed and companies want to push a product as fast as possible and cut as many corners as possible. More and more companies will be like this to cut costs
- choppaface 3y agoOne conspicuous omission in the ex-google is reflection on killed products like Google Wave, Plus, Glass etc etc etc .. for many of the [whatever] was the gross imbalance between Eng owning the product but ignoring the userbase. What ex-googlers often fail to grapple with is the product lifecycle (how short it may be) and the value of having diversity in the loop of product testing. Google is designed to be a safe place to focus, and that’s not what the real world is like outside the plex.
- nostrademons 3y agoGoogle's penchant for killing promising products is 100% the result of poor incentives. People are incentivized for launching challenging projects, but they are generally not responsible for the bottom line (which is going to be dwarfed by Search Ads revenue anyway) or for user happiness & brand loyalty (which is challenging to measure). As a result, lots of promising and exciting products are brought to market and then killed, as the easiest way to bring new products to market is to cannibalize the stuff your predecessors did and show how great your alternative is instead.
- hn_throwaway_99 3y agoI'm not sure why your comment was previously downvoted. I've often heard, and it's not hard to find these comments from ex-Googlers on HN, that Google's "promotion-oriented development" is one of their biggest factors in some of their cultural shortcomings. That is, launching a big new product is seen as one of the best ways to get promoted, while working on the little nits (which in my experience, especially with some of Google's enterprise products, can languish for years, even though they can be really important but "boring" issues to fix) is not seen as high-value work.
- jeffbee 3y agoActually I think it is never-Googlers who have the wrong perspective here. The fact that Google constantly produces and destroys products demonstrates that it is extremely easy for that company to churn out code, and validates their software development methodology. It's incredibly easy to just dash off a product building on their gigantic foundation of source code, infrastructure, and launch process. The fact that Plus and Glass got canceled and Wallet has been canceled sixteen different times is merely a consequence of the fact that leadership and product is often led by imbeciles. That's an organizational problem and I hope nobody is out there cargo-culting Google's org (even though I know they are, with OKRs and Perf being widely copied).