3 ms·

You're not wrong. Google gets to sidestep a lot of problems other lower-margin companies have because they've spent their whole history being near market-leader

You're not wrong. Google gets to sidestep a lot of problems other lower-margin companies have because they've spent their whole history being near market-leader on search and ads. Things like "deadlines" were foreign to the company for a long time; when you're already the pack leader, you release new features when you feel like it, not to play catch-up with competitors. You can see them struggle in spaces where that's not true (Cloud, for example). There is one underlying technology special sauce they refined to an art form, which is building a reliable system on incredibly unreliable components. This comes from their inception when the two guys building out the company knew how to write code but not build custom machines; their initial attempts at racks (hard drives and motherboards mounted to flexible plywood) are hilarious-looking by modern standards and broke down all the time, so they had to learn how to write software that was fault-tolerant at every link in the chain. That resulted in building an infrastructure that was extremely supportive of experimentation (if the system can survive half of it disappearing, it can survive a software bug crashing half the machines), and the rest of the shape of the company kind of flows from that. It is worth repeating a lot that most companies do not operate at the scale or constraints Google does, and their approach to engineering is emphatically not one-size-fits-all.

lokar 3y ago
I think people forget that the standard back then in the industry was very high margin servers with redundant everything (psu, network/nic, disks, etc). Going to (eventually) mostly dependable simple servers with no redundancy was a big leap.
- shadowgovt 3y ago
  There were a couple places in Google's internal infrastructure where they had picked up traditional "big iron" architectures (mostly by acquisitions), and it was funny to observe the sheer plane between their regular best practices and those systems (when they hadn't yet been replaced). One high-ticket item relied on a huge Oracle database. Every other aspect of that product was patch-on-the-fly, silent-release new versions, forward-and-backwards compatibility of software... But the whole product had regular scheduled outages on the weekends because the Oracle part could not be patched on the fly, had a rigid schema baked into its relational architecture, and had to be brought offline for updates and migrations. It was like everyone was zipping around in racecars and then they all had to pause for Grandpa to cross the street.

lokar 3y ago

I think people forget that the standard back then in the industry was very high margin servers with redundant everything (psu, network/nic, disks, etc). Going to (eventually) mostly dependable simple servers with no redundancy was a big leap.

shadowgovt 3y ago
There were a couple places in Google's internal infrastructure where they had picked up traditional "big iron" architectures (mostly by acquisitions), and it was funny to observe the sheer plane between their regular best practices and those systems (when they hadn't yet been replaced). One high-ticket item relied on a huge Oracle database. Every other aspect of that product was patch-on-the-fly, silent-release new versions, forward-and-backwards compatibility of software... But the whole product had regular scheduled outages on the weekends because the Oracle part could not be patched on the fly, had a rigid schema baked into its relational architecture, and had to be brought offline for updates and migrations. It was like everyone was zipping around in racecars and then they all had to pause for Grandpa to cross the street.