3 ms·

I think you oversimplify a lot and miss many real world things that a Google sized org has to deal with. Machine maintenance - someone has to coordinate update

I think you oversimplify a lot and miss many real world things that a Google sized org has to deal with. Machine maintenance - someone has to coordinate updates across multiple data centers run by different organizations. Imagine you want to update networking from 1Gbs to 10Gbs across 3 DS in 3 different countries and time zones. After a while you’ll wish to own data centers, but that means multiple legal entities, bank accounts, etc. in different countries. Machine procurement - one DC can buy Intels XYZ while another cannot because there are no local vendors who can deliver. So now you have a dilemma - build software working on different hardware, complicate all future updates or go and negotiate with Intel and other vendors. Regulatory changes is when Malaysian government decides that it wants to protect children from something and you have to take extra measures before you can show search results or publish ads. Or EU votes for something called GDPR. Or Russia suddenly decides that Ads must be VAT taxed. And don’t forget about a Microsoft exec who wants to discuss how they can use Google Search in their browser. Honestly I don’t see how it can be done with 50 engineers. You’ll need hundreds of people, mostly managers and clerks to keep this thing running.

ZeroSolstice 3y ago
I appreciate the examples but none of these items are difficult regardless of size of the organization. These are standard system and network administrative items or on-site technician work. Unless you have other data about the borg architecture to show, its entire premise from the paper[1] is that it just runs jobs and removes the developers needing to know about the underlying hardware or OS that its running on it. From the paper I referenced its huge cluster and if its their own software I'm sure with the development experience at google they have accounted for loosing a cluster node. Losing any number of cluster nodes is the same scenario if you were performing an software or hardware upgrade in which case you would be taking a node offline. I would be very skeptical that there are people manually kicking off upgrade jobs for nodes in the cluster. Maybe an A/B deployment or burn-in for a week before fully pushing it out with a pipeline workflow. I'd more so say that their borg implementation probably runs at different minor versions frequently. Google runs their own data centers, thats the video documentary I referenced[2]. For machine procurement I don't see that as a SRE task. If they are making custom boards and systems they are going to have an entire group supporting that. At the size of google I'm thinking they wouldn't be running into supplier issues that they could figure out easily or account for in their borg software. Again its supposed to be one big abstraction. For the regulatory changes for services like SEO/advertising, GMAIL, search, etc they all have separate teams for those services and would seem separate from SRE's that are maintaining borg the service which is provided to those groups. [1] https://research.google/pubs/pub43438/ https://research.google/pubs/pub43438/ (research paper on borg) [2] https://www.youtube.com/watch?v=XZmGGAbHqa0 https://www.youtube.com/watch?v=XZmGGAbHqa0 (documentary at their data centers)

ZeroSolstice 3y ago

I appreciate the examples but none of these items are difficult regardless of size of the organization. These are standard system and network administrative items or on-site technician work. Unless you have other data about the borg architecture to show, its entire premise from the paper[1] is that it just runs jobs and removes the developers needing to know about the underlying hardware or OS that its running on it. From the paper I referenced its huge cluster and if its their own software I'm sure with the development experience at google they have accounted for loosing a cluster node. Losing any number of cluster nodes is the same scenario if you were performing an software or hardware upgrade in which case you would be taking a node offline. I would be very skeptical that there are people manually kicking off upgrade jobs for nodes in the cluster. Maybe an A/B deployment or burn-in for a week before fully pushing it out with a pipeline workflow. I'd more so say that their borg implementation probably runs at different minor versions frequently. Google runs their own data centers, thats the video documentary I referenced[2]. For machine procurement I don't see that as a SRE task. If they are making custom boards and systems they are going to have an entire group supporting that. At the size of google I'm thinking they wouldn't be running into supplier issues that they could figure out easily or account for in their borg software. Again its supposed to be one big abstraction. For the regulatory changes for services like SEO/advertising, GMAIL, search, etc they all have separate teams for those services and would seem separate from SRE's that are maintaining borg the service which is provided to those groups. [1] https://research.google/pubs/pub43438/ https://research.google/pubs/pub43438/ (research paper on borg) [2] https://www.youtube.com/watch?v=XZmGGAbHqa0 https://www.youtube.com/watch?v=XZmGGAbHqa0 (documentary at their data centers)