4 ms·
I appreciate the examples but none of these items are difficult regardless of size of the organization. These are standard system and network administrative ite
by ZeroSolstice 3y ago
I appreciate the examples but none of these items are difficult regardless of size of the organization. These are standard system and network administrative items or on-site technician work. Unless you have other data about the borg architecture to show, its entire premise from the paper[1] is that it just runs jobs and removes the developers needing to know about the underlying hardware or OS that its running on it.
From the paper I referenced its huge cluster and if its their own software I'm sure with the development experience at google they have accounted for loosing a cluster node. Losing any number of cluster nodes is the same scenario if you were performing an software or hardware upgrade in which case you would be taking a node offline. I would be very skeptical that there are people manually kicking off upgrade jobs for nodes in the cluster. Maybe an A/B deployment or burn-in for a week before fully pushing it out with a pipeline workflow. I'd more so say that their borg implementation probably runs at different minor versions frequently.
Google runs their own data centers, thats the video documentary I referenced[2].
For machine procurement I don't see that as a SRE task. If they are making custom boards and systems they are going to have an entire group supporting that. At the size of google I'm thinking they wouldn't be running into supplier issues that they could figure out easily or account for in their borg software. Again its supposed to be one big abstraction.
For the regulatory changes for services like SEO/advertising, GMAIL, search, etc they all have separate teams for those services and would seem separate from SRE's that are maintaining borg the service which is provided to those groups.
[1] https://research.google/pubs/pub43438/ https://research.google/pubs/pub43438/ (research paper on borg)
[2] https://www.youtube.com/watch?v=XZmGGAbHqa0 https://www.youtube.com/watch?v=XZmGGAbHqa0 (documentary at their data centers)