5 ms·

Large-scale cluster management at Google with Borg

54 points by mrry 11y ago

gtaylor 11y ago
I've been hesitant to spend a ton of time playing with Kubernetes yet, since it is still in such a state of flux (and the docs are woefully inadequate). With that said, if anyone can make something like this awesome, I'd put my money on Google. Exciting times!
- samkone 11y ago
  Try Mesos/Marathon in addition to Kubernetes. Been running Kubernetes on Mesos and it's just great.
  - throwaway1979 11y ago
    Swarm isn't shabby either.
deleted 11y ago
[deleted]
AdmiralAsshat 11y ago
Does anyone else feel like calling your massive infrastructure software "Borg" might be tempting fate? I mean, I'm not superstitious, but I would be weary of calling my satellite network "Skynet."
- oesmith 11y ago
  Skynet, like the UK military satellite comms network? https://en.m.wikipedia.org/wiki/Skynet_(satellites) https://en.m.wikipedia.org/wiki/Skynet_(satellites)
_dark_matter_ 11y ago
Can anyone explain how MapReduce is built on top of Borg, when Borg doesn't support scheduling with data locality? I'm having a hard time reconciling that. One of the principal use cases of the MapReduce framework is reducing data movement...
- nrr 11y ago
  From the MapReduce paper: "The MapReduce master takes the location information of the input files into account and attempts to schedule a map task on a machine that contains a replica of the corresponding input data." One of two possible designs immediately come to mind: (1) The MapReduce master interacts with Borg directly on behalf of the programmer and places a requirement on the task to run on or near any node that has a replica of the data required or (2) for each Borglet, there is an accompanying MapReduce agent running to accept RPCs from the MapReduce master.
  - GauntletWizard 11y ago
    It's the former.
- powera 11y ago
  The MR master handles it. Mapreduce workers can request shards (work) that have data local to the host it is running on. For large mapreduces this basically works because both the data and the tasks are in most of the cluster, and for small mapreduces it doesn't matter.
- thrownaway2424 11y ago
  You seem to be suggesting that using local i/o to read inputs would be preferable to using remote i/o, but I don't think that's a supportable statement in a datacenter with a high-performance network.
  - wmf 11y ago
    But it's a fact that original MapReduce/Borg used data locality (and networks weren't as fast 15 years ago).
dcsommer 11y ago
Does Kubernetes use chroot jails like Borg, or has it advanced to LXC containers? Nowadays LXC isolation seems strictly preferable.
- jhspaybar 11y ago
  LXC uses chroot + other things. chroot is essentially the foundation of any containerization tool.
  - zobzu 11y ago
    Not really- chroot provides no isolation, namespaces do. choot is the visible portion (changes root.. ie changes /mnt/blah/bleh to /) but not "the foundation" Things can in fact work isolated without chroot, too. Its just convenient.
  - exacube 11y ago
    Note that chroot will provide containerization, but without isolation (processes can always fork out of their chroot jails). You still need things like pid namespaces etc.
- jbeda 11y ago
  Kubernetes uses Docker (or now rkt) to do the actual containing. The focus is on scheduling and managing lots of containers. You may run O(10) containers on a single machine. When you run O(10k)+ containers in a cluster you need new tools to manage things.
- wffurr 11y ago
  Borg also uses cgroups, just like LXC. From the paper: "We use a Linux chroot jail as the primary security isolation mechanism between multiple tasks on the same machine… all Borg tasks run inside a Linux cgroup-based resource container."
  - samkone 11y ago
    Remember that Borg is an old thing. At the time they probably started out with chroot jails, because cgroups weren't ready yet. But Kubernetes, Mesos use docker and cgroups(Mesos).
mandeepj 11y ago
Can anyone throw some light on what google is doing with tens of thousands of machines in these clusters? Are they hosting crawlers on them?
- coops 11y ago
  - crawlers - websearch - google brain - video transcoding for youtube - appengine (think snapchat) - gmail Almost everything Google does runs on Borg.
- WestCoastJustin 11y ago
  They host pretty much everything -- Maps, Gmail, Search, etc. Highly suggest reading the Wired link below as it talks about this in detail [1], but if you want a more technical look, check out the Youtube video [2], or the Borg paper [3]. One really interesting development, is that Mesos + Marathon + Chronos + Docker, puts a very similar system into the hands of your average IT team today [4]. [1] http://www.wired.com/2013/03/google-borg-twitter-mesos/ http://www.wired.com/2013/03/google-borg-twitter-mesos/ [2] https://www.youtube.com/watch?v=0ZFMlO98Jkc https://www.youtube.com/watch?v=0ZFMlO98Jkc [3] https://research.google.com/pubs/archive/43438.pdf https://research.google.com/pubs/archive/43438.pdf [4] https://www.youtube.com/watch?v=hZNGST2vIds https://www.youtube.com/watch?v=hZNGST2vIds
- dekhn 11y ago
  When I ran Exacycle http://googleresearch.blogspot.com/2013/12/groundbreaking-simulations-by-google.html http://googleresearch.blogspot.com/2013/12/groundbreaking-si... the worker pool was just a low-priority (priority 0 in the paper) job like everything else, soaking up idle cycles. The command and control jobs were just regular borg jobs that managed the worker pools by assigning tasks to them. So whatever else was going on, protein folding and drug docking and telescope simulation was soaking up whatever leftover cycles there were.

54 points by mrry 11y ago

I've been hesitant to spend a ton of time playing with Kubernetes yet, since it is still in such a state of flux (and the docs are woefully inadequate). With that said, if anyone can make something like this awesome, I'd put my money on Google. Exciting times!

samkone 11y ago
Try Mesos/Marathon in addition to Kubernetes. Been running Kubernetes on Mesos and it's just great.
- throwaway1979 11y ago
  Swarm isn't shabby either.

deleted 11y ago

[deleted]

AdmiralAsshat 11y ago

Does anyone else feel like calling your massive infrastructure software "Borg" might be tempting fate? I mean, I'm not superstitious, but I would be weary of calling my satellite network "Skynet."

oesmith 11y ago
Skynet, like the UK military satellite comms network? https://en.m.wikipedia.org/wiki/Skynet_(satellites) https://en.m.wikipedia.org/wiki/Skynet_(satellites)

_dark_matter_ 11y ago

Can anyone explain how MapReduce is built on top of Borg, when Borg doesn't support scheduling with data locality? I'm having a hard time reconciling that. One of the principal use cases of the MapReduce framework is reducing data movement...

nrr 11y ago
From the MapReduce paper: "The MapReduce master takes the location information of the input files into account and attempts to schedule a map task on a machine that contains a replica of the corresponding input data." One of two possible designs immediately come to mind: (1) The MapReduce master interacts with Borg directly on behalf of the programmer and places a requirement on the task to run on or near any node that has a replica of the data required or (2) for each Borglet, there is an accompanying MapReduce agent running to accept RPCs from the MapReduce master.
- GauntletWizard 11y ago
  It's the former.
powera 11y ago
The MR master handles it. Mapreduce workers can request shards (work) that have data local to the host it is running on. For large mapreduces this basically works because both the data and the tasks are in most of the cluster, and for small mapreduces it doesn't matter.
thrownaway2424 11y ago
You seem to be suggesting that using local i/o to read inputs would be preferable to using remote i/o, but I don't think that's a supportable statement in a datacenter with a high-performance network.
- wmf 11y ago
  But it's a fact that original MapReduce/Borg used data locality (and networks weren't as fast 15 years ago).

dcsommer 11y ago

Does Kubernetes use chroot jails like Borg, or has it advanced to LXC containers? Nowadays LXC isolation seems strictly preferable.

jhspaybar 11y ago
LXC uses chroot + other things. chroot is essentially the foundation of any containerization tool.
- zobzu 11y ago
  Not really- chroot provides no isolation, namespaces do. choot is the visible portion (changes root.. ie changes /mnt/blah/bleh to /) but not "the foundation" Things can in fact work isolated without chroot, too. Its just convenient.
- exacube 11y ago
  Note that chroot will provide containerization, but without isolation (processes can always fork out of their chroot jails). You still need things like pid namespaces etc.
jbeda 11y ago
Kubernetes uses Docker (or now rkt) to do the actual containing. The focus is on scheduling and managing lots of containers. You may run O(10) containers on a single machine. When you run O(10k)+ containers in a cluster you need new tools to manage things.
wffurr 11y ago
Borg also uses cgroups, just like LXC. From the paper: "We use a Linux chroot jail as the primary security isolation mechanism between multiple tasks on the same machine… all Borg tasks run inside a Linux cgroup-based resource container."
- samkone 11y ago
  Remember that Borg is an old thing. At the time they probably started out with chroot jails, because cgroups weren't ready yet. But Kubernetes, Mesos use docker and cgroups(Mesos).

mandeepj 11y ago

Can anyone throw some light on what google is doing with tens of thousands of machines in these clusters? Are they hosting crawlers on them?

coops 11y ago
- crawlers - websearch - google brain - video transcoding for youtube - appengine (think snapchat) - gmail Almost everything Google does runs on Borg.
WestCoastJustin 11y ago
They host pretty much everything -- Maps, Gmail, Search, etc. Highly suggest reading the Wired link below as it talks about this in detail [1], but if you want a more technical look, check out the Youtube video [2], or the Borg paper [3]. One really interesting development, is that Mesos + Marathon + Chronos + Docker, puts a very similar system into the hands of your average IT team today [4]. [1] http://www.wired.com/2013/03/google-borg-twitter-mesos/ http://www.wired.com/2013/03/google-borg-twitter-mesos/ [2] https://www.youtube.com/watch?v=0ZFMlO98Jkc https://www.youtube.com/watch?v=0ZFMlO98Jkc [3] https://research.google.com/pubs/archive/43438.pdf https://research.google.com/pubs/archive/43438.pdf [4] https://www.youtube.com/watch?v=hZNGST2vIds https://www.youtube.com/watch?v=hZNGST2vIds
dekhn 11y ago
When I ran Exacycle http://googleresearch.blogspot.com/2013/12/groundbreaking-simulations-by-google.html http://googleresearch.blogspot.com/2013/12/groundbreaking-si... the worker pool was just a low-priority (priority 0 in the paper) job like everything else, soaking up idle cycles. The command and control jobs were just regular borg jobs that managed the worker pools by assigning tasks to them. So whatever else was going on, protein folding and drug docking and telescope simulation was soaking up whatever leftover cycles there were.