8 ms·

Borg: The Next Generation [pdf]

104 points by blopeur 6y ago

blopeur 6y ago
Paper : https://dl.acm.org/doi/abs/10.1145/3342195.3387517 https://dl.acm.org/doi/abs/10.1145/3342195.3387517 Video : https://www.youtube.com/watch?v=1qUQsZwbu6s https://www.youtube.com/watch?v=1qUQsZwbu6s
kgraves 6y ago
Interesting, I can only wish to run this sort of tech for my own small company.
- yongjik 6y ago
  I think you shouldn't - it's like wishing you could drive an 18-wheeler to do grocery. Most people don't need that kind of firepower ever, and that's a good thing - where are you going to park your 18-wheeler, for starters?
- deleted 6y ago
  [deleted]
- jmalicki 6y ago
  Can't you run Kubernetes? (which is the open-source project from Google inspired by Borg)
  - twblalock 6y ago
    Kubernetes is very different from Borg.
    - tra3 6y ago
      Can you share how? I’ve heard of borg previously as a inspiration for open source k8s but I’m surprised to hear that it’s very different.
      - GauntletWizard 6y ago
        As a former Google SRE: Kubernetes is very different for Borg, but it's also not. Most of the core concepts apply (A Borg "Alloc" is a Kubernetes "Pod", a "Task" in an alloc is a "Container" inside a pod, they're often implemented very differently but the abstraction can be reasoned about in much the same ways. However, the developer experience is hugely different, both in the things that Kubernetes does have (Configuration Canaries) and the things it doesn't have (Global Load Balancing). There's patterns and hacks around most of these, but they're often created by developers who didn't understand the tradeoffs. Most importantly: Borg is very well sanded. Borg was a production-ready system ten years ago, and while it's had to grow and change, there's many things that "work together" just nicely in the Google Ecosystem, that just don't yet outside. Partially because they support many more alternative implementations that don't quite have a standard API, but mostly because there's just rough spots about how you deal with marginal/edge cases that aren't found and documentation written on how to avoid them.
    - justicezyx 6y ago
      It's still an inspiration. K8s was meant to fix borg's broken application management semantics and toolings.
- GauntletWizard 6y ago
  You can! I have all the equivalent graphs and most of the equivalent features on the k8s clusters I run. Google has commoditized borg, at least for the scales that most companies will run.
  - justicezyx 6y ago
    Not really, Borg is itself only at the scale it operates at. It's like Michael Jordan is not the baseball god, if it's only allowed to playing with 5 year old kiddos.
    - GauntletWizard 6y ago
      Some scale is needed to see the true awesomeness of Borg, and you can't built a K8s cluster anywhere near as big as a Borg cell, but many of the same "Cool features" can be built on K8s as well as they can with borg. Things like rolling deploys and stateless, self-healing systems work at k8s scale. I recently got back on a project that was on maintenance mode for a year - But because it had a nice K8s setup, it had more or less managed itself through OS upgrades and rebuilds with very little user intervention.
justicezyx 6y ago
Was involved in this research project for a few months in early 2019. Feel free to post some questions that are not sensitive to internal details, and I can answer them here. Edit: not one of the coauthors, since I left the team in April 2019.
- sandGorgon 6y ago
  How do people who work on Borg look at kubernetes? Is it the real "next generation"
  - justicezyx 6y ago
    When comparing Borg vs. K8s: * Borg is primarily for managing hardware resources. K8s, in comparison, is for managing applications. I often call them a complementing twins. They both are container based cluster manager, but their focuses are opposite. I also use "machine oriented" for Borg, and "application oriented" for k8s. * Borg emphasize on scheduling capabilities, performance, and scalability. And various integration capabilities for wiring together the base line software foundation, and interfacing with hardwares. For example, SDN, security, name service, needs to hook up with Borg, to interact with all applications. * K8s emphasize on modeling application semantics and how to map them into containers. K8s provides abstractions for on boarding apps and a lot of toolings. In summary: * K8s is considered a layer on top of Borg, if it can ever be incorporated into Google production. * As of now, there is nothing in sight that can be considered next generation Borg. Just like there isn't any next generation Linux. Software like Borg, and Linux, can only be supplanted, not succeeded. Their vast amount capabilities defy succeeding. In other words, if one want to build a better Borg (or Linux) they are doing it wrong. K8s did not try to better mesos or open stacks, or Borg perse, it's just a different thing. Edit: There are numerous difference between Borg and kubernetes. Send me an email listed in my profile, if you are really into the gory details. We can setup some discussion.
    - sandGorgon 6y ago
      from what i read as comments on this post, there is a bit of a gray area here It seems that K8s takes networking under its purview, Borg does not and leaves it to the client (another comment here). K8s also implements StatefulSets for storage.
jeffbee 6y ago
This is a really valuable contribution, just to benchmark whatever you are doing vs what's possible, in a similar way that the large datacenter operators have shown everyone that PUE of 1.1 is achievable. This data shows that you can achieve > 60% utilization of both compressible and incompressible resources, overcommit both kinds of resources by 200%, while scheduling task arrivals at over 100 per second. It really is an extremely valuable glimpse into how the bigs operate.
polskibus 6y ago
How much different/better/worse is this compared to Kubernetes?
- fh973 6y ago
  Much simpler to use as a user. I'd say the k8s API is very much over designed and has artifacts that you don't find in other job description languages. Or can anyone explain me the need for StorageClasses, PVCs and PVs with CSI mixins just to access some files? As a scheduler it's much more powerful in capabilities, scalability and performance. It was designed to react to different workload demands in huge clusters very quickly. Imagine a data center goes offline and you need to spin up the search engine while killing a map reduce with 20k workers. A borg cluster is war zone, priorities and quotas win.
  - jrockway 6y ago
    At Google, a vast majority of jobs persisted data ("wrote files") over an RPC API, so the container orchestrator didn't have to care about storage. Over in the real world, that seems to be extremely uncommon. Best case: cloud providers give you a block device tied to an AZ, and you're on your own (or you rewrite your app to use S3, or something that implements a similar API). On prem, people use lots of crazy things ranging from NFS to multi-million-dollar black boxes. Kubernetes has to support them all. CSIs exist because there are a billion vendors of "cloud storage" solutions, and apps need a way to decouple themselves from the details. If you just need a block device, your Kubernetes app can run against anything that implements that API; your app you developed on DigitalOcean will Just Work on GKE. That's the idea behind those. For workload management, Borg feels very close to StatefulSets; your replica count could increase and decrease as you demanded it, but each task was individually named (0.your-job.cell). I ran all my jobs with semantics similar to Deployments (tasks did not care which id they had; they all showed up as load balancer targets) and I think Borg had different abstractions for things in Kubernetes like ReplicaSets (i.e., a particular [config,code] tuple that is running in production that logically makes up a deployment). You didn't ask so I won't go into the details as it gets very complicated, but networking is vastly different. (Kubernetes tries to provide IPAM and connection-balancing that is transparent to workloads; Google preferred client-side "smart" load-balancing libraries. That meant that load-balancing complexity was outside the scope of Borg, but for Kubernetes, it's very much in scope. Often with confusing results, but I digress.) My experience having used both extensively is that there are about the same to the end user. Before I worked at Google, I always let "someone else" handle deployment and maintenance of production. At Google, I felt like I could write and release a piece of software to Borg in the same day (and did once!) After Google, I struggled with a bunch of orchestrators until I landed on Kubernetes, which gives me similar confidence and ease of use. Kubernetes has more API surface, probably because it has more users. At Google you could declare by fiat how jobs were to be run. In the real world, people aren't going to use your thing if you don't support their favorite feature. I have tried the opinionated container orchestrators in the real world (Heroku, Convox, ECS) and they didn't make me happy, while Kubernetes did.
deleted 6y ago
[deleted]
epistasis 6y ago
Does Borg also track and allocate network usage? That has been an issue for me in compute environments in the past for high IO situations.
- fh973 6y ago
  Yes. Not directly Borg but another infrastructure component does.
- jeffbee 6y ago
  Google's network tracks and manages network usage at the edge. I'm not sure if experts would say that was done "by borg" or not. On each host there is a "host enforcer" which manages flows to and from that host. You can read all about it at [1]. I don't think there are any publications suggesting that Borg uses network flow rate for scheduling. 1: https://research.google/pubs/pub43838/ https://research.google/pubs/pub43838/
- tweenagedream 6y ago
  For the most part, other services take care of network resources. Borg might have some host level network limits, but I haven't seen them used much.
pwg 6y ago
Also a name clash with Borg the backup utility (which is the #5 item returned from a google search for "borg" for me when I just tested what would come back): https://borgbackup.readthedocs.io/en/stable/ https://borgbackup.readthedocs.io/en/stable/
- justicezyx 6y ago
  Nah, Borg as a cluster manager way cooler and appropriate than Borg the backup utility.
- altgoogler 6y ago
  Googler here. Borg @ Google predates Borg the backup utility by several years.
- Youden 6y ago
  Because I was curious: - The Borg are a fictional race of aliens in Star Trek: Next Generation. They were first shown on TV in 1989. This appears to be the basis for the software projects' naming. - Borg, Google's cluster management system, was introduced at some point prior to 2006 (per [0], published in 2016, which states Borg was introduced "over a decade ago"). - Borg the backup system was forked from Attic in 2015 [1] [0]: https://cloud.google.com/blog/products/gcp/from-google-to-the-world-the-kubernetes-origin-story https://cloud.google.com/blog/products/gcp/from-google-to-th... [1]: https://borgbackup.readthedocs.io/en/stable/changes.html#version-0-23-0-2015-06-11 https://borgbackup.readthedocs.io/en/stable/changes.html#ver...
gautamcgoel 6y ago
Is Borg essentially Google's version of Condor?
- justicezyx 6y ago
  What is condor?
  - scott_s 6y ago
    They probably mean the Condor project which started in academia in the late '90s as a part of what was called grid computing: https://research.cs.wisc.edu/htcondor https://research.cs.wisc.edu/htcondor They also have an extensive list of publications over the years: https://research.cs.wisc.edu/htcondor/publications.html https://research.cs.wisc.edu/htcondor/publications.html
  - deleted 6y ago
    [deleted]
- joshuamorton 6y ago
  Borg is more like Kubernetes than Condor. It's for managing multiple computing jobs across many machines, instead of managing one highly parallel job across many machines.
throwaway29303 6y ago
What is Borg written in? C, C++ or Go?
- boulos 6y ago
  Borgmaster itself is written in C++, though like many internal Google systems using protobufs for defining a lot of the interfaces and RPC bits. It’s been around a long time, before Go existed :).
remus 6y ago
The resource usage section at the end was really interesting, and surprising to me. 1% of jobs use 99% of resources! It would be interesting to try and understand how this pattern came about and if there's particular engineering decisions that tend to lead to this situation where you have a handful of incredibly resource intensive jobs and loads of very lightweight jobs.
- jeffbee 6y ago
  That seems more related to weird choice of denominator than to any underlying facts of large scale cluster management. Large services have stable names and run forever. Search is just "search" and bigtable is just "bigtable" (some irrelevant details have been elided for clarity). If I run a batch logs analysis job though it is transient and has a unique name every time I run it. So there's a huge long tail of transient job _names_ and a few dominant services with permanent names, which makes the number-of-jobs denominator quirky.
  - yeshengm 6y ago
    Yeah. Likely a small amount of analytics jobs taking up most memory/CPU footprint, while critical prod OLTP jobs take very little resource comparably.

104 points by blopeur 6y ago

Paper : https://dl.acm.org/doi/abs/10.1145/3342195.3387517 https://dl.acm.org/doi/abs/10.1145/3342195.3387517 Video : https://www.youtube.com/watch?v=1qUQsZwbu6s https://www.youtube.com/watch?v=1qUQsZwbu6s

kgraves 6y ago

Interesting, I can only wish to run this sort of tech for my own small company.

yongjik 6y ago
I think you shouldn't - it's like wishing you could drive an 18-wheeler to do grocery. Most people don't need that kind of firepower ever, and that's a good thing - where are you going to park your 18-wheeler, for starters?
deleted 6y ago
[deleted]
jmalicki 6y ago
Can't you run Kubernetes? (which is the open-source project from Google inspired by Borg)
- twblalock 6y ago
  Kubernetes is very different from Borg.
  - tra3 6y ago
    Can you share how? I’ve heard of borg previously as a inspiration for open source k8s but I’m surprised to hear that it’s very different.
    - GauntletWizard 6y ago
      As a former Google SRE: Kubernetes is very different for Borg, but it's also not. Most of the core concepts apply (A Borg "Alloc" is a Kubernetes "Pod", a "Task" in an alloc is a "Container" inside a pod, they're often implemented very differently but the abstraction can be reasoned about in much the same ways. However, the developer experience is hugely different, both in the things that Kubernetes does have (Configuration Canaries) and the things it doesn't have (Global Load Balancing). There's patterns and hacks around most of these, but they're often created by developers who didn't understand the tradeoffs. Most importantly: Borg is very well sanded. Borg was a production-ready system ten years ago, and while it's had to grow and change, there's many things that "work together" just nicely in the Google Ecosystem, that just don't yet outside. Partially because they support many more alternative implementations that don't quite have a standard API, but mostly because there's just rough spots about how you deal with marginal/edge cases that aren't found and documentation written on how to avoid them.
  - justicezyx 6y ago
    It's still an inspiration. K8s was meant to fix borg's broken application management semantics and toolings.
GauntletWizard 6y ago
You can! I have all the equivalent graphs and most of the equivalent features on the k8s clusters I run. Google has commoditized borg, at least for the scales that most companies will run.
- justicezyx 6y ago
  Not really, Borg is itself only at the scale it operates at. It's like Michael Jordan is not the baseball god, if it's only allowed to playing with 5 year old kiddos.
  - GauntletWizard 6y ago
    Some scale is needed to see the true awesomeness of Borg, and you can't built a K8s cluster anywhere near as big as a Borg cell, but many of the same "Cool features" can be built on K8s as well as they can with borg. Things like rolling deploys and stateless, self-healing systems work at k8s scale. I recently got back on a project that was on maintenance mode for a year - But because it had a nice K8s setup, it had more or less managed itself through OS upgrades and rebuilds with very little user intervention.

justicezyx 6y ago

Was involved in this research project for a few months in early 2019. Feel free to post some questions that are not sensitive to internal details, and I can answer them here. Edit: not one of the coauthors, since I left the team in April 2019.

sandGorgon 6y ago
How do people who work on Borg look at kubernetes? Is it the real "next generation"
- justicezyx 6y ago
  When comparing Borg vs. K8s: * Borg is primarily for managing hardware resources. K8s, in comparison, is for managing applications. I often call them a complementing twins. They both are container based cluster manager, but their focuses are opposite. I also use "machine oriented" for Borg, and "application oriented" for k8s. * Borg emphasize on scheduling capabilities, performance, and scalability. And various integration capabilities for wiring together the base line software foundation, and interfacing with hardwares. For example, SDN, security, name service, needs to hook up with Borg, to interact with all applications. * K8s emphasize on modeling application semantics and how to map them into containers. K8s provides abstractions for on boarding apps and a lot of toolings. In summary: * K8s is considered a layer on top of Borg, if it can ever be incorporated into Google production. * As of now, there is nothing in sight that can be considered next generation Borg. Just like there isn't any next generation Linux. Software like Borg, and Linux, can only be supplanted, not succeeded. Their vast amount capabilities defy succeeding. In other words, if one want to build a better Borg (or Linux) they are doing it wrong. K8s did not try to better mesos or open stacks, or Borg perse, it's just a different thing. Edit: There are numerous difference between Borg and kubernetes. Send me an email listed in my profile, if you are really into the gory details. We can setup some discussion.
  - sandGorgon 6y ago
    from what i read as comments on this post, there is a bit of a gray area here It seems that K8s takes networking under its purview, Borg does not and leaves it to the client (another comment here). K8s also implements StatefulSets for storage.

jeffbee 6y ago

This is a really valuable contribution, just to benchmark whatever you are doing vs what's possible, in a similar way that the large datacenter operators have shown everyone that PUE of 1.1 is achievable. This data shows that you can achieve > 60% utilization of both compressible and incompressible resources, overcommit both kinds of resources by 200%, while scheduling task arrivals at over 100 per second. It really is an extremely valuable glimpse into how the bigs operate.

polskibus 6y ago

How much different/better/worse is this compared to Kubernetes?

fh973 6y ago
Much simpler to use as a user. I'd say the k8s API is very much over designed and has artifacts that you don't find in other job description languages. Or can anyone explain me the need for StorageClasses, PVCs and PVs with CSI mixins just to access some files? As a scheduler it's much more powerful in capabilities, scalability and performance. It was designed to react to different workload demands in huge clusters very quickly. Imagine a data center goes offline and you need to spin up the search engine while killing a map reduce with 20k workers. A borg cluster is war zone, priorities and quotas win.
- jrockway 6y ago
  At Google, a vast majority of jobs persisted data ("wrote files") over an RPC API, so the container orchestrator didn't have to care about storage. Over in the real world, that seems to be extremely uncommon. Best case: cloud providers give you a block device tied to an AZ, and you're on your own (or you rewrite your app to use S3, or something that implements a similar API). On prem, people use lots of crazy things ranging from NFS to multi-million-dollar black boxes. Kubernetes has to support them all. CSIs exist because there are a billion vendors of "cloud storage" solutions, and apps need a way to decouple themselves from the details. If you just need a block device, your Kubernetes app can run against anything that implements that API; your app you developed on DigitalOcean will Just Work on GKE. That's the idea behind those. For workload management, Borg feels very close to StatefulSets; your replica count could increase and decrease as you demanded it, but each task was individually named (0.your-job.cell). I ran all my jobs with semantics similar to Deployments (tasks did not care which id they had; they all showed up as load balancer targets) and I think Borg had different abstractions for things in Kubernetes like ReplicaSets (i.e., a particular [config,code] tuple that is running in production that logically makes up a deployment). You didn't ask so I won't go into the details as it gets very complicated, but networking is vastly different. (Kubernetes tries to provide IPAM and connection-balancing that is transparent to workloads; Google preferred client-side "smart" load-balancing libraries. That meant that load-balancing complexity was outside the scope of Borg, but for Kubernetes, it's very much in scope. Often with confusing results, but I digress.) My experience having used both extensively is that there are about the same to the end user. Before I worked at Google, I always let "someone else" handle deployment and maintenance of production. At Google, I felt like I could write and release a piece of software to Borg in the same day (and did once!) After Google, I struggled with a bunch of orchestrators until I landed on Kubernetes, which gives me similar confidence and ease of use. Kubernetes has more API surface, probably because it has more users. At Google you could declare by fiat how jobs were to be run. In the real world, people aren't going to use your thing if you don't support their favorite feature. I have tried the opinionated container orchestrators in the real world (Heroku, Convox, ECS) and they didn't make me happy, while Kubernetes did.

deleted 6y ago

[deleted]

epistasis 6y ago

Does Borg also track and allocate network usage? That has been an issue for me in compute environments in the past for high IO situations.

fh973 6y ago
Yes. Not directly Borg but another infrastructure component does.
jeffbee 6y ago
Google's network tracks and manages network usage at the edge. I'm not sure if experts would say that was done "by borg" or not. On each host there is a "host enforcer" which manages flows to and from that host. You can read all about it at [1]. I don't think there are any publications suggesting that Borg uses network flow rate for scheduling. 1: https://research.google/pubs/pub43838/ https://research.google/pubs/pub43838/
tweenagedream 6y ago
For the most part, other services take care of network resources. Borg might have some host level network limits, but I haven't seen them used much.

pwg 6y ago

Also a name clash with Borg the backup utility (which is the #5 item returned from a google search for "borg" for me when I just tested what would come back): https://borgbackup.readthedocs.io/en/stable/ https://borgbackup.readthedocs.io/en/stable/

justicezyx 6y ago
Nah, Borg as a cluster manager way cooler and appropriate than Borg the backup utility.
altgoogler 6y ago
Googler here. Borg @ Google predates Borg the backup utility by several years.
Youden 6y ago
Because I was curious: - The Borg are a fictional race of aliens in Star Trek: Next Generation. They were first shown on TV in 1989. This appears to be the basis for the software projects' naming. - Borg, Google's cluster management system, was introduced at some point prior to 2006 (per [0], published in 2016, which states Borg was introduced "over a decade ago"). - Borg the backup system was forked from Attic in 2015 [1] [0]: https://cloud.google.com/blog/products/gcp/from-google-to-the-world-the-kubernetes-origin-story https://cloud.google.com/blog/products/gcp/from-google-to-th... [1]: https://borgbackup.readthedocs.io/en/stable/changes.html#version-0-23-0-2015-06-11 https://borgbackup.readthedocs.io/en/stable/changes.html#ver...

gautamcgoel 6y ago

Is Borg essentially Google's version of Condor?

justicezyx 6y ago
What is condor?
- scott_s 6y ago
  They probably mean the Condor project which started in academia in the late '90s as a part of what was called grid computing: https://research.cs.wisc.edu/htcondor https://research.cs.wisc.edu/htcondor They also have an extensive list of publications over the years: https://research.cs.wisc.edu/htcondor/publications.html https://research.cs.wisc.edu/htcondor/publications.html
- deleted 6y ago
  [deleted]
joshuamorton 6y ago
Borg is more like Kubernetes than Condor. It's for managing multiple computing jobs across many machines, instead of managing one highly parallel job across many machines.

throwaway29303 6y ago

What is Borg written in? C, C++ or Go?

boulos 6y ago
Borgmaster itself is written in C++, though like many internal Google systems using protobufs for defining a lot of the interfaces and RPC bits. It’s been around a long time, before Go existed :).

remus 6y ago

The resource usage section at the end was really interesting, and surprising to me. 1% of jobs use 99% of resources! It would be interesting to try and understand how this pattern came about and if there's particular engineering decisions that tend to lead to this situation where you have a handful of incredibly resource intensive jobs and loads of very lightweight jobs.

jeffbee 6y ago
That seems more related to weird choice of denominator than to any underlying facts of large scale cluster management. Large services have stable names and run forever. Search is just "search" and bigtable is just "bigtable" (some irrelevant details have been elided for clarity). If I run a batch logs analysis job though it is transient and has a unique name every time I run it. So there's a huge long tail of transient job _names_ and a few dominant services with permanent names, which makes the number-of-jobs denominator quirky.
- yeshengm 6y ago
  Yeah. Likely a small amount of analytics jobs taking up most memory/CPU footprint, while critical prod OLTP jobs take very little resource comparably.