6 ms·

How do people who work on Borg look at kubernetes? Is it the real "next generation"

justicezyx 7y ago
When comparing Borg vs. K8s: * Borg is primarily for managing hardware resources. K8s, in comparison, is for managing applications. I often call them a complementing twins. They both are container based cluster manager, but their focuses are opposite. I also use "machine oriented" for Borg, and "application oriented" for k8s. * Borg emphasize on scheduling capabilities, performance, and scalability. And various integration capabilities for wiring together the base line software foundation, and interfacing with hardwares. For example, SDN, security, name service, needs to hook up with Borg, to interact with all applications. * K8s emphasize on modeling application semantics and how to map them into containers. K8s provides abstractions for on boarding apps and a lot of toolings. In summary: * K8s is considered a layer on top of Borg, if it can ever be incorporated into Google production. * As of now, there is nothing in sight that can be considered next generation Borg. Just like there isn't any next generation Linux. Software like Borg, and Linux, can only be supplanted, not succeeded. Their vast amount capabilities defy succeeding. In other words, if one want to build a better Borg (or Linux) they are doing it wrong. K8s did not try to better mesos or open stacks, or Borg perse, it's just a different thing. Edit: There are numerous difference between Borg and kubernetes. Send me an email listed in my profile, if you are really into the gory details. We can setup some discussion.
- sandGorgon 7y ago
  from what i read as comments on this post, there is a bit of a gray area here It seems that K8s takes networking under its purview, Borg does not and leaves it to the client (another comment here). K8s also implements StatefulSets for storage.
- sterlind 7y ago
  There's still nothing really like Borg in the open-source world. Linux can't be replaced because it's open-source, Borg is secret sauce. Amazon and Microsoft built their infrastructure to be based around long-lived VM tenants, Borg based Google around tiny containers. It's doubtful another cloud will come along or any of us giants will rebuild from the ground up, so I guess it's unlikely anyone will ever need a Borg besides Google. I find it sad though, because I fell in love with it from the rumors I heard and papers I read when I redesigned Azure's cluster scheduler algorithm. I think it's sad that 99% of the resources are used by hogs, though. I always thought it'd be neat to build on tiny containers; this suggests that you really can be fine with a few giant tenants and minimal colocation. sigh some day I'll get to design my dream Borg successor, even if it's a TempleOS-like art project after I've descended into dementia.
  - justicezyx 7y ago
    Mind to connect me at info@nascentcore.ai? We might have something to show.
    - sterlind 7y ago
      I tried, but your email server is down! Gmail is saying it's timing out.
      - justicezyx 6y ago
        It's info@nascentcore.com Sorry about that...
  - thu2111 7y ago
    Don't be fooled by the 99% figure. It says in 2011 the figure wasn't much different. Well that was a bit surprising to me because when I was using Borg in 2011 it didn't feel that way. Borg is vast. Unimaginably vast. Any engineer can start a job in the free tier for a personal web server in minutes, all the way up to saturating 100,000 cores to process some dataset. You can browse cluster job lists forever and never reach the bottom. 1% of jobs is still a huge number of different jobs. The "hogs" are going to be jobs like web search serving, indexing and related, logs processing etc. The "mice" are going to be the long tail of jobs. Remember that this sort of paper presents Borg as a kind of exemplar of what large cluster systems look like, but as you already observe, that's not really true. Borg is really unique to Google and Google is really unique. In particular due to the personalities of its founders and its financial position Google has a massive long tail of products and web sites that don't see much usage, or see decent usage by non-Google standards but which gets lost in the noise of search, ads, YouTube etc. So the hogs vs mice phenomenon is telling us more about the nature of Google and how it does projects than something fundamental about job scheduler systems.
    - alexeldeib 7y ago
      Interesting point re: the long tail, but isn't the same true of all the hyperscalars? It's certainly true for Amazon and Microsoft, who both have a better reputation for keeping the long tail of products alive than Google. So if anything, that would point at Google having optimized for this scenario better and earlier.

When comparing Borg vs. K8s: * Borg is primarily for managing hardware resources. K8s, in comparison, is for managing applications. I often call them a complementing twins. They both are container based cluster manager, but their focuses are opposite. I also use "machine oriented" for Borg, and "application oriented" for k8s. * Borg emphasize on scheduling capabilities, performance, and scalability. And various integration capabilities for wiring together the base line software foundation, and interfacing with hardwares. For example, SDN, security, name service, needs to hook up with Borg, to interact with all applications. * K8s emphasize on modeling application semantics and how to map them into containers. K8s provides abstractions for on boarding apps and a lot of toolings. In summary: * K8s is considered a layer on top of Borg, if it can ever be incorporated into Google production. * As of now, there is nothing in sight that can be considered next generation Borg. Just like there isn't any next generation Linux. Software like Borg, and Linux, can only be supplanted, not succeeded. Their vast amount capabilities defy succeeding. In other words, if one want to build a better Borg (or Linux) they are doing it wrong. K8s did not try to better mesos or open stacks, or Borg perse, it's just a different thing. Edit: There are numerous difference between Borg and kubernetes. Send me an email listed in my profile, if you are really into the gory details. We can setup some discussion.

sandGorgon 7y ago
from what i read as comments on this post, there is a bit of a gray area here It seems that K8s takes networking under its purview, Borg does not and leaves it to the client (another comment here). K8s also implements StatefulSets for storage.
sterlind 7y ago
There's still nothing really like Borg in the open-source world. Linux can't be replaced because it's open-source, Borg is secret sauce. Amazon and Microsoft built their infrastructure to be based around long-lived VM tenants, Borg based Google around tiny containers. It's doubtful another cloud will come along or any of us giants will rebuild from the ground up, so I guess it's unlikely anyone will ever need a Borg besides Google. I find it sad though, because I fell in love with it from the rumors I heard and papers I read when I redesigned Azure's cluster scheduler algorithm. I think it's sad that 99% of the resources are used by hogs, though. I always thought it'd be neat to build on tiny containers; this suggests that you really can be fine with a few giant tenants and minimal colocation. sigh some day I'll get to design my dream Borg successor, even if it's a TempleOS-like art project after I've descended into dementia.
- justicezyx 7y ago
  Mind to connect me at info@nascentcore.ai? We might have something to show.
  - sterlind 7y ago
    I tried, but your email server is down! Gmail is saying it's timing out.
    - justicezyx 6y ago
      It's info@nascentcore.com Sorry about that...
- thu2111 7y ago
  Don't be fooled by the 99% figure. It says in 2011 the figure wasn't much different. Well that was a bit surprising to me because when I was using Borg in 2011 it didn't feel that way. Borg is vast. Unimaginably vast. Any engineer can start a job in the free tier for a personal web server in minutes, all the way up to saturating 100,000 cores to process some dataset. You can browse cluster job lists forever and never reach the bottom. 1% of jobs is still a huge number of different jobs. The "hogs" are going to be jobs like web search serving, indexing and related, logs processing etc. The "mice" are going to be the long tail of jobs. Remember that this sort of paper presents Borg as a kind of exemplar of what large cluster systems look like, but as you already observe, that's not really true. Borg is really unique to Google and Google is really unique. In particular due to the personalities of its founders and its financial position Google has a massive long tail of products and web sites that don't see much usage, or see decent usage by non-Google standards but which gets lost in the noise of search, ads, YouTube etc. So the hogs vs mice phenomenon is telling us more about the nature of Google and how it does projects than something fundamental about job scheduler systems.
  - alexeldeib 7y ago
    Interesting point re: the long tail, but isn't the same true of all the hyperscalars? It's certainly true for Amazon and Microsoft, who both have a better reputation for keeping the long tail of products alive than Google. So if anything, that would point at Google having optimized for this scenario better and earlier.