3 ms·

There's still nothing really like Borg in the open-source world. Linux can't be replaced because it's open-source, Borg is secret sauce. Amazon and Microsoft bu

There's still nothing really like Borg in the open-source world. Linux can't be replaced because it's open-source, Borg is secret sauce. Amazon and Microsoft built their infrastructure to be based around long-lived VM tenants, Borg based Google around tiny containers. It's doubtful another cloud will come along or any of us giants will rebuild from the ground up, so I guess it's unlikely anyone will ever need a Borg besides Google. I find it sad though, because I fell in love with it from the rumors I heard and papers I read when I redesigned Azure's cluster scheduler algorithm. I think it's sad that 99% of the resources are used by hogs, though. I always thought it'd be neat to build on tiny containers; this suggests that you really can be fine with a few giant tenants and minimal colocation. sigh some day I'll get to design my dream Borg successor, even if it's a TempleOS-like art project after I've descended into dementia.

justicezyx 7y ago
Mind to connect me at info@nascentcore.ai? We might have something to show.
- sterlind 7y ago
  I tried, but your email server is down! Gmail is saying it's timing out.
  - justicezyx 7y ago
    It's info@nascentcore.com Sorry about that...
thu2111 7y ago
Don't be fooled by the 99% figure. It says in 2011 the figure wasn't much different. Well that was a bit surprising to me because when I was using Borg in 2011 it didn't feel that way. Borg is vast. Unimaginably vast. Any engineer can start a job in the free tier for a personal web server in minutes, all the way up to saturating 100,000 cores to process some dataset. You can browse cluster job lists forever and never reach the bottom. 1% of jobs is still a huge number of different jobs. The "hogs" are going to be jobs like web search serving, indexing and related, logs processing etc. The "mice" are going to be the long tail of jobs. Remember that this sort of paper presents Borg as a kind of exemplar of what large cluster systems look like, but as you already observe, that's not really true. Borg is really unique to Google and Google is really unique. In particular due to the personalities of its founders and its financial position Google has a massive long tail of products and web sites that don't see much usage, or see decent usage by non-Google standards but which gets lost in the noise of search, ads, YouTube etc. So the hogs vs mice phenomenon is telling us more about the nature of Google and how it does projects than something fundamental about job scheduler systems.
- alexeldeib 7y ago
  Interesting point re: the long tail, but isn't the same true of all the hyperscalars? It's certainly true for Amazon and Microsoft, who both have a better reputation for keeping the long tail of products alive than Google. So if anything, that would point at Google having optimized for this scenario better and earlier.

justicezyx 7y ago

Mind to connect me at info@nascentcore.ai? We might have something to show.

sterlind 7y ago
I tried, but your email server is down! Gmail is saying it's timing out.
- justicezyx 7y ago
  It's info@nascentcore.com Sorry about that...

thu2111 7y ago

Don't be fooled by the 99% figure. It says in 2011 the figure wasn't much different. Well that was a bit surprising to me because when I was using Borg in 2011 it didn't feel that way. Borg is vast. Unimaginably vast. Any engineer can start a job in the free tier for a personal web server in minutes, all the way up to saturating 100,000 cores to process some dataset. You can browse cluster job lists forever and never reach the bottom. 1% of jobs is still a huge number of different jobs. The "hogs" are going to be jobs like web search serving, indexing and related, logs processing etc. The "mice" are going to be the long tail of jobs. Remember that this sort of paper presents Borg as a kind of exemplar of what large cluster systems look like, but as you already observe, that's not really true. Borg is really unique to Google and Google is really unique. In particular due to the personalities of its founders and its financial position Google has a massive long tail of products and web sites that don't see much usage, or see decent usage by non-Google standards but which gets lost in the noise of search, ads, YouTube etc. So the hogs vs mice phenomenon is telling us more about the nature of Google and how it does projects than something fundamental about job scheduler systems.

alexeldeib 7y ago
Interesting point re: the long tail, but isn't the same true of all the hyperscalars? It's certainly true for Amazon and Microsoft, who both have a better reputation for keeping the long tail of products alive than Google. So if anything, that would point at Google having optimized for this scenario better and earlier.