4 ms·
Can anyone throw some light on what google is doing with tens of thousands of machines in these clusters? Are they hosting crawlers on them?
by mandeepj 12y ago
Can anyone throw some light on what google is doing with tens of thousands of machines in these clusters? Are they hosting crawlers on them?
- coops 12y ago- crawlers - websearch - google brain - video transcoding for youtube - appengine (think snapchat) - gmail Almost everything Google does runs on Borg.
- WestCoastJustin 12y agoThey host pretty much everything -- Maps, Gmail, Search, etc. Highly suggest reading the Wired link below as it talks about this in detail [1], but if you want a more technical look, check out the Youtube video [2], or the Borg paper [3]. One really interesting development, is that Mesos + Marathon + Chronos + Docker, puts a very similar system into the hands of your average IT team today [4]. [1] http://www.wired.com/2013/03/google-borg-twitter-mesos/ http://www.wired.com/2013/03/google-borg-twitter-mesos/ [2] https://www.youtube.com/watch?v=0ZFMlO98Jkc https://www.youtube.com/watch?v=0ZFMlO98Jkc [3] https://research.google.com/pubs/archive/43438.pdf https://research.google.com/pubs/archive/43438.pdf [4] https://www.youtube.com/watch?v=hZNGST2vIds https://www.youtube.com/watch?v=hZNGST2vIds
- dekhn 12y agoWhen I ran Exacycle http://googleresearch.blogspot.com/2013/12/groundbreaking-simulations-by-google.html http://googleresearch.blogspot.com/2013/12/groundbreaking-si... the worker pool was just a low-priority (priority 0 in the paper) job like everything else, soaking up idle cycles. The command and control jobs were just regular borg jobs that managed the worker pools by assigning tasks to them. So whatever else was going on, protein folding and drug docking and telescope simulation was soaking up whatever leftover cycles there were.