4 ms·

> So, Google's using something like this internally too for their own workloads A public example of this is Cloud Run [1, 2] [1] https://news.ycombinator.com/

> So, Google's using something like this internally too for their own workloads A public example of this is Cloud Run [1, 2] [1] https://news.ycombinator.com/item?id=19616832 https://news.ycombinator.com/item?id=19616832 [2] https://cloud.google.com/run/docs/reference/container-contract#sandbox https://cloud.google.com/run/docs/reference/container-contra...

WestCoastJustin 7y ago
Ah, cool, thanks. I didn't know they were running that under the hood. Yeah, I've checked out Cloud Run via a screencast I did on it a few weeks back [1]. I really like the concept and am looking forward to seeing the evolution of it! [1] https://sysadmincasts.com/episodes/69-cloud-run-with-knative https://sysadmincasts.com/episodes/69-cloud-run-with-knative
- spyspy 7y ago
  The newest generation of AppEngine runs on this as well. In fact Cloud run and 2nd Gen GAE are exactly the same under the hood afaik. It allowed Google to ditch the custom APIs and toolchains they forced apps to use in order to keep their infra secure. Fun fact: Cloud Run and GAE both run code in Google's main search clusters, rather than their separate Google Cloud infra.
  - cameronbrown 7y ago
    > Cloud Run and GAE both run code in Google's main search clusters, rather than their separate Google Cloud infra What's the reasoning behind this?
    - deleted 7y ago
      [deleted]
    - asciimike 7y ago
      Cloud Run/App Engine PM Run and GAE run directly on Borg (which is the shared infrastructure that underpins all Google services, including Cloud products), rather than on VMs. Search/Ads/Maps/etc. run on Borg as well, but there's significant isolation between all those products.
      - derefr 7y ago
        That's the "what", but what's the "why"? Why run these in the main Borg cluster, rather than running them in the (separate, if I'm understanding you) Borg cluster that GCP uses as its substrate? Is it that the GCP Borg cluster is just big enough for GCP's control-plane, and then the rest of GCP is all Borg-less VM hypervisor boxes (running ESXi or what-have-you), so these gVisor-on-Borg workloads wouldn't have anywhere to "live" in the GCP cluster? If that is the issue, then I would have (naively) expected the solution to that to be adding a second, GCP-scale data-plane Borg cluster per zone, just for client workloads; rather than inviting these client workloads to co-mingle with Google's own workloads in the non-GCP part of the DC.
        tweenagedream 7y ago
        Isolation is often done in software, Google has invested a lot of effort in making sure the distinct services that they run e.g. Youtube transcoding on the same machine search is running on don't interfere with each other. Whether through cpu constrains or some other priority levels. These are features of borg. https://scholar.google.com/scholar?lr&ie=UTF-8&oe=UTF-8&q=Large-scale+cluster+management+at+Google+with+Borg+Verma+Pedrosa+Korupolu+Oppenheimer+Tune+Wilkes#d=gs_qabs&u=%23p%3DygeSI21kh_0J https://scholar.google.com/scholar?lr&ie=UTF-8&oe=UTF-8&q=La...
        spyspy 7y ago
        likely just due to the age of appengine vs. all other gcp products
        sayhello 7y ago
        Used to work on 2nd gen AppEngine. I helped ship the runtimes! Yes, this is due to age. GAE and Run depend on pieces of infrastructure going back a long time.

WestCoastJustin 7y ago

Ah, cool, thanks. I didn't know they were running that under the hood. Yeah, I've checked out Cloud Run via a screencast I did on it a few weeks back [1]. I really like the concept and am looking forward to seeing the evolution of it! [1] https://sysadmincasts.com/episodes/69-cloud-run-with-knative https://sysadmincasts.com/episodes/69-cloud-run-with-knative

spyspy 7y ago
The newest generation of AppEngine runs on this as well. In fact Cloud run and 2nd Gen GAE are exactly the same under the hood afaik. It allowed Google to ditch the custom APIs and toolchains they forced apps to use in order to keep their infra secure. Fun fact: Cloud Run and GAE both run code in Google's main search clusters, rather than their separate Google Cloud infra.
- cameronbrown 7y ago
  > Cloud Run and GAE both run code in Google's main search clusters, rather than their separate Google Cloud infra What's the reasoning behind this?
  - deleted 7y ago
    [deleted]
  - asciimike 7y ago
    Cloud Run/App Engine PM Run and GAE run directly on Borg (which is the shared infrastructure that underpins all Google services, including Cloud products), rather than on VMs. Search/Ads/Maps/etc. run on Borg as well, but there's significant isolation between all those products.
    - derefr 7y ago
      That's the "what", but what's the "why"? Why run these in the main Borg cluster, rather than running them in the (separate, if I'm understanding you) Borg cluster that GCP uses as its substrate? Is it that the GCP Borg cluster is just big enough for GCP's control-plane, and then the rest of GCP is all Borg-less VM hypervisor boxes (running ESXi or what-have-you), so these gVisor-on-Borg workloads wouldn't have anywhere to "live" in the GCP cluster? If that is the issue, then I would have (naively) expected the solution to that to be adding a second, GCP-scale data-plane Borg cluster per zone, just for client workloads; rather than inviting these client workloads to co-mingle with Google's own workloads in the non-GCP part of the DC.
      - tweenagedream 7y ago
        Isolation is often done in software, Google has invested a lot of effort in making sure the distinct services that they run e.g. Youtube transcoding on the same machine search is running on don't interfere with each other. Whether through cpu constrains or some other priority levels. These are features of borg. https://scholar.google.com/scholar?lr&ie=UTF-8&oe=UTF-8&q=Large-scale+cluster+management+at+Google+with+Borg+Verma+Pedrosa+Korupolu+Oppenheimer+Tune+Wilkes#d=gs_qabs&u=%23p%3DygeSI21kh_0J https://scholar.google.com/scholar?lr&ie=UTF-8&oe=UTF-8&q=La...
      - spyspy 7y ago
        likely just due to the age of appengine vs. all other gcp products
        sayhello 7y ago
        Used to work on 2nd gen AppEngine. I helped ship the runtimes! Yes, this is due to age. GAE and Run depend on pieces of infrastructure going back a long time.