3 ms·

The newest generation of AppEngine runs on this as well. In fact Cloud run and 2nd Gen GAE are exactly the same under the hood afaik. It allowed Google to ditch

by spyspy 7y ago

The newest generation of AppEngine runs on this as well. In fact Cloud run and 2nd Gen GAE are exactly the same under the hood afaik. It allowed Google to ditch the custom APIs and toolchains they forced apps to use in order to keep their infra secure. Fun fact: Cloud Run and GAE both run code in Google's main search clusters, rather than their separate Google Cloud infra.

cameronbrown 7y ago
> Cloud Run and GAE both run code in Google's main search clusters, rather than their separate Google Cloud infra What's the reasoning behind this?
- deleted 7y ago
  [deleted]
- asciimike 7y ago
  Cloud Run/App Engine PM Run and GAE run directly on Borg (which is the shared infrastructure that underpins all Google services, including Cloud products), rather than on VMs. Search/Ads/Maps/etc. run on Borg as well, but there's significant isolation between all those products.
  - derefr 7y ago
    That's the "what", but what's the "why"? Why run these in the main Borg cluster, rather than running them in the (separate, if I'm understanding you) Borg cluster that GCP uses as its substrate? Is it that the GCP Borg cluster is just big enough for GCP's control-plane, and then the rest of GCP is all Borg-less VM hypervisor boxes (running ESXi or what-have-you), so these gVisor-on-Borg workloads wouldn't have anywhere to "live" in the GCP cluster? If that is the issue, then I would have (naively) expected the solution to that to be adding a second, GCP-scale data-plane Borg cluster per zone, just for client workloads; rather than inviting these client workloads to co-mingle with Google's own workloads in the non-GCP part of the DC.
    - tweenagedream 7y ago
      Isolation is often done in software, Google has invested a lot of effort in making sure the distinct services that they run e.g. Youtube transcoding on the same machine search is running on don't interfere with each other. Whether through cpu constrains or some other priority levels. These are features of borg. https://scholar.google.com/scholar?lr&ie=UTF-8&oe=UTF-8&q=Large-scale+cluster+management+at+Google+with+Borg+Verma+Pedrosa+Korupolu+Oppenheimer+Tune+Wilkes#d=gs_qabs&u=%23p%3DygeSI21kh_0J https://scholar.google.com/scholar?lr&ie=UTF-8&oe=UTF-8&q=La...
    - spyspy 7y ago
      likely just due to the age of appengine vs. all other gcp products
      - sayhello 7y ago
        Used to work on 2nd gen AppEngine. I helped ship the runtimes! Yes, this is due to age. GAE and Run depend on pieces of infrastructure going back a long time.
    - jkaplowitz 7y ago
      I know nothing about the decisions behind where Cloud Run and GAE run, but even customer GCE VMs run on top of Borg, not just the control plane. GAE predates most or all of GCP, and there weren't separate GCP clusters when it got launched. (Used to work for Google including the GCP team, but haven't worked for them for over 4 years and I'm not speaking for them now. I'm reasonably sure this is all already public info.)

by spyspy 7y ago

cameronbrown 7y ago

> Cloud Run and GAE both run code in Google's main search clusters, rather than their separate Google Cloud infra What's the reasoning behind this?

deleted 7y ago
[deleted]
asciimike 7y ago
Cloud Run/App Engine PM Run and GAE run directly on Borg (which is the shared infrastructure that underpins all Google services, including Cloud products), rather than on VMs. Search/Ads/Maps/etc. run on Borg as well, but there's significant isolation between all those products.
- derefr 7y ago
  That's the "what", but what's the "why"? Why run these in the main Borg cluster, rather than running them in the (separate, if I'm understanding you) Borg cluster that GCP uses as its substrate? Is it that the GCP Borg cluster is just big enough for GCP's control-plane, and then the rest of GCP is all Borg-less VM hypervisor boxes (running ESXi or what-have-you), so these gVisor-on-Borg workloads wouldn't have anywhere to "live" in the GCP cluster? If that is the issue, then I would have (naively) expected the solution to that to be adding a second, GCP-scale data-plane Borg cluster per zone, just for client workloads; rather than inviting these client workloads to co-mingle with Google's own workloads in the non-GCP part of the DC.
  - tweenagedream 7y ago
    Isolation is often done in software, Google has invested a lot of effort in making sure the distinct services that they run e.g. Youtube transcoding on the same machine search is running on don't interfere with each other. Whether through cpu constrains or some other priority levels. These are features of borg. https://scholar.google.com/scholar?lr&ie=UTF-8&oe=UTF-8&q=Large-scale+cluster+management+at+Google+with+Borg+Verma+Pedrosa+Korupolu+Oppenheimer+Tune+Wilkes#d=gs_qabs&u=%23p%3DygeSI21kh_0J https://scholar.google.com/scholar?lr&ie=UTF-8&oe=UTF-8&q=La...
  - spyspy 7y ago
    likely just due to the age of appengine vs. all other gcp products
    - sayhello 7y ago
      Used to work on 2nd gen AppEngine. I helped ship the runtimes! Yes, this is due to age. GAE and Run depend on pieces of infrastructure going back a long time.
  - jkaplowitz 7y ago
    I know nothing about the decisions behind where Cloud Run and GAE run, but even customer GCE VMs run on top of Borg, not just the control plane. GAE predates most or all of GCP, and there weren't separate GCP clusters when it got launched. (Used to work for Google including the GCP team, but haven't worked for them for over 4 years and I'm not speaking for them now. I'm reasonably sure this is all already public info.)