4 ms·

The state of App Engine environments seems super confusing, but I _think_ this uses the "second generation standard environment" like the Python 3.7 and Node 8

The state of App Engine environments seems super confusing, but I _think_ this uses the "second generation standard environment" like the Python 3.7 and Node 8 beta environments, which means that unlike App Engine of Yore, you have access to the full package ecosystem and full network access. I'm not actually 100% sure on this because it's not listed on the runtime page (https://cloud.google.com/appengine/docs/standard/appengine-generation https://cloud.google.com/appengine/docs/standard/appengine-g...), but the docs show that you can use composer, unlike the PHP 5.5 runtime. The documentation for all these second generation runtimes is shockingly bad. I consider Google to be _generally_ pretty good at documentation, but this is one of those cases where their propensity for confusing naming and classification of their products makes it hard as hell to figure out what is going on. It doesn't help that the top-level "product overview" page is useless marketing junk (microservices! serverless!) instead of a practical description, which is buried several links deep. Having never used App Engine but occasionally wandering into its docs when a new product gets announced like this, am I right in assuming that App Engine's "second generation standard environment" is more or less equivalent to the feature set you'd get from Heroku? The flexible environment also continues to confuse me - it talks about being a managed Docker container, but it doesn't seem like you actually get access to anything like a Dockerfile and that's more of an implementation detail on their end.

kylecordes 8y ago
For the moment, unfortunately it appears this second generation also lacks some of the great parts of the older platform. For example, with the shiny new GAE Standard Node Beta, you can't declaratively put your app behind authentication, you can't access the (extraordinarily useful) work queue feature from the older GAE, and probably more I have forgotten about for the moment. I hope that stuff is missing just temporarily - there is a lot of genius in the original GAE.
- floatboth 8y ago
  genius… or vendor lock-in?
  - myelin 8y ago
    Member of the App Engine team here. In the context of the second-generation runtimes, this is one of the big reasons we're pushing people to use standard libraries and public cloud services rather than App Engine specific services accessed through syscall magic that doesn't work elsewhere. We want you to be able to run your app locally or on a non-Google server (or on Flex, GCE, GKE, etc).
    - detaro 8y ago
      You should really consider either very detailed porting guides or compatibility libraries, at least from what I hear those required changes will keep some projects on first-gen otherwise (and thus Python 2, and thus requests to dependencies to please keep Python 2 supported and ...)
- wasd 8y ago
  Based on some comments from Python 3 on GAE Standard [1], I think Cloud Task supersedes queue. DSL Authentication sounds cool though. [1]: https://news.ycombinator.com/item?id=17717045 https://news.ycombinator.com/item?id=17717045
- derefr 8y ago
  The first-generation GAE Standard Environment felt a lot like† programming in a port of the given language's runtime to a "cluster OS" like Mesos or Plan9—except it went even further: rather than the "GAE OS" just exposing things like "the cluster's KV storage" as a character device, and then making language runtimes implement a userland protocol library to speak to it over that device, the GAE OS's ABI just has system calls for things like key-value storage or job-queuing (calls which aren't even parameterized by a handle, since there's only one possible kvstore you could mean—the cluster's.) The language runtime ports are, then, essentially just exposing those system-calls, 1:1. It reminds me a lot of being back in the 80s, when you could write raw assembly that simply called BIOS interrupts to read and write from a disk (by logical disk number!), rather than there being this whole edifice of an OS kernel in between, mapping high-level conceits like logical disk requests to low-level SCSI/ATA/etc. wire protocol messages. That mapping was instead happening at a firmware level, such that the hardware (the CPU) could expose intrinsics that would look like any other ISA instruction, but which would turn around and use the firmware (the BIOS) to execute the instruction. That BIOS-interrupt style really feels, to me, like how computers should be: rather than everything compiling down to RISC code that inevitably gets bogged down in kernel context switches as it attempts to express its intent through thousands of low-level system calls, why not have an ultra-high-level CISC ISA that can just express its intent to the hardware [and firmware] directly, with no impedance mismatch? All the "expansion" of intent to RISC would happen in ring 0, with no switching back and forth. (A good example of this kind of CISC-expression-of-intent: kernel packet filtering.) Or, if you don't want to go that far: why not an abstract machine, like the JVM, with these hyper-CISC semantics? I know there's already one example of this approach in the way Erlang's BEAM VM implements its "send" instruction. In the expensive cases—when a module is not loaded in the local VM, or when you need to send the message to a remote node over the distribution protocol with a custom transport—the "send" op can bounce back down into calling other Erlang modules (essentially "firmware"!) to get its job done. I'd love an abstract machine that had that approach... but for everything. --- † I say "felt a lot like", but from my understanding, this is how it actually was/is! GAE Standard Environment v1's interpreters are compiled as Portable Native Client [PNaCl] binaries, with extensions to the "ABI side" of the PPAPI for the specific services the Native Client VM host provided to its guest. The language runtimes then 1:1 expose these PNaCl instruction-set extensions as the functions of modules like ndb, taskqueue, etc.
- jkaplowitz 8y ago
  Anyone know if Cloud IAP can handle at least some of the auth wall use cases in this environment, and how to do it? I suspect yes, but the last time I checked the Cloud IAP docs (for GAE standard's recent Python 3.7 launch) they didn't yet address the unique situation of second-generation App Engine standard environment runtimes.
derefr 8y ago
> The flexible environment also continues to confuse me - it talks about being a managed Docker container, but it doesn't seem like you actually get access to anything like a Dockerfile and that's more of an implementation detail on their end. You're right that you don't get access to the Dockerfile. Docker isn't a 100%-isolated environment for multitenant use-cases, so Google needs to control the stack themselves, using specific trusted versions of base-image layers and app layers. There's maybe extra "Docker-image compile time" security restrictions made on top of that as well. However, it's not just an implementation detail—it has pretty visible effects. 1. A Flexible Environment app slug gets baked into a regular Docker image and placed into your Google Container Registry alongside container-images you've built yourself with GCR. (And your GCR registry, in turn, is backed by a Google Cloud Storage bucket in your own account—meaning that you're paying for storing the resulting images just like any other uploaded objects. You should prune old App Engine Flexible Environment image versions if you don't want to pay!) 2. You can do whatever you want with the filesystem of your container, because it is just a Docker container. You can mount volumes to the container in the container spec (app.yaml), just like if you were writing a k8s container spec. (In fact, sharing IPC sockets through a Docker volume-mount is how the Google Cloud SQL proxy works.) You can internally auto-update your container by downloading new Python/Node/etc. code into the container and getting the interpreter to re-exec(2) itself. Pretty much anything that can run in a "Python Docker container environment" can run in Google's "Python Flexible Environment", because they're pretty much the same (save for those container-build-time security restrictions.)
bus_error 8y ago
Disclosure: I work for Google on the App Engine team doing documentation. Firstly, a sincere thank you for reading and caring about our documentation. With regards to the page you referenced, would you please elaborate on what information would help you come to the appropriate conclusion about using a second generation runtime? Your feedback would be very useful to us in order to shape this page into something that provides you with actionable information rather than "useless marketing junk" :) Also to address a point you made in the first paragraph of your post, the PHP 7.2 beta runtime is a "second generation" runtime. Cheers
- avolcano 8y ago
  Sure, and apologies for the harshness in my original post. My main complaint about the marketing landing page at https://cloud.google.com/appengine/ https://cloud.google.com/appengine/ is that it doesn't contain any immediate information about the different environments. It does have a link to "Choosing the right Environment" but it's _waaay_ down below the fold. In general I wish Google Cloud product pages had their big "View Documentation" buttons up at the top next to the free trial button, instead of way down below the pricing section. In addition, neither the "Choosing the right Environment" nor the "App Engine Standard Environment" docs linked at the bottom of the product page contain detailed information about the second-generation standard environment, which is frustrating. I do appreciate that the language-specific pages in the main docs link contain a short comparison that includes the beta environments, at least. That said, the primary document comparing the environments (https://cloud.google.com/appengine/docs/the-appengine-environments https://cloud.google.com/appengine/docs/the-appengine-enviro...) seems very out of date. For example, it mentions you should use the flexible environment if your app: > Depends on other software, including operating system packages such as imagemagick, ffmpeg, libgit2, or others through apt-get. However, the Node standard environment contains these three packages and more (https://cloud.google.com/appengine/docs/standard/nodejs/reference/system-packages https://cloud.google.com/appengine/docs/standard/nodejs/refe...). There might be other nit-picks to be said, but really, my main criticism of the documentation is that trying to understand the split between "standard/flexible/second-generation standard" environments at first glance is tricky. I'm sure y'all have plans for this going forward as these environments come out of beta, so I'll hold off on whining too much more. Thanks for asking for further feedback, and for clarifying that the new PHP runtime is a second generation one.

kylecordes 8y ago

For the moment, unfortunately it appears this second generation also lacks some of the great parts of the older platform. For example, with the shiny new GAE Standard Node Beta, you can't declaratively put your app behind authentication, you can't access the (extraordinarily useful) work queue feature from the older GAE, and probably more I have forgotten about for the moment. I hope that stuff is missing just temporarily - there is a lot of genius in the original GAE.

floatboth 8y ago
genius… or vendor lock-in?
- myelin 8y ago
  Member of the App Engine team here. In the context of the second-generation runtimes, this is one of the big reasons we're pushing people to use standard libraries and public cloud services rather than App Engine specific services accessed through syscall magic that doesn't work elsewhere. We want you to be able to run your app locally or on a non-Google server (or on Flex, GCE, GKE, etc).
  - detaro 8y ago
    You should really consider either very detailed porting guides or compatibility libraries, at least from what I hear those required changes will keep some projects on first-gen otherwise (and thus Python 2, and thus requests to dependencies to please keep Python 2 supported and ...)
wasd 8y ago
Based on some comments from Python 3 on GAE Standard [1], I think Cloud Task supersedes queue. DSL Authentication sounds cool though. [1]: https://news.ycombinator.com/item?id=17717045 https://news.ycombinator.com/item?id=17717045
derefr 8y ago
The first-generation GAE Standard Environment felt a lot like† programming in a port of the given language's runtime to a "cluster OS" like Mesos or Plan9—except it went even further: rather than the "GAE OS" just exposing things like "the cluster's KV storage" as a character device, and then making language runtimes implement a userland protocol library to speak to it over that device, the GAE OS's ABI just has system calls for things like key-value storage or job-queuing (calls which aren't even parameterized by a handle, since there's only one possible kvstore you could mean—the cluster's.) The language runtime ports are, then, essentially just exposing those system-calls, 1:1. It reminds me a lot of being back in the 80s, when you could write raw assembly that simply called BIOS interrupts to read and write from a disk (by logical disk number!), rather than there being this whole edifice of an OS kernel in between, mapping high-level conceits like logical disk requests to low-level SCSI/ATA/etc. wire protocol messages. That mapping was instead happening at a firmware level, such that the hardware (the CPU) could expose intrinsics that would look like any other ISA instruction, but which would turn around and use the firmware (the BIOS) to execute the instruction. That BIOS-interrupt style really feels, to me, like how computers should be: rather than everything compiling down to RISC code that inevitably gets bogged down in kernel context switches as it attempts to express its intent through thousands of low-level system calls, why not have an ultra-high-level CISC ISA that can just express its intent to the hardware [and firmware] directly, with no impedance mismatch? All the "expansion" of intent to RISC would happen in ring 0, with no switching back and forth. (A good example of this kind of CISC-expression-of-intent: kernel packet filtering.) Or, if you don't want to go that far: why not an abstract machine, like the JVM, with these hyper-CISC semantics? I know there's already one example of this approach in the way Erlang's BEAM VM implements its "send" instruction. In the expensive cases—when a module is not loaded in the local VM, or when you need to send the message to a remote node over the distribution protocol with a custom transport—the "send" op can bounce back down into calling other Erlang modules (essentially "firmware"!) to get its job done. I'd love an abstract machine that had that approach... but for everything. --- † I say "felt a lot like", but from my understanding, this is how it actually was/is! GAE Standard Environment v1's interpreters are compiled as Portable Native Client [PNaCl] binaries, with extensions to the "ABI side" of the PPAPI for the specific services the Native Client VM host provided to its guest. The language runtimes then 1:1 expose these PNaCl instruction-set extensions as the functions of modules like ndb, taskqueue, etc.
jkaplowitz 8y ago
Anyone know if Cloud IAP can handle at least some of the auth wall use cases in this environment, and how to do it? I suspect yes, but the last time I checked the Cloud IAP docs (for GAE standard's recent Python 3.7 launch) they didn't yet address the unique situation of second-generation App Engine standard environment runtimes.

derefr 8y ago

> The flexible environment also continues to confuse me - it talks about being a managed Docker container, but it doesn't seem like you actually get access to anything like a Dockerfile and that's more of an implementation detail on their end. You're right that you don't get access to the Dockerfile. Docker isn't a 100%-isolated environment for multitenant use-cases, so Google needs to control the stack themselves, using specific trusted versions of base-image layers and app layers. There's maybe extra "Docker-image compile time" security restrictions made on top of that as well. However, it's not just an implementation detail—it has pretty visible effects. 1. A Flexible Environment app slug gets baked into a regular Docker image and placed into your Google Container Registry alongside container-images you've built yourself with GCR. (And your GCR registry, in turn, is backed by a Google Cloud Storage bucket in your own account—meaning that you're paying for storing the resulting images just like any other uploaded objects. You should prune old App Engine Flexible Environment image versions if you don't want to pay!) 2. You can do whatever you want with the filesystem of your container, because it is just a Docker container. You can mount volumes to the container in the container spec (app.yaml), just like if you were writing a k8s container spec. (In fact, sharing IPC sockets through a Docker volume-mount is how the Google Cloud SQL proxy works.) You can internally auto-update your container by downloading new Python/Node/etc. code into the container and getting the interpreter to re-exec(2) itself. Pretty much anything that can run in a "Python Docker container environment" can run in Google's "Python Flexible Environment", because they're pretty much the same (save for those container-build-time security restrictions.)

bus_error 8y ago

Disclosure: I work for Google on the App Engine team doing documentation. Firstly, a sincere thank you for reading and caring about our documentation. With regards to the page you referenced, would you please elaborate on what information would help you come to the appropriate conclusion about using a second generation runtime? Your feedback would be very useful to us in order to shape this page into something that provides you with actionable information rather than "useless marketing junk" :) Also to address a point you made in the first paragraph of your post, the PHP 7.2 beta runtime is a "second generation" runtime. Cheers

avolcano 8y ago
Sure, and apologies for the harshness in my original post. My main complaint about the marketing landing page at https://cloud.google.com/appengine/ https://cloud.google.com/appengine/ is that it doesn't contain any immediate information about the different environments. It does have a link to "Choosing the right Environment" but it's _waaay_ down below the fold. In general I wish Google Cloud product pages had their big "View Documentation" buttons up at the top next to the free trial button, instead of way down below the pricing section. In addition, neither the "Choosing the right Environment" nor the "App Engine Standard Environment" docs linked at the bottom of the product page contain detailed information about the second-generation standard environment, which is frustrating. I do appreciate that the language-specific pages in the main docs link contain a short comparison that includes the beta environments, at least. That said, the primary document comparing the environments (https://cloud.google.com/appengine/docs/the-appengine-environments https://cloud.google.com/appengine/docs/the-appengine-enviro...) seems very out of date. For example, it mentions you should use the flexible environment if your app: > Depends on other software, including operating system packages such as imagemagick, ffmpeg, libgit2, or others through apt-get. However, the Node standard environment contains these three packages and more (https://cloud.google.com/appengine/docs/standard/nodejs/reference/system-packages https://cloud.google.com/appengine/docs/standard/nodejs/refe...). There might be other nit-picks to be said, but really, my main criticism of the documentation is that trying to understand the split between "standard/flexible/second-generation standard" environments at first glance is tricky. I'm sure y'all have plans for this going forward as these environments come out of beta, so I'll hold off on whining too much more. Thanks for asking for further feedback, and for clarifying that the new PHP runtime is a second generation one.