10 ms·

App Engine’s New Go 1.11 Runtime

168 points by SamWhited 8y ago

buss 8y ago
I'm the tech lead and primary software engineer on the Go runtime. If you have any questions, please don't hesitate to ask! I'm super thrilled to announce that Go 1.11 is now available on App Engine! We now support... * vendoring * regular best-practice package structures * go modules * the regular Google Cloud client libraries: https://github.com/GoogleCloudPlatform/google-cloud-go https://github.com/GoogleCloudPlatform/google-cloud-go This is a "second-generation" runtime (https://cloud.google.com/blog/products/gcp/introducing-app-engine-second-generation-runtimes-and-python-3-7 https://cloud.google.com/blog/products/gcp/introducing-app-e...), meaning that we're now running stock Go in the gVisor sandbox (https://github.com/google/gvisor https://github.com/google/gvisor). We've removed all of the restrictions present in the old runtime, like limited socket and file access. You can even import "unsafe"! If you're a current Go-on-App Engine customer, you should check out our migration guide at https://cloud.google.com/appengine/docs/standard/go111/go-differences https://cloud.google.com/appengine/docs/standard/go111/go-di... to learn how to migrate from the Go 1.9 runtime to the new Go 1.11 runtime. For the time being, you can still use the legacy App Engine APIs with the Go 1.11 runtime, but you should start migrating to the Google Cloud client libraries.
- lvillani 8y ago
  Yes! Thanks for releasing this and making my life easier :) I have an old app deployed to App Engine. The app itself is rock solid and chugging along fine (I touch it once a year or so), but I dreaded having to deploy it due to the lack of vendoring support and other... peculiarities of the Go runtime. Glad to see this is no longer an issue!
- bpye 8y ago
  Several of your links are broken, they have been elipsised.
  - deleted 8y ago
    [deleted]
  - enocom 8y ago
    The ellipsis in full is "go-differences" for the migration guide.
  - buss 8y ago
    Sorry, I copy pasted without double checking. Now fixed!
- skybrian 8y ago
  I take it the datastore change is just an api change? (There isn't any data migration needed, is there?) It looks like memcache is moving to a third-party service now instead of a built-in api. Is performance different? Does anyone have experience with this?
  - buss 8y ago
    > I take it the datastore change is just an api change? (There isn't any data migration needed, is there?) Yup! > It looks like memcache is moving to a third-party service now instead of a built-in api. Is performance different? Does anyone have experience with this? I'm not sure about the performance impact. But I can tell you that we're working with the Cloud Memorystore team to have a better memcache story for the App Engine second generation runtimes
    - Strom 8y ago
      Both google.golang.org/appengine/datastore and cloud.google.com/go/datastore connect to the same data? Is there a list of incompatibilities somewhere? I remember seing some serialization differences between these two datastore libraries.
      - steren 8y ago
        Hi (App Engine PM here). Yes both libraries connect to the same database (Cloud Datastore). cloud.google.com/go/datastore is preferred as it uses the Cloud Datastore API instead of the App Engine specific API, so your code will be portable. You are right that the devil is in the details, and there might be some slight serialization differences. I cannot find an exact list. Feel free to post here if you need more help.
      - chrisbroadfoot 8y ago
        You remember right - I think you might be referring to this issue: https://groups.google.com/forum/#!topic/google-api-go-announce/79jtrdeuJAg https://groups.google.com/forum/#!topic/google-api-go-announ... > the cloud datastore package will default to writing your nested structs as entity values, while the appengine datastore packages will only write your nested structs as flattened sets of attributes. Other than that, there are no differences. (GCP Gopher)
- selljamhere 8y ago
  Does the move to gVisor pave the way to connecting CloudSQL instances? The removed limitations sound like they were road blocks.
  - merb 8y ago
    you already could connect to cloudsql instances. however you needed a proxy, but not because of AppEngine, you need it because of security. either you whitelisted your ips or you used the proxy (which go appengine could already use). so with go you just needed to use https://github.com/GoogleCloudPlatform/cloudsql-proxy https://github.com/GoogleCloudPlatform/cloudsql-proxy which means you just need to change the query string. however a new feature is emerging where you can have a private ip that is connected to your internal services, however it is still a beta feature. Source: myself, but not a googler, just a regular user
    - chrisbroadfoot 8y ago
      Notable, though, that this new runtime better supports connecting to Postgres instances through the unix socket (/cloudsql/). (GCP Gopher)
      - deleted 8y ago
        [deleted]
  - buss 8y ago
    For cloudsql in particular, you can connect via the `/cloudsql` Unix socket. But anything that needs raw socket access can now be used from App Engine. See https://cloud.google.com/appengine/docs/standard/go111/using-cloud-sql https://cloud.google.com/appengine/docs/standard/go111/using... and scroll way down to the sample code.
    - selljamhere 8y ago
      Awesome!
- merb 8y ago
  when will appengine support setting environment variables with gcloud command or inside the gcloud console instead of in the app.yaml, i.e. so that each environment can have their own variables without copying app.yaml's?!
  - steren 8y ago
    (App Engine PM here) At this time, app.yaml is the only way to set env vars. If you want to separate environments, you could use different .yaml files, I personally use app.staging.yaml and app.prod.yaml
    - dfischer 8y ago
      Not worried about secrets in repo?
      - CydeWeys 8y ago
        Tech Lead of Google Registry ( https://registry.google https://registry.google ) here. We run on GCP. You definitely don't want to check secrets into the repo for the same reason you don't set them as environment variables: It's not secure. The solution is to use Cloud Key Management Service. More info here: https://cloud.google.com/kms/ https://cloud.google.com/kms/ We use that to either store secrets directly, or to decrypt encrypted entities in a DB (e.g. Datastore). You can see how we use it here (our project is open source): https://github.com/google/nomulus/tree/master/java/google/registry/keyring/kms https://github.com/google/nomulus/tree/master/java/google/re...
        asciimike 8y ago
        Another great resource: https://sethvargo.com/secrets-in-serverless/ https://sethvargo.com/secrets-in-serverless/ Shows how to encrypt env vars with Cloud KMS then decrypt them in memory.
        dfischer 8y ago
- Strom 8y ago
  > I'm the tech lead and primary software engineer on the Go runtime. Hey thanks a lot for your efforts! Your direct communication about the Go runtime progress has been a breath of fresh air compared to the years preceding you. I wonder though, how many developers are there on the App Engine Standard Go team?
- throwaway9981 8y ago
  is gRPC supported now or is that more of an upstream HTTP 1.1 constraint?
  - steren 8y ago
    Hi (App Engine PM here) You can use any go package in the Go 1.11 runtime, including the ones relying on GRPC. Did I answer your question?
    - rhodysurf 8y ago
      You cant run a GRPC server though still right? Thats all I really want to be able to do especially now that the app engine endpoints libraries are no longer supported on python
  - buss 8y ago
    Inbound GRPC will be limited to HTTP1.1; that's a limitation of the platform rather than the language runtime. We're working on HTTP2 support, but I can't promise any timelines.
    - rhodysurf 8y ago
      By inbound GRPC you mean when running an app engine instance as a client right? So running a GRPC server is completely out of question still I would presume?
      - buss 8y ago
        I mean running a GRPC server, yeah. It's a limitation of our networking stack which we're working on fixing.
- wiradikusuma 8y ago
  It's always been my impression that Go on GAE is the most lightweight compared to Java/Python, as it not "memory hungry" like Java nor "slow" like Python. Am I correct?
  - buss 8y ago
    It is quite lightweight, yes! Though...I wouldn't necessarily call Python slow! Python keeps getting faster, and we now support Python 3.7 on our second gen runtimes: https://cloud.google.com/blog/products/gcp/introducing-app-engine-second-generation-runtimes-and-python-3-7 https://cloud.google.com/blog/products/gcp/introducing-app-e...
    - melling 8y ago
      Does anyone have numbers to compare? Go is a compiled language. Should perform much better. Python might be perfectly acceptable of course.
      - tomcam 8y ago
        I would absolutely love those numbers too. Always wondered about the working set of a minimal program, and also one with database access.
        pkroll 8y ago
        Well, there's the Techempower benchmarks at: https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/ You can go to the filter, turn off all languages but Go and Python, and get a general idea. There's also the old Computer Benchmark Game, which should be taken with a whole barrel of salt: https://benchmarksgame-team.pages.debian.net/benchmarksgame/faster/go-python3.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
        eiieirurjdndjd 8y ago
        I actually looked at this before making my other comment. You’re absolutely right about the salt. One of the go benchmarks is basically pure cgo using vector intrinsics. It naturally blows idiomatic JS out of the water. So you might get those kind of results if that’s the kind of code you’re going to write, but it’s far from what I consider idiomatic. Another language’s proponent would be fair to point out that most languages have C FFIs.
- B-Scan 8y ago
  Any plans to support the cloud endpoints with the standard env? Wondering if the new runtime would be able to be embed that. That would be only reason for me not to switch from flex env.
  - asciimike 8y ago
    (GAE PM) Cloud Endpoints currently runs on GAE Standard Gen 1 and GAE Flex. We're investigating a solution for GAE Standard Gen 2 (which go 1.11 is, along with python 3.7, node 8, etc.) and GCF.
    - B-Scan 8y ago
      Thanks for the clarification. Wasn’t aware that it works with Go on Standard Gen 1. Any ETA for Gen 2 or alpha signup?
      - asciimike 8y ago
        Clarification: GAE Standard uses the Endpoints framework, which is Python and Java only (https://cloud.google.com/endpoints/docs/frameworks/about-cloud-endpoints-frameworks https://cloud.google.com/endpoints/docs/frameworks/about-clo...). Sorry for the confusion. Flex uses ESP (https://github.com/cloudendpoints/esp https://github.com/cloudendpoints/esp), which is deployed as a sidecar, and works with any runtime. As for EAP: unfortunately not. Personally, I'd like to see a managed version as opposed to the current framework or sidecar approach, as it'll be easier to implement across products, but that means it'll likely take a little longer.
- warrentr 8y ago
  How long will the legacy App Engine APIs continue to work? Will they be removed before the runtime is GA? These new runtimes seem amazing (and fix a lot of problems), but we also lose a ton of functionality that made app engine so desirable (easy users auth, images api, built in email sending, search, cron, and others). It kind of feels like we've thrown the baby out with the bathwater.
  - steren 8y ago
    (App Engine PM here) The App Engine APIs will be present in the Go 1.11 runtime when it goes GA and until it is turned down. We do not know exactly which future version of the runtime will stop supporting them. We are working on Google Cloud standalone products to replace most missing features (e.g. Cloud Scheduler for cron jobs). Not all of them are ready yet. App Engine APIs were great when there was no alternative (when GCP did not exist, or when App Engine could not use arbitrary packages), but at the same time, contributed to the "lock-in" of App Engine, which was one of the main criticism. We believe it is in the long term benefit of our users to use standalone Google Cloud or third party services, instead of replying on APIs only accessible in App Engine.
    - warrentr 8y ago
      Thanks for the explanation. It makes sense and hopefully replacements are generally available before we lose the old apis.
- Sir_Cmpwn 8y ago
  Hey buss, thanks for stopping by. How do you feel about the distinction between Google and Go? It sometimes feels like they're joined at the hip, something which historically had made people queasy (Java and Oracle, .NET and Microsoft, etc). Though Go is a project which came from Google, I personally feel that it better serves the interests of the Go community to treat them separately. Google receives special treatment from Go - the announcement that AWS Lambda would support Go did not receive similar fanfare on golang.org. Side note: there is some activity on the Go blog which is more mutual cooperation and less Google hivemind: https://blog.golang.org/go-cloud https://blog.golang.org/go-cloud But I still wonder why this stuff belongs here.
  - deleted 8y ago
    [deleted]
  - justinclift 8y ago
    Agreed. To even submit content to the Go Blog requires a Google account. Ugh.
  - jhabdas 8y ago
    Three smart guys made go. Google only has one or two left.
- heromat 8y ago
  I just tried to use go111, but I get the following message: >[7] Access Not Configured. Cloud Build has not been used in project <project> before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudbuild.googleapis.com/overview?project=<project> https://console.developers.google.com/apis/api/cloudbuild.go... then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry. So go111 cannot be used without Cloud Build, for which I have to activate billing?
  - buss 8y ago
    Ah, yeah, looks like you have to activate billing. You'll be solidly within the free tier, though, so you won't be charged.
    - chrisbroadfoot 8y ago
      You can set a $0 limit on your billing account, too.
- paulddraper 8y ago
  I thought Go was statically compiled....i.e. it doesn't have a separate runtime from the program. Does anyone have any elucidation?
  - chrisbroadfoot 8y ago
    Correct, Go's runtime is compiled into the program's binary. However, there's still an environment needed to run that binary. For Go on App Engine, this includes the sandbox (backed by gVisor) and the operating system (Ubuntu). We also tend to call everything in the toolchain the "runtime" - this includes the CLI for uploading and staging your app, and the builder used to compile your program. Not the strictest definition :) (GCP Gopher)
    - paulddraper 8y ago
      So App Engine is a build system as well?
      - chrisbroadfoot 8y ago
        Yes, with App Engine, you upload source code, and it's compiled for you remotely. For this newer runtime, it happens inside a Cloud Build invocation. For App Engine flexible, you can skip that build step and provide your own container, but that isn't possible for App Engine standard. (If you're interested in something like that, see https://g.co/serverlesscontainers https://g.co/serverlesscontainers)
- openbasic 8y ago
  Vendoring! Modules! Finally. Do you have any examples with proposed solutions for local module setups, without having to rely on GOPATH changes?
  - TheDong 8y ago
    This is a go question, not an app engine question. See this page on the go wiki for an example of how to use go modules: https://github.com/kubernetes/kubernetes/pull/58098 https://github.com/kubernetes/kubernetes/pull/58098 Note: you need to have go1.11, and by default you need to be in a directory NOT in your $GOPATH for modules to work in go1.11
    - TheDong 8y ago
      Edit: https://github.com/golang/go/wiki/Modules#how-to-use-modules https://github.com/golang/go/wiki/Modules#how-to-use-modules Copy paste error I guess.
- iamgopal 8y ago
  Does context.background() works ?
  - chrisbroadfoot 8y ago
    (GCP Gopher) Yes, you don't need to use the appengine context for anything (except to access the services at google.golang.org/appengine/...) You can use the standard net/http package to make HTTP requests, for example.
- jostein 8y ago
  This is a big improvement, but the restriction that prevents streaming responses unfortunately still apply: https://cloud.google.com/appengine/docs/standard/go/how-requests-are-handled#streaming_responses https://cloud.google.com/appengine/docs/standard/go/how-requ... Would be great if that was solved.
genericone 8y ago
Am I understanding this correctly, App Engine has made it so "go run goapp.go" on my local environment is basically accomplished with "runtime: goapp" in the app.yaml? No other changes necessary in the code?
- buss 8y ago
  runtime: go111, but otherwise -- yes!
bpye 8y ago
This actually seemed to be live a week or two ago, I managed to run Miniflux [1] on App Engine with no changes though to get it working properly it would need it's scheduling changing. [1] - https://github.com/miniflux/miniflux https://github.com/miniflux/miniflux
- steren 8y ago
  (App Engine PM here) Nice to hear! Yes indeed, we announced Go 1.11 on App Engine last week at Cloud Next London.
  - bpye 8y ago
    Ah I hadn't seen that, I was quite surprised when I used goapp111 and it deployed successfully. It's great to be able to take a normal Go application and run it on Google Cloud. Thanks!
knodi 8y ago
I'm loving the new vendoring via go modules. Can't wait for it to be the default in Go 1.12
yegle 8y ago
Before anyone asks, this is still a beta release that's not covered by App Engine SLA.
dfischer 8y ago
Hmm, I was just running into some context annoyances but this seems to remedy some nuances regarding that. Cool! Good timing! I’m assuming urlfetcher is deprecated now too? I’ll go read the migration guide now ... Awesome work! Love Go and GAE! Very happy so far.
- buss 8y ago
  You can use net/http to fetch URLs now :) Glad you're enjoying both Go and GAE!
  - dfischer 8y ago
    What was the reason urlfetcher had to be engineered in the past? Curious.
    - buss 8y ago
      I don't think I can say much more than: technical choices that required a different path to fetch resources from the broader internet.
deckarep 8y ago
Does this new classic+ environment now supersede the Flex environment? I’m curious because now it’s seems like there are 3 solutions. Although I really like the simplicity of this solution and the modernizing from the constraints of true classic mode.
- buss 8y ago
  Eh, not exactly supersede, but the runtimes are definitely slowly converging. It's a goal of mine to make them easy to switch between -- sometimes you'll want flex because you need a VM, most times you wont. But your code should be easily portable between them. With this release, they are nearly identical already. So two of the solutions are flex and go1.11 on standard. Is the third solution the older runtime versions (eg 1.9 and earlier)? If so, then there's really just two solutions, since 1.11 replaces 1.9 on standard.
- chrisbroadfoot 8y ago
  (GCP Gopher here) If you can run within the standard environment, I'd suggest you do (but you should check pricing, etc to see if it suits). Especially if you want 0-1 scaling. I migrated golang.org from standard to flexible a couple weeks ago, primarily because of increased control over the runtime (custom Docker image) and increased instance sizes (configurable memory, CPU past the App Engine limits, which caps out at 2 GB). See https://golang.org/issues/27205 https://golang.org/issues/27205 if you want to read more.
  - iamgopal 8y ago
    Do you have blog post with data, number of instances etc previously and now ?
    - chrisbroadfoot 8y ago
      I don't, sorry. While the app was unhealthy, it was spinning up dozens of instances. While it was healthy (on standard), I recall it ran on less than a handful of instances. It currently runs on two fat flexible instances (GCE VM), but could easily run on one. We only run on two for some redundancy. I've measured throughput of ~thousands of requests per second on a F4_1G on apps I've worked on previously. They were CPU-constrained, but not doing very much (JSON serialization). The new runtime might be even better, since GOMAXPROCS is no longer 1. correction to my post above: 1 GB is the max memory for standard: https://cloud.google.com/appengine/docs/standard/#instance_classes https://cloud.google.com/appengine/docs/standard/#instance_c...
frakkingcylons 8y ago
I'm not trying to stir shit up...buuut it seems odd that an announcement for a Google Cloud product is being published on the Go project's blog? I say this as a huge fan of Google Cloud and the experience of using Go on App Engine.
- justinclift 8y ago
  Likely because it's clearly Go related? "Announcing [some cloud]’s New Go 1.11 Runtime" That being said, I have no idea how similar submissions of Go content from competing vendors would er... go. ;)
ptype 8y ago
F
thstart 8y ago
I have some C++ libraries called from Go - can I use them now?
- helpfulgoogler 8y ago
  cgo is supported.
deleted 8y ago
[deleted]
xaduha 8y ago
I haven't used GAE since early days, but I heard it supports Docker now. Why wouldn't I just use Docker with whatever runtime I want?
- chrisbroadfoot 8y ago
  (GCP Gopher) We support custom images for the flexible runtime, but not for the standard runtime. If you're as keen for that as I am, sign up here: http://g.co/serverlesscontainers http://g.co/serverlesscontainers
XorNot 8y ago
Is this a GA product or still "beta"? Google's consistent announcement of products as available while in beta is getting annoying, because their SLAs for "beta" products are still 0 and thus impossible to be able to use in actual business applications.
- chrisbroadfoot 8y ago
  (GCP Gopher) Yes, it's beta. You're right - if you require an SLA, don't use beta products. As per https://cloud.google.com/terms/launch-stages https://cloud.google.com/terms/launch-stages: > Beta is the point at which we are ready to open a release for any customer to use. There are no SLA or technical support obligations in a Beta release, and charges may be waived in some cases. Products will be complete from a feature perspective, but may have some open outstanding issues. Beta releases are suitable for limited production use cases.
nixarn 8y ago
Wonder if it's still possible to get answers here :) giving it a shot although it's 13h old by now. We use Go for a production server for multiple of our games, it's used to sync gamestates between devices and allows us to remotely inspect and change player's gamestate. Super handy and it's worked flawlessly! We used to use python, but Go is much better suited as it's less error prone. One question though, we've been having connectivity issues in China, the GAE is deployed in the US, and as it seems there's no multiregional support. What can we do? Ideally keeping as much of the convenience of the GAE as possible.

168 points by SamWhited 8y ago

buss 8y ago

I'm the tech lead and primary software engineer on the Go runtime. If you have any questions, please don't hesitate to ask! I'm super thrilled to announce that Go 1.11 is now available on App Engine! We now support... * vendoring * regular best-practice package structures * go modules * the regular Google Cloud client libraries: https://github.com/GoogleCloudPlatform/google-cloud-go https://github.com/GoogleCloudPlatform/google-cloud-go This is a "second-generation" runtime (https://cloud.google.com/blog/products/gcp/introducing-app-engine-second-generation-runtimes-and-python-3-7 https://cloud.google.com/blog/products/gcp/introducing-app-e...), meaning that we're now running stock Go in the gVisor sandbox (https://github.com/google/gvisor https://github.com/google/gvisor). We've removed all of the restrictions present in the old runtime, like limited socket and file access. You can even import "unsafe"! If you're a current Go-on-App Engine customer, you should check out our migration guide at https://cloud.google.com/appengine/docs/standard/go111/go-differences https://cloud.google.com/appengine/docs/standard/go111/go-di... to learn how to migrate from the Go 1.9 runtime to the new Go 1.11 runtime. For the time being, you can still use the legacy App Engine APIs with the Go 1.11 runtime, but you should start migrating to the Google Cloud client libraries.

lvillani 8y ago
Yes! Thanks for releasing this and making my life easier :) I have an old app deployed to App Engine. The app itself is rock solid and chugging along fine (I touch it once a year or so), but I dreaded having to deploy it due to the lack of vendoring support and other... peculiarities of the Go runtime. Glad to see this is no longer an issue!
bpye 8y ago
Several of your links are broken, they have been elipsised.
- deleted 8y ago
  [deleted]
- enocom 8y ago
  The ellipsis in full is "go-differences" for the migration guide.
- buss 8y ago
  Sorry, I copy pasted without double checking. Now fixed!
skybrian 8y ago
I take it the datastore change is just an api change? (There isn't any data migration needed, is there?) It looks like memcache is moving to a third-party service now instead of a built-in api. Is performance different? Does anyone have experience with this?
- buss 8y ago
  > I take it the datastore change is just an api change? (There isn't any data migration needed, is there?) Yup! > It looks like memcache is moving to a third-party service now instead of a built-in api. Is performance different? Does anyone have experience with this? I'm not sure about the performance impact. But I can tell you that we're working with the Cloud Memorystore team to have a better memcache story for the App Engine second generation runtimes
  - Strom 8y ago
    Both google.golang.org/appengine/datastore and cloud.google.com/go/datastore connect to the same data? Is there a list of incompatibilities somewhere? I remember seing some serialization differences between these two datastore libraries.
    - steren 8y ago
      Hi (App Engine PM here). Yes both libraries connect to the same database (Cloud Datastore). cloud.google.com/go/datastore is preferred as it uses the Cloud Datastore API instead of the App Engine specific API, so your code will be portable. You are right that the devil is in the details, and there might be some slight serialization differences. I cannot find an exact list. Feel free to post here if you need more help.
    - chrisbroadfoot 8y ago
      You remember right - I think you might be referring to this issue: https://groups.google.com/forum/#!topic/google-api-go-announce/79jtrdeuJAg https://groups.google.com/forum/#!topic/google-api-go-announ... > the cloud datastore package will default to writing your nested structs as entity values, while the appengine datastore packages will only write your nested structs as flattened sets of attributes. Other than that, there are no differences. (GCP Gopher)
selljamhere 8y ago
Does the move to gVisor pave the way to connecting CloudSQL instances? The removed limitations sound like they were road blocks.
- merb 8y ago
  you already could connect to cloudsql instances. however you needed a proxy, but not because of AppEngine, you need it because of security. either you whitelisted your ips or you used the proxy (which go appengine could already use). so with go you just needed to use https://github.com/GoogleCloudPlatform/cloudsql-proxy https://github.com/GoogleCloudPlatform/cloudsql-proxy which means you just need to change the query string. however a new feature is emerging where you can have a private ip that is connected to your internal services, however it is still a beta feature. Source: myself, but not a googler, just a regular user
  - chrisbroadfoot 8y ago
    Notable, though, that this new runtime better supports connecting to Postgres instances through the unix socket (/cloudsql/). (GCP Gopher)
    - deleted 8y ago
      [deleted]
- buss 8y ago
  For cloudsql in particular, you can connect via the `/cloudsql` Unix socket. But anything that needs raw socket access can now be used from App Engine. See https://cloud.google.com/appengine/docs/standard/go111/using-cloud-sql https://cloud.google.com/appengine/docs/standard/go111/using... and scroll way down to the sample code.
  - selljamhere 8y ago
    Awesome!
merb 8y ago
when will appengine support setting environment variables with gcloud command or inside the gcloud console instead of in the app.yaml, i.e. so that each environment can have their own variables without copying app.yaml's?!
- steren 8y ago
  (App Engine PM here) At this time, app.yaml is the only way to set env vars. If you want to separate environments, you could use different .yaml files, I personally use app.staging.yaml and app.prod.yaml
  - dfischer 8y ago
    Not worried about secrets in repo?
    - CydeWeys 8y ago
      Tech Lead of Google Registry ( https://registry.google https://registry.google ) here. We run on GCP. You definitely don't want to check secrets into the repo for the same reason you don't set them as environment variables: It's not secure. The solution is to use Cloud Key Management Service. More info here: https://cloud.google.com/kms/ https://cloud.google.com/kms/ We use that to either store secrets directly, or to decrypt encrypted entities in a DB (e.g. Datastore). You can see how we use it here (our project is open source): https://github.com/google/nomulus/tree/master/java/google/registry/keyring/kms https://github.com/google/nomulus/tree/master/java/google/re...
      - asciimike 8y ago
        Another great resource: https://sethvargo.com/secrets-in-serverless/ https://sethvargo.com/secrets-in-serverless/ Shows how to encrypt env vars with Cloud KMS then decrypt them in memory.
        dfischer 8y ago
Strom 8y ago
> I'm the tech lead and primary software engineer on the Go runtime. Hey thanks a lot for your efforts! Your direct communication about the Go runtime progress has been a breath of fresh air compared to the years preceding you. I wonder though, how many developers are there on the App Engine Standard Go team?
throwaway9981 8y ago
is gRPC supported now or is that more of an upstream HTTP 1.1 constraint?
- steren 8y ago
  Hi (App Engine PM here) You can use any go package in the Go 1.11 runtime, including the ones relying on GRPC. Did I answer your question?
  - rhodysurf 8y ago
    You cant run a GRPC server though still right? Thats all I really want to be able to do especially now that the app engine endpoints libraries are no longer supported on python
- buss 8y ago
  Inbound GRPC will be limited to HTTP1.1; that's a limitation of the platform rather than the language runtime. We're working on HTTP2 support, but I can't promise any timelines.
  - rhodysurf 8y ago
    By inbound GRPC you mean when running an app engine instance as a client right? So running a GRPC server is completely out of question still I would presume?
    - buss 8y ago
      I mean running a GRPC server, yeah. It's a limitation of our networking stack which we're working on fixing.
wiradikusuma 8y ago
It's always been my impression that Go on GAE is the most lightweight compared to Java/Python, as it not "memory hungry" like Java nor "slow" like Python. Am I correct?
- buss 8y ago
  It is quite lightweight, yes! Though...I wouldn't necessarily call Python slow! Python keeps getting faster, and we now support Python 3.7 on our second gen runtimes: https://cloud.google.com/blog/products/gcp/introducing-app-engine-second-generation-runtimes-and-python-3-7 https://cloud.google.com/blog/products/gcp/introducing-app-e...
  - melling 8y ago
    Does anyone have numbers to compare? Go is a compiled language. Should perform much better. Python might be perfectly acceptable of course.
    - tomcam 8y ago
      I would absolutely love those numbers too. Always wondered about the working set of a minimal program, and also one with database access.
      - pkroll 8y ago
        Well, there's the Techempower benchmarks at: https://www.techempower.com/benchmarks/ https://www.techempower.com/benchmarks/ You can go to the filter, turn off all languages but Go and Python, and get a general idea. There's also the old Computer Benchmark Game, which should be taken with a whole barrel of salt: https://benchmarksgame-team.pages.debian.net/benchmarksgame/faster/go-python3.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
        eiieirurjdndjd 8y ago
        I actually looked at this before making my other comment. You’re absolutely right about the salt. One of the go benchmarks is basically pure cgo using vector intrinsics. It naturally blows idiomatic JS out of the water. So you might get those kind of results if that’s the kind of code you’re going to write, but it’s far from what I consider idiomatic. Another language’s proponent would be fair to point out that most languages have C FFIs.
B-Scan 8y ago
Any plans to support the cloud endpoints with the standard env? Wondering if the new runtime would be able to be embed that. That would be only reason for me not to switch from flex env.
- asciimike 8y ago
  (GAE PM) Cloud Endpoints currently runs on GAE Standard Gen 1 and GAE Flex. We're investigating a solution for GAE Standard Gen 2 (which go 1.11 is, along with python 3.7, node 8, etc.) and GCF.
  - B-Scan 8y ago
    Thanks for the clarification. Wasn’t aware that it works with Go on Standard Gen 1. Any ETA for Gen 2 or alpha signup?
    - asciimike 8y ago
      Clarification: GAE Standard uses the Endpoints framework, which is Python and Java only (https://cloud.google.com/endpoints/docs/frameworks/about-cloud-endpoints-frameworks https://cloud.google.com/endpoints/docs/frameworks/about-clo...). Sorry for the confusion. Flex uses ESP (https://github.com/cloudendpoints/esp https://github.com/cloudendpoints/esp), which is deployed as a sidecar, and works with any runtime. As for EAP: unfortunately not. Personally, I'd like to see a managed version as opposed to the current framework or sidecar approach, as it'll be easier to implement across products, but that means it'll likely take a little longer.
warrentr 8y ago
How long will the legacy App Engine APIs continue to work? Will they be removed before the runtime is GA? These new runtimes seem amazing (and fix a lot of problems), but we also lose a ton of functionality that made app engine so desirable (easy users auth, images api, built in email sending, search, cron, and others). It kind of feels like we've thrown the baby out with the bathwater.
- steren 8y ago
  (App Engine PM here) The App Engine APIs will be present in the Go 1.11 runtime when it goes GA and until it is turned down. We do not know exactly which future version of the runtime will stop supporting them. We are working on Google Cloud standalone products to replace most missing features (e.g. Cloud Scheduler for cron jobs). Not all of them are ready yet. App Engine APIs were great when there was no alternative (when GCP did not exist, or when App Engine could not use arbitrary packages), but at the same time, contributed to the "lock-in" of App Engine, which was one of the main criticism. We believe it is in the long term benefit of our users to use standalone Google Cloud or third party services, instead of replying on APIs only accessible in App Engine.
  - warrentr 8y ago
    Thanks for the explanation. It makes sense and hopefully replacements are generally available before we lose the old apis.
Sir_Cmpwn 8y ago
Hey buss, thanks for stopping by. How do you feel about the distinction between Google and Go? It sometimes feels like they're joined at the hip, something which historically had made people queasy (Java and Oracle, .NET and Microsoft, etc). Though Go is a project which came from Google, I personally feel that it better serves the interests of the Go community to treat them separately. Google receives special treatment from Go - the announcement that AWS Lambda would support Go did not receive similar fanfare on golang.org. Side note: there is some activity on the Go blog which is more mutual cooperation and less Google hivemind: https://blog.golang.org/go-cloud https://blog.golang.org/go-cloud But I still wonder why this stuff belongs here.
- deleted 8y ago
  [deleted]
- justinclift 8y ago
  Agreed. To even submit content to the Go Blog requires a Google account. Ugh.
- jhabdas 8y ago
  Three smart guys made go. Google only has one or two left.
heromat 8y ago
I just tried to use go111, but I get the following message: >[7] Access Not Configured. Cloud Build has not been used in project <project> before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudbuild.googleapis.com/overview?project=<project> https://console.developers.google.com/apis/api/cloudbuild.go... then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry. So go111 cannot be used without Cloud Build, for which I have to activate billing?
- buss 8y ago
  Ah, yeah, looks like you have to activate billing. You'll be solidly within the free tier, though, so you won't be charged.
  - chrisbroadfoot 8y ago
    You can set a $0 limit on your billing account, too.
paulddraper 8y ago
I thought Go was statically compiled....i.e. it doesn't have a separate runtime from the program. Does anyone have any elucidation?
- chrisbroadfoot 8y ago
  Correct, Go's runtime is compiled into the program's binary. However, there's still an environment needed to run that binary. For Go on App Engine, this includes the sandbox (backed by gVisor) and the operating system (Ubuntu). We also tend to call everything in the toolchain the "runtime" - this includes the CLI for uploading and staging your app, and the builder used to compile your program. Not the strictest definition :) (GCP Gopher)
  - paulddraper 8y ago
    So App Engine is a build system as well?
    - chrisbroadfoot 8y ago
      Yes, with App Engine, you upload source code, and it's compiled for you remotely. For this newer runtime, it happens inside a Cloud Build invocation. For App Engine flexible, you can skip that build step and provide your own container, but that isn't possible for App Engine standard. (If you're interested in something like that, see https://g.co/serverlesscontainers https://g.co/serverlesscontainers)
openbasic 8y ago
Vendoring! Modules! Finally. Do you have any examples with proposed solutions for local module setups, without having to rely on GOPATH changes?
- TheDong 8y ago
  This is a go question, not an app engine question. See this page on the go wiki for an example of how to use go modules: https://github.com/kubernetes/kubernetes/pull/58098 https://github.com/kubernetes/kubernetes/pull/58098 Note: you need to have go1.11, and by default you need to be in a directory NOT in your $GOPATH for modules to work in go1.11
  - TheDong 8y ago
    Edit: https://github.com/golang/go/wiki/Modules#how-to-use-modules https://github.com/golang/go/wiki/Modules#how-to-use-modules Copy paste error I guess.
iamgopal 8y ago
Does context.background() works ?
- chrisbroadfoot 8y ago
  (GCP Gopher) Yes, you don't need to use the appengine context for anything (except to access the services at google.golang.org/appengine/...) You can use the standard net/http package to make HTTP requests, for example.
jostein 8y ago
This is a big improvement, but the restriction that prevents streaming responses unfortunately still apply: https://cloud.google.com/appengine/docs/standard/go/how-requests-are-handled#streaming_responses https://cloud.google.com/appengine/docs/standard/go/how-requ... Would be great if that was solved.

genericone 8y ago

Am I understanding this correctly, App Engine has made it so "go run goapp.go" on my local environment is basically accomplished with "runtime: goapp" in the app.yaml? No other changes necessary in the code?

buss 8y ago
runtime: go111, but otherwise -- yes!

bpye 8y ago

This actually seemed to be live a week or two ago, I managed to run Miniflux [1] on App Engine with no changes though to get it working properly it would need it's scheduling changing. [1] - https://github.com/miniflux/miniflux https://github.com/miniflux/miniflux

steren 8y ago
(App Engine PM here) Nice to hear! Yes indeed, we announced Go 1.11 on App Engine last week at Cloud Next London.
- bpye 8y ago
  Ah I hadn't seen that, I was quite surprised when I used goapp111 and it deployed successfully. It's great to be able to take a normal Go application and run it on Google Cloud. Thanks!

knodi 8y ago

I'm loving the new vendoring via go modules. Can't wait for it to be the default in Go 1.12

yegle 8y ago

Before anyone asks, this is still a beta release that's not covered by App Engine SLA.

dfischer 8y ago

Hmm, I was just running into some context annoyances but this seems to remedy some nuances regarding that. Cool! Good timing! I’m assuming urlfetcher is deprecated now too? I’ll go read the migration guide now ... Awesome work! Love Go and GAE! Very happy so far.

buss 8y ago
You can use net/http to fetch URLs now :) Glad you're enjoying both Go and GAE!
- dfischer 8y ago
  What was the reason urlfetcher had to be engineered in the past? Curious.
  - buss 8y ago
    I don't think I can say much more than: technical choices that required a different path to fetch resources from the broader internet.

deckarep 8y ago

Does this new classic+ environment now supersede the Flex environment? I’m curious because now it’s seems like there are 3 solutions. Although I really like the simplicity of this solution and the modernizing from the constraints of true classic mode.

buss 8y ago
Eh, not exactly supersede, but the runtimes are definitely slowly converging. It's a goal of mine to make them easy to switch between -- sometimes you'll want flex because you need a VM, most times you wont. But your code should be easily portable between them. With this release, they are nearly identical already. So two of the solutions are flex and go1.11 on standard. Is the third solution the older runtime versions (eg 1.9 and earlier)? If so, then there's really just two solutions, since 1.11 replaces 1.9 on standard.
chrisbroadfoot 8y ago
(GCP Gopher here) If you can run within the standard environment, I'd suggest you do (but you should check pricing, etc to see if it suits). Especially if you want 0-1 scaling. I migrated golang.org from standard to flexible a couple weeks ago, primarily because of increased control over the runtime (custom Docker image) and increased instance sizes (configurable memory, CPU past the App Engine limits, which caps out at 2 GB). See https://golang.org/issues/27205 https://golang.org/issues/27205 if you want to read more.
- iamgopal 8y ago
  Do you have blog post with data, number of instances etc previously and now ?
  - chrisbroadfoot 8y ago
    I don't, sorry. While the app was unhealthy, it was spinning up dozens of instances. While it was healthy (on standard), I recall it ran on less than a handful of instances. It currently runs on two fat flexible instances (GCE VM), but could easily run on one. We only run on two for some redundancy. I've measured throughput of ~thousands of requests per second on a F4_1G on apps I've worked on previously. They were CPU-constrained, but not doing very much (JSON serialization). The new runtime might be even better, since GOMAXPROCS is no longer 1. correction to my post above: 1 GB is the max memory for standard: https://cloud.google.com/appengine/docs/standard/#instance_classes https://cloud.google.com/appengine/docs/standard/#instance_c...

frakkingcylons 8y ago

I'm not trying to stir shit up...buuut it seems odd that an announcement for a Google Cloud product is being published on the Go project's blog? I say this as a huge fan of Google Cloud and the experience of using Go on App Engine.

justinclift 8y ago
Likely because it's clearly Go related? "Announcing [some cloud]’s New Go 1.11 Runtime" That being said, I have no idea how similar submissions of Go content from competing vendors would er... go. ;)

ptype 8y ago

thstart 8y ago

I have some C++ libraries called from Go - can I use them now?

helpfulgoogler 8y ago
cgo is supported.

deleted 8y ago

[deleted]

xaduha 8y ago

I haven't used GAE since early days, but I heard it supports Docker now. Why wouldn't I just use Docker with whatever runtime I want?

chrisbroadfoot 8y ago
(GCP Gopher) We support custom images for the flexible runtime, but not for the standard runtime. If you're as keen for that as I am, sign up here: http://g.co/serverlesscontainers http://g.co/serverlesscontainers

XorNot 8y ago

Is this a GA product or still "beta"? Google's consistent announcement of products as available while in beta is getting annoying, because their SLAs for "beta" products are still 0 and thus impossible to be able to use in actual business applications.

chrisbroadfoot 8y ago
(GCP Gopher) Yes, it's beta. You're right - if you require an SLA, don't use beta products. As per https://cloud.google.com/terms/launch-stages https://cloud.google.com/terms/launch-stages: > Beta is the point at which we are ready to open a release for any customer to use. There are no SLA or technical support obligations in a Beta release, and charges may be waived in some cases. Products will be complete from a feature perspective, but may have some open outstanding issues. Beta releases are suitable for limited production use cases.

nixarn 8y ago

Wonder if it's still possible to get answers here :) giving it a shot although it's 13h old by now. We use Go for a production server for multiple of our games, it's used to sync gamestates between devices and allows us to remotely inspect and change player's gamestate. Super handy and it's worked flawlessly! We used to use python, but Go is much better suited as it's less error prone. One question though, we've been having connectivity issues in China, the GAE is deployed in the US, and as it seems there's no multiregional support. What can we do? Ideally keeping as much of the convenience of the GAE as possible.