6 ms·

Python 3 on Google App Engine flexible environment now in beta

105 points by arouzrokh 10y ago

jo909 10y ago
I'm really glad this big fat blemish is now finally going away. Anybody has a reasonable theory for why this took so long, despite users asking for it for years?
- daveguy 10y ago
  I think you misread it. Unfortunately it looks like Python 3 is not going away. You did say big fat blemish, right?
  - advisedwang 10y ago
    I think the "lack of Python 3 support" is the blemish referred to, not Python 3 itself.
    - deleted 10y ago
      [deleted]
    - _quick_q 10y ago
      I think you may have missed the sarcasm in the parent post.
- dragonwriter 10y ago
  > Anybody has a reasonable theory for why this took so long, despite users asking for it for years? Well, initially, their investment in more runtimes was covering additional languages, not covering more versions of languages they already had. (IIRC, AE started with Python 2, added Java first, and then later Go and PHP.) As that was happening, they gradually shifted focus from building out the old, fairly limited model of App Engine to building out other parts of the cloud platform, including (particularly relevant to future App Engine work) Compute Engine, to have a more robust cloud offering covering more use cases. Then, they went back and returned to improving App Engine -- leveraging the now-broader cloud platform, and particularly Compute Engine -- shifting from the old model for new runtimes to the Flexible Environment (originally called "Managed VMs") model (built on Compute Engine, and which, in addition to including some additional Google-provided runtimes, means people can package up their own runtimes for App Engine). Since doing so, they've released official Google runtimes for Python 3, Ruby (which people have been asking for longer than Python 3), and Node built on the Flexible Environment.
  - jo909 10y ago
    The architectural switch to the "Flexible Environment" is what I missed. I can appreciate why this broader new approach for language runtimes took some time. But then again the Flexible Environment seems to be a very different product with different features/characteristics and pricing than what I once knew as "App Engine". I would then argue that Python 3 never came to "App Engine", only to their new product. Which is a valid strategy for Google of course, but now stifled my enthusiasm. Probably I only need to learn more about Flexible Environment style App Engine to become hyped again.
    - lern_too_spel 10y ago
      Flexible environments are significantly worse if your application works on the original style of app engine.
      - vsrinivas 10y ago
        Just wondering, why do you think so? (It wasn't obvious to me why, given a first read of the docs; the fact that applications run in VMs in Flex seems more like an implementation detail. Am I missing something?)
        kujenga 10y ago
        We use both and try to stick with standard runtimes wherever possible, resorting to flexible runtimes for things doing heavier processing and/or with lots of instances. The standard runtimes are superior in a lot of ways, two big wins being lightning fast deployments (seconds vs. many minutes) and batteries included for all the app engine services. Logging in the flexible environment can also be a bit lacking. When you have something running in standard runtimes you basically don't have to think about many of the things that can normally go wrong on a server. That said, many things about the flexible environment are rapidly improving, they've certainly come a long way since we first started playing with them around 6 months ago.
- theptip 10y ago
  Especially amusing was that Guido was working on the GAE team for a while, and even that wasn't enough to get them to support Python3.
  - brianwawok 10y ago
    I assume that was depressing.
  - branchly2 10y ago
    I wouldn't say amusing, though it is interesting, and taking note of Go and Dart, it may reflect what role Google thinks Python plays.
- cyberpanther 10y ago
  I think this is my main problem with Google services. It takes them forever to listen to people. And I don't think other cloud services listen any better but at least it seems they give you what you want faster. So they have a better mechanism to know what works and making a product of it.
rch 10y ago
> you can use the tools and databases you already know and love But not Postgres or Redis? Edit: OK, I'm seeing some references in the docs. Still seems limited to deployment on generic instances though.
- advisedwang 10y ago
  You can absolutely use Postgres or Redis. If you spin them up on GCE you can connect to it from GAE Flexible. The only support lacking is a managed Postgres or Redis service like Cloud SQL does for MySQL.
  - rch 10y ago
    Granted, but it still makes for a salient point when comparing RDS to GCE.
- jonwayne 10y ago
  You can use Postgres and Redis. :) I'm working on writing new tutorials for both of those, but in the meantime: You can host your own Postgres on Compute Engine [1] or use a hosted provider like ElephantSQL [2]. You can use whatever client library you want to connect to postgres in App Engine flexible. Same story for redis - you can host it on Compute Engine [3] or use something like redislabs. [1] https://cloud.google.com/solutions/set-up-postgres https://cloud.google.com/solutions/set-up-postgres [2] https://www.elephantsql.com/ https://www.elephantsql.com/ [3] https://cloud.google.com/launcher/?q=redis https://cloud.google.com/launcher/?q=redis [4] https://redislabs.com/ https://redislabs.com/ Note: I'm in Developer Relations for Google Cloud Platform
  - rch 10y ago
    Both of those third party providers look interesting. Thanks for pointing them out. Edit: I notice that ElephantSQL doesn't surface in a partner search but EnterpriseDB does. https://cloud.google.com/partners/#?q=postgresql https://cloud.google.com/partners/#?q=postgresql EnterpriseDB seems relatively expensive, even when just getting started (Estimated costs: $891.00/month). https://console.cloud.google.com/launcher/details/public-edb-ppas/edb-postgres-enterprise https://console.cloud.google.com/launcher/details/public-edb... Redis is provided by Click to Deploy, i.e. "Popular open stacks on Google Compute Engine packaged by Google", and is in beta (Estimated costs: $307.80/month). https://console.cloud.google.com/launcher/details/click-to-deploy-images/redis https://console.cloud.google.com/launcher/details/click-to-d... Looks like Redis Labs has a trivial free tier, and the 1GB plan is cheaper on GCE than AWS ($57/mo vs. $71/mo). I'm pretty annoyed that I had to create a Redis Labs account to see that though.
ben_jones 10y ago
Really cool. Would be interesting if App Engine becomes a spot for larger framework driven web apps like Django, or if/when ruby is supported, ROR. It could be a very efficient way of doing so because you could have a single instance running with a gazillion gunicorn/unicorn worker processes to distribute the load across processes.
- dragonwriter 10y ago
  > if/when ruby is supported Ruby (and Node) are supported on the Flex Env the same way Python 3 is. JRuby may also be usable on the standard environment's Java runtime (ISTR seeing a how-to in the past, but I also seem to recall at one point seeing that there were some issues.)
- rrdharan 10y ago
  Note that Django has worked on App Engine standard for many years now and in fact the App Engine SDK still distributes a Django build. More info on the different Django+AppEngine options here: https://cloud.google.com/python/django/ https://cloud.google.com/python/django/ https://code.djangoproject.com/wiki/AppEngine https://code.djangoproject.com/wiki/AppEngine Disclosure: I'm an engineer at Google.
coldtea 10y ago
This changes everything! Just kidding. Python 3 adoption will continue to be slow, and judging from PyPI stats, nothing to write home about even atm.
- dswalter 10y ago
  I wish there were a clearer delineation between developer downloads on PyPI and automated builds, since it seems there's been a bit of a cultural sea change over the past year or so in favor of Py3. I'd love to know which version humans are downloading.
  - coldtea 10y ago
    Wouldn't Python 3 ALSO have installations doing automated builds though?
    - gbin 10y ago
      Yes but if automated pip installs numbers are way larger than manual installs you end up with projects supporting both with roughly 50-50 numbers. I agree with the parent post: now is the time. Open source projects should to stop supporting py2 in their new major release and cleanup their codebase from the hacks made to support both languages.
      - coldtea 10y ago
        >I agree with the parent post: now is the time. Open source projects should to stop supporting py2 in their new major release and cleanup their codebase from the hacks made to support both languages. The danger with this for Python is that it can give the impetus (is that the word?) and motivation for people with py2 legacy projects (which is the majority of people) to say "fuck this" and go try another language that doesn't pull this shit on them. Go, JS with Node, Ruby, whatever... If they're not gonna get compatibility anyway, they might jump ship altogether. At least for green-field projects...
borplk 10y ago
I recently signed up for a trial and tried to use Google Cloud Platform. Oh my god the UI was amongst the worst I have ever experienced on the web. Super slow and hard to navigate. Stupid "tutorial". I tried for a good two hours, tried to accept it but finally gave up on it. Google, cut down on your Angular and material design kool aid and try harder next time.
- matt4077 10y ago
  I felt Google was much easier to get started in than AWS. And it's easy to forget that it's quite possibly the most complicated system most people who use it interact with. It probably has a few magnitudes more expressive power of a, say, a webpack+babel+package.config (yeah, hard to measure, I know). And that's a competition it easily wins.
- waprin 10y ago
  Developer Relations team exists so that doesn't happen so sorry to hear. Part of the problem is some of our feedback mechanisms need work. If you go to: cloud.google.com/python You'll see we have a lot of tutorials that link to Github repos . If you file issues there you are more likely to get help. The caveat is we are not a support team and have no guarantees (just a likelihood), so if you need paid support look into our options for that instead.
- brianwawok 10y ago
  Have you compared it to AWS?
  - borplk 10y ago
    Yes I have used AWS a lot and been happy with the UI, it gets the job done. Google Cloud Platform UI (and speed) was just actively hostile towards me. For example I went to the App Engine section to create a simple flask app. It keeps forcing you through a stupid tutorial trying to guide you around the site. Believe me being a developer myself I'm no stranger to web and its interfaces and can quickly navigate my way around this stuff but this had me hitting my head on the desk.
pokstad 10y ago
Does anyone have any numbers on the cost difference between standard vs flexible app engine usage? Flexible environment supports any language/environment you want as long as you implement the HTTP endpoints it expects. I would be much more impressed if they brought Python3 to the standard environment which seems to be much more cost effective for many types of web apps. Flexible environment essentially requires you to run a VM all the time, which can get pricey, versus a much more managed sandboxed environment that can sleep and be awaken quickly as needed to serve requests. Also, Python 3.5 would be awesome to leverage asyncio, but that's probably still a bit too bleeding edge.
- dabeeeenster 10y ago
  On lowish traffic apps, it's way way more expensive as you have to pay for the VM's to be permanently running. AFAIK you also need to have 2 app servers running as they get restarted once w eek (at least thats how it is for Java apps I believe)
- alooPotato 10y ago
  If you have very little traffic then what you are saying is true, flex GAE requires you to keep one VM running. For our our app (Streak) we run 100's of instances, we moved some of our services from standard GAE to flex GAE and saw a HUGE reduction in costs. Depending on workload we saw 5x-10x reduction in cost. To see why this is possible, look at the price of a standard GAE instance and the price of a default flex instance. For roughly the same price you get a way more capable machine. You just have to have enough traffic where you'd like to keep at least one instance running all the time. Enough traffic in this case is a really small number. If you're traffic is lower than that number than who cares about optimizing costs, its gonna be cheap no matter what.
- kujenga 10y ago
  We use a combination of both at my company. We use the standard environment for the majority of our services, doing things like proving APIs, serving webpages, performing datastore interactions, etc. The flexible environment starts to make a lot of sense when you're doing heavier weight processing, text analytics in our case. App Engine instances are relatively cheap when you're using the smallest instance sizes, and have even more benefits if your application has idle time when they can spin down completely, but costs start to go up very quickly if you need to move to the bigger instances in the standard environment [1]. The flexible environment is current billed at the rate of the machines you're using, so is much cheaper for bigger instances [2]. [1] https://cloud.google.com/appengine/pricing#standard_instance_pricing https://cloud.google.com/appengine/pricing#standard_instance... [2] https://cloud.google.com/compute/pricing#predefined_machine_types https://cloud.google.com/compute/pricing#predefined_machine_...

105 points by arouzrokh 10y ago

jo909 10y ago

I'm really glad this big fat blemish is now finally going away. Anybody has a reasonable theory for why this took so long, despite users asking for it for years?

daveguy 10y ago
I think you misread it. Unfortunately it looks like Python 3 is not going away. You did say big fat blemish, right?
- advisedwang 10y ago
  I think the "lack of Python 3 support" is the blemish referred to, not Python 3 itself.
  - deleted 10y ago
    [deleted]
  - _quick_q 10y ago
    I think you may have missed the sarcasm in the parent post.
dragonwriter 10y ago
> Anybody has a reasonable theory for why this took so long, despite users asking for it for years? Well, initially, their investment in more runtimes was covering additional languages, not covering more versions of languages they already had. (IIRC, AE started with Python 2, added Java first, and then later Go and PHP.) As that was happening, they gradually shifted focus from building out the old, fairly limited model of App Engine to building out other parts of the cloud platform, including (particularly relevant to future App Engine work) Compute Engine, to have a more robust cloud offering covering more use cases. Then, they went back and returned to improving App Engine -- leveraging the now-broader cloud platform, and particularly Compute Engine -- shifting from the old model for new runtimes to the Flexible Environment (originally called "Managed VMs") model (built on Compute Engine, and which, in addition to including some additional Google-provided runtimes, means people can package up their own runtimes for App Engine). Since doing so, they've released official Google runtimes for Python 3, Ruby (which people have been asking for longer than Python 3), and Node built on the Flexible Environment.
- jo909 10y ago
  The architectural switch to the "Flexible Environment" is what I missed. I can appreciate why this broader new approach for language runtimes took some time. But then again the Flexible Environment seems to be a very different product with different features/characteristics and pricing than what I once knew as "App Engine". I would then argue that Python 3 never came to "App Engine", only to their new product. Which is a valid strategy for Google of course, but now stifled my enthusiasm. Probably I only need to learn more about Flexible Environment style App Engine to become hyped again.
  - lern_too_spel 10y ago
    Flexible environments are significantly worse if your application works on the original style of app engine.
    - vsrinivas 10y ago
      Just wondering, why do you think so? (It wasn't obvious to me why, given a first read of the docs; the fact that applications run in VMs in Flex seems more like an implementation detail. Am I missing something?)
      - kujenga 10y ago
        We use both and try to stick with standard runtimes wherever possible, resorting to flexible runtimes for things doing heavier processing and/or with lots of instances. The standard runtimes are superior in a lot of ways, two big wins being lightning fast deployments (seconds vs. many minutes) and batteries included for all the app engine services. Logging in the flexible environment can also be a bit lacking. When you have something running in standard runtimes you basically don't have to think about many of the things that can normally go wrong on a server. That said, many things about the flexible environment are rapidly improving, they've certainly come a long way since we first started playing with them around 6 months ago.
theptip 10y ago
Especially amusing was that Guido was working on the GAE team for a while, and even that wasn't enough to get them to support Python3.
- brianwawok 10y ago
  I assume that was depressing.
- branchly2 10y ago
  I wouldn't say amusing, though it is interesting, and taking note of Go and Dart, it may reflect what role Google thinks Python plays.
cyberpanther 10y ago
I think this is my main problem with Google services. It takes them forever to listen to people. And I don't think other cloud services listen any better but at least it seems they give you what you want faster. So they have a better mechanism to know what works and making a product of it.

rch 10y ago

> you can use the tools and databases you already know and love But not Postgres or Redis? Edit: OK, I'm seeing some references in the docs. Still seems limited to deployment on generic instances though.

advisedwang 10y ago
You can absolutely use Postgres or Redis. If you spin them up on GCE you can connect to it from GAE Flexible. The only support lacking is a managed Postgres or Redis service like Cloud SQL does for MySQL.
- rch 10y ago
  Granted, but it still makes for a salient point when comparing RDS to GCE.
jonwayne 10y ago
You can use Postgres and Redis. :) I'm working on writing new tutorials for both of those, but in the meantime: You can host your own Postgres on Compute Engine [1] or use a hosted provider like ElephantSQL [2]. You can use whatever client library you want to connect to postgres in App Engine flexible. Same story for redis - you can host it on Compute Engine [3] or use something like redislabs. [1] https://cloud.google.com/solutions/set-up-postgres https://cloud.google.com/solutions/set-up-postgres [2] https://www.elephantsql.com/ https://www.elephantsql.com/ [3] https://cloud.google.com/launcher/?q=redis https://cloud.google.com/launcher/?q=redis [4] https://redislabs.com/ https://redislabs.com/ Note: I'm in Developer Relations for Google Cloud Platform
- rch 10y ago
  Both of those third party providers look interesting. Thanks for pointing them out. Edit: I notice that ElephantSQL doesn't surface in a partner search but EnterpriseDB does. https://cloud.google.com/partners/#?q=postgresql https://cloud.google.com/partners/#?q=postgresql EnterpriseDB seems relatively expensive, even when just getting started (Estimated costs: $891.00/month). https://console.cloud.google.com/launcher/details/public-edb-ppas/edb-postgres-enterprise https://console.cloud.google.com/launcher/details/public-edb... Redis is provided by Click to Deploy, i.e. "Popular open stacks on Google Compute Engine packaged by Google", and is in beta (Estimated costs: $307.80/month). https://console.cloud.google.com/launcher/details/click-to-deploy-images/redis https://console.cloud.google.com/launcher/details/click-to-d... Looks like Redis Labs has a trivial free tier, and the 1GB plan is cheaper on GCE than AWS ($57/mo vs. $71/mo). I'm pretty annoyed that I had to create a Redis Labs account to see that though.

ben_jones 10y ago

Really cool. Would be interesting if App Engine becomes a spot for larger framework driven web apps like Django, or if/when ruby is supported, ROR. It could be a very efficient way of doing so because you could have a single instance running with a gazillion gunicorn/unicorn worker processes to distribute the load across processes.

dragonwriter 10y ago
> if/when ruby is supported Ruby (and Node) are supported on the Flex Env the same way Python 3 is. JRuby may also be usable on the standard environment's Java runtime (ISTR seeing a how-to in the past, but I also seem to recall at one point seeing that there were some issues.)
rrdharan 10y ago
Note that Django has worked on App Engine standard for many years now and in fact the App Engine SDK still distributes a Django build. More info on the different Django+AppEngine options here: https://cloud.google.com/python/django/ https://cloud.google.com/python/django/ https://code.djangoproject.com/wiki/AppEngine https://code.djangoproject.com/wiki/AppEngine Disclosure: I'm an engineer at Google.

coldtea 10y ago

This changes everything! Just kidding. Python 3 adoption will continue to be slow, and judging from PyPI stats, nothing to write home about even atm.

dswalter 10y ago
I wish there were a clearer delineation between developer downloads on PyPI and automated builds, since it seems there's been a bit of a cultural sea change over the past year or so in favor of Py3. I'd love to know which version humans are downloading.
- coldtea 10y ago
  Wouldn't Python 3 ALSO have installations doing automated builds though?
  - gbin 10y ago
    Yes but if automated pip installs numbers are way larger than manual installs you end up with projects supporting both with roughly 50-50 numbers. I agree with the parent post: now is the time. Open source projects should to stop supporting py2 in their new major release and cleanup their codebase from the hacks made to support both languages.
    - coldtea 10y ago
      >I agree with the parent post: now is the time. Open source projects should to stop supporting py2 in their new major release and cleanup their codebase from the hacks made to support both languages. The danger with this for Python is that it can give the impetus (is that the word?) and motivation for people with py2 legacy projects (which is the majority of people) to say "fuck this" and go try another language that doesn't pull this shit on them. Go, JS with Node, Ruby, whatever... If they're not gonna get compatibility anyway, they might jump ship altogether. At least for green-field projects...

borplk 10y ago

I recently signed up for a trial and tried to use Google Cloud Platform. Oh my god the UI was amongst the worst I have ever experienced on the web. Super slow and hard to navigate. Stupid "tutorial". I tried for a good two hours, tried to accept it but finally gave up on it. Google, cut down on your Angular and material design kool aid and try harder next time.

matt4077 10y ago
I felt Google was much easier to get started in than AWS. And it's easy to forget that it's quite possibly the most complicated system most people who use it interact with. It probably has a few magnitudes more expressive power of a, say, a webpack+babel+package.config (yeah, hard to measure, I know). And that's a competition it easily wins.
waprin 10y ago
Developer Relations team exists so that doesn't happen so sorry to hear. Part of the problem is some of our feedback mechanisms need work. If you go to: cloud.google.com/python You'll see we have a lot of tutorials that link to Github repos . If you file issues there you are more likely to get help. The caveat is we are not a support team and have no guarantees (just a likelihood), so if you need paid support look into our options for that instead.
brianwawok 10y ago
Have you compared it to AWS?
- borplk 10y ago
  Yes I have used AWS a lot and been happy with the UI, it gets the job done. Google Cloud Platform UI (and speed) was just actively hostile towards me. For example I went to the App Engine section to create a simple flask app. It keeps forcing you through a stupid tutorial trying to guide you around the site. Believe me being a developer myself I'm no stranger to web and its interfaces and can quickly navigate my way around this stuff but this had me hitting my head on the desk.

pokstad 10y ago

Does anyone have any numbers on the cost difference between standard vs flexible app engine usage? Flexible environment supports any language/environment you want as long as you implement the HTTP endpoints it expects. I would be much more impressed if they brought Python3 to the standard environment which seems to be much more cost effective for many types of web apps. Flexible environment essentially requires you to run a VM all the time, which can get pricey, versus a much more managed sandboxed environment that can sleep and be awaken quickly as needed to serve requests. Also, Python 3.5 would be awesome to leverage asyncio, but that's probably still a bit too bleeding edge.

dabeeeenster 10y ago
On lowish traffic apps, it's way way more expensive as you have to pay for the VM's to be permanently running. AFAIK you also need to have 2 app servers running as they get restarted once w eek (at least thats how it is for Java apps I believe)
alooPotato 10y ago
If you have very little traffic then what you are saying is true, flex GAE requires you to keep one VM running. For our our app (Streak) we run 100's of instances, we moved some of our services from standard GAE to flex GAE and saw a HUGE reduction in costs. Depending on workload we saw 5x-10x reduction in cost. To see why this is possible, look at the price of a standard GAE instance and the price of a default flex instance. For roughly the same price you get a way more capable machine. You just have to have enough traffic where you'd like to keep at least one instance running all the time. Enough traffic in this case is a really small number. If you're traffic is lower than that number than who cares about optimizing costs, its gonna be cheap no matter what.
kujenga 10y ago
We use a combination of both at my company. We use the standard environment for the majority of our services, doing things like proving APIs, serving webpages, performing datastore interactions, etc. The flexible environment starts to make a lot of sense when you're doing heavier weight processing, text analytics in our case. App Engine instances are relatively cheap when you're using the smallest instance sizes, and have even more benefits if your application has idle time when they can spin down completely, but costs start to go up very quickly if you need to move to the bigger instances in the standard environment [1]. The flexible environment is current billed at the rate of the machines you're using, so is much cheaper for bigger instances [2]. [1] https://cloud.google.com/appengine/pricing#standard_instance_pricing https://cloud.google.com/appengine/pricing#standard_instance... [2] https://cloud.google.com/compute/pricing#predefined_machine_types https://cloud.google.com/compute/pricing#predefined_machine_...