8 ms·

Google is working on Python 3 for App Engine Standard Environment

159 points by ognyankulev 8y ago

shaohua 8y ago
at 10 years birthday of app engine and 10 years birthday of python3
- sago 8y ago
  I like to remind people that Python 3 was already more than a year old when Apple announced the first iPad. I'm not sure anyone should be getting a cookie for 'working on' Py3 support more than nine years later. Though I do understand that GAE has supported Py 3 for a while in its more configurable service.
  - raverbashing 8y ago
    True, though I have to say it only got usable around 3.4 (which still is 4y.o. at this point)
    - sago 8y ago
      What did 3.4 give you that 2.7 already had? I have found bundled pip and asyncio useful, but I was using Py3 from 3.2, successfully. It didn't suddenly become usable at 3.4 for me.
      - raverbashing 8y ago
        Yeah, pip. You could install it manually, but to be honest, pip and virtualenv are fundamental today (I mean, before containers, but still)
      - toyg 8y ago
        I think what happened was that 3.2 was the first "real" release, when a lot of .0 bugs were shaken out; the 'u' prefix was added back around that time too, iirc. This resulted in more developers actually porting their libraries, an effort that had been basically non-existent before. The result was that the 3.4 ecosystem actually got mature enough that Real Work could get done, and more end-users started moving. IIRC 3.3 also had some big performance problems that were sorted in 3.4, but I cannot remember the details. Personally, I did very little with 3.2 but actually built Real Stuff only with 3.4.
- yegle 8y ago
  To be fair: PEP 3333 that addresses WSGI support with Python 3 Unicode was initially drafted in September 2010. I would consider that to be the first day we can write a web server in Python 3.
Godel_unicode 8y ago
Reminder that Python 2 dies in 18 months. https://pythonclock.org https://pythonclock.org
- yorwba 8y ago
  It will likely remain around for longer, since depending on unsupported software doesn't immediately kill a project. When old Python 2 code hits a bug in the CPython interpreter and nobody is around to fix it, there'll finally be an incentive to migrate away (maybe to Python 3, maybe to another language), but not before that.
  - Narkov 8y ago
    True, however lots of large corporates won't use an unsupported version if the vendor drops it.
    - candiodari 8y ago
      There, the vendor will be redhat or ... who will be supporting it longer. What enterprises mean by support isn't provided by the PSF anyway. They support their products 10 years, or indefinitely if you pay extra for it, so Redhat will support Python 2 until June 30, 2024. If I understand it correctly, that means CentOS will too.
  - u801e 8y ago
    > It will likely remain around for longer, since depending on unsupported software doesn't immediately kill a project. It also depends on how long companies like Redhat support it (CentOS 6 comes with Python 2.6 and CentOS 7 comes with Python 2.7).
- acqq 8y ago
  I wouldn't be surprised that vendors will still have to do the fixes to Python 2 for many years past that date, and that a kind of "fork" will keep the 2.7 branch decently alive much longer. It's just the current "official" maintainers that announce the cutoff date for their work.
berti 8y ago
For anyone confused, currently you have a choice of either "Standard Environment", which is Python 2.7 only, or "Flexible Environment" which allows you to choose Python 2.7, Python 3.6, or your own Python runtime [1][2]. This is about supporting Python 3 in the "Standard Environment". It's unclear whether the intention is to replace Python 2 or allow a choice? [1] https://cloud.google.com/appengine/docs/python/ https://cloud.google.com/appengine/docs/python/ [2] https://cloud.google.com/appengine/docs/flexible/python/runtime https://cloud.google.com/appengine/docs/flexible/python/runt...
- ognyankulev 8y ago
  tl;dr Standard environment apps "scale nearly instantaneously to handle huge traffic spikes", "cost-effective for applications that have significant periods where they are not serving traffic", use reasonably limited standard API + proprietary API
  - dudus 8y ago
    More importantly standard scales down to 0 while flexible needs a minimum of 1 instance running at all times. So when just prototyping/developing or creating a small app only you or a few users will use it's very cost effective to use standard.
cheriot 8y ago
I love the new movement in on GAE Standard. With GAE Flexible, GKE, Cloud SQL, etc there's actually a growth path for something that out grows it. When Standard first launched, that additional complexity meant a rewrite. Can the Java runtime implement the full Servlet 3.1 spec now? It's 5 years old (and most Scala http libs don't support blocking Servlets anymore).
privateSFacct 8y ago
I never get this about google. They had a great product here (I and I know others actively interested) but they just drop development on these things for the new shiny toy. We're not asking for a ton, but if you could assign even a few folks to maintain the non-cool stuff, I think a lot more folks would build on GCP. I'm on AWS currently. They keep their old / uncool stuff around in a nice way (and totally seem to drop ball on other stuff that matters less like their new Amazon Linux 2 effort - python 3 support in that is totally wonky etc).
- yegle 8y ago
  Feature development of GAE (especially the Standard Environment) never stopped. Some of the features require multi quarters of work (e.g. https://cloudplatform.googleblog.com/2018/05/Increase-performance-while-reducing-costs-with-the-new-App-Engine-scheduler.html https://cloudplatform.googleblog.com/2018/05/Increase-perfor...), some are more enterprise-focused, some are not user visible but we improve the reliability with a new stack behind the curtain. We are changing the engine while flying (pun not intended). For various reasons, supporting new runtimes was never an easy task, even for small version bumps. This has changed drastically, starting with Java8 runtime, to the recently announced NodeJS beta, and to the closed alpha of Python 3 (remember this is Python 3.7). If you followed recent announcements from Google you probably can connect the dots. I think we are pretty open on what we are working on to our customers, as long as an NDA is signed. If you are interested be sure to let our cloud support team know! Disclaimer: I'm on App Engine SRE team.
  - mijamo 8y ago
    I mean things have been pretty bad for GAE standard in Python for a very long time. It looks so dead I was actually expecting Google to completely remove it in 2 years or so. I work on a project on GAE standard and the platform was chosen before I started. I would say this was the worst possible solution to chose and the biggest cost for us right now. The most annoying point is that it is actually not easy at all to switch to the Flexible environment, because libraries are completely different. Examples of troubles : - new google-cloud libraries run in the flexible environment but not in the standard one (ex: google-cloud-pubsub doesn't run in the standard env after 0.27, and the doc for 0.27 is dead: https://google-cloud-python.readthedocs.io/en/0.27.0/pubsub/index.html https://google-cloud-python.readthedocs.io/en/0.27.0/pubsub/...) - traditional standard env libraries do not run in the flexible env. The most typical example is NDB for the Google datastore. Which means that migrating from the standard to the flexible env requires you to rebuild EVERYTHING that touches Google services... Great, then I'd better just switch to AWS instead! What's moer, flexible env is nice but for some reason it is not possible to have basic scaling anymore, nor is it possible to scale down to 0 when needed, which makes modular apps much more costly (previously we would have several services and when unused they just scaled down to 0). On top of that we just watched services get deprecated one after the other (mail service, then the datastore backup etc.) and old troubles not getting fixed (like memory problems with queries).
olskool 8y ago
I tried Appengine when it first came out but Google never seemed to support it much so I gave up on it.
k__ 8y ago
What lead to this whole Python 2 and 3 divide? I mean, other languages have new versions too...
- ptx 8y ago
  Python 1 didn't know about Unicode, so characters were bytes and things were simple (for latin1 users). Python 2 bolted on Unicode support, so now you had two kinds of strings (old-style strings and Unicode strings) with implicit conversions between them (that would blow up spectacularly when you least expected it) and overall a huge mess. Python 3 removes the old-style strings (replacing them with the "bytes" type and explicit conversions), so that we once again have only one type of string, and once again things are simple. This clean-up unfortunately means that code relying the Python 2 string type confusion will break.
  - k__ 8y ago
    Well, at least someone tried to clean things up. Other languages simply carry these bugs around forever.
- dwheeler 8y ago
  The transition from Python 2 into Python 3 was badly botched, and for many years. First of all, they confused language with implementation, so the Python 3 interpreter cannot handle the Python 2 language while the Python2 interpreter cannot handle the Python 3 language (I refer to the main C interpreter, not others like PyPy). The automated tool "2to3" wasn't reliable on real-world programs, either. For many years it was impractical to write programs or libraries that worked on both 2 and 3. To write any Python 3 code, ALL of your transitively-dependent libraries had to convert to Python 3, and effectively simultaneously. By itself, that was rediculous. In addition, Python 3 doesn't add any radically new capabilities over Python 2, so it was basically all pain and no gain. Python 3's new approach to strings is conceptually clean, but it struggles with the messiness of the real world and the Python 3 developers had to add a lot of hacks to make it sort-of work. For example, file names are not strings of characters on Unix/Linux, there is no standard encoding for stdin, and real-world data doesn't always follow any particular encoding and there's often no need to require it. Python 3 was effectively ignored by most programmers for at least 5 years after its release, because it created more problems than it solved. What changed is that Python 3 was modified, and libraries were added to both Python 2 and 3, so that it started to became practical to write code that could work on both 2 and 3. The result: instead of having to transition your entire program and all its dependencies simultaneously, people could work incrementally, making little improvements here and there so that programs would keep working in 2, and eventually work in 3 as well. Various hacks were also created so that Python 3 could handle stuff like filenames (!). It would have been much better had the Python 3 developers made it easier to transition from 2 to 3 from the start, but it's finally easier. It's still somewhat painful; mercurial still hasn't transitioned, and it's not clear that they ever will. I'm unhappy about what the Python developers did earlier, but I think they've made good strides in trying to finally make it much easier to transition from Python 2 to Python 3. The main takeaway here should be, "make it as PAINLESS as POSSIBLE for your existing users to transition to a new version". It's okay to change APIs, but use different names or use versioned APIs. It's okay to change other things, but make it easy for people to modify parts of their program at a time. You'd think that'd be obvious, but it's clear that really smart people don't always think about that.
savrajsingh 8y ago
Appengine standard is amazing. I hope they continue to support 2.7 as well.
oooooof 8y ago
Google also committed to releasing an update to its App Engine Gw Basic and Turbo Pascal environments, “some time in the 2021 timeframe”.
mikece 8y ago
Just in case you're like me and had to Bing what the difference is between Docker and Google App Engine, it's summed up really well in this Stack Overflow answer: https://stackoverflow.com/a/49950687/3799 https://stackoverflow.com/a/49950687/3799 I wonder, though, if future development should target Docker/K8S and GAE should only be the domain of legacy applications.
- deklerk 8y ago
  > the difference is between Docker and Google App Engine Neither the question nor the answer in your link compare Docker and App Engine. Docker is a containerization platform, whereas App Engine is a PaaS (platform as a service). Their functions are wholly different. Very, very loosely; Docker is about building and running an individual package of some software. Basically, bundling software and running bundles. PaaS are platforms for developing, running, and managing (often pre-bundled) software.
  - mikece 8y ago
    Would it be more accurate to compare Google App Engine (PaaS) to Azure Web Apps or App Service (also PaaS)? Docker is in a weird spot of not really being PaaS but yet too light to be considered IaaS... I tend to think of it as sufficiently PaaS-like except it runs on everyone's cloud as well as my laptop.
IshKebab 8y ago
Yet another example to throw in the face of people who say that the Python 2/3 problem is solved.

159 points by ognyankulev 8y ago

shaohua 8y ago

at 10 years birthday of app engine and 10 years birthday of python3

sago 8y ago
I like to remind people that Python 3 was already more than a year old when Apple announced the first iPad. I'm not sure anyone should be getting a cookie for 'working on' Py3 support more than nine years later. Though I do understand that GAE has supported Py 3 for a while in its more configurable service.
- raverbashing 8y ago
  True, though I have to say it only got usable around 3.4 (which still is 4y.o. at this point)
  - sago 8y ago
    What did 3.4 give you that 2.7 already had? I have found bundled pip and asyncio useful, but I was using Py3 from 3.2, successfully. It didn't suddenly become usable at 3.4 for me.
    - raverbashing 8y ago
      Yeah, pip. You could install it manually, but to be honest, pip and virtualenv are fundamental today (I mean, before containers, but still)
    - toyg 8y ago
      I think what happened was that 3.2 was the first "real" release, when a lot of .0 bugs were shaken out; the 'u' prefix was added back around that time too, iirc. This resulted in more developers actually porting their libraries, an effort that had been basically non-existent before. The result was that the 3.4 ecosystem actually got mature enough that Real Work could get done, and more end-users started moving. IIRC 3.3 also had some big performance problems that were sorted in 3.4, but I cannot remember the details. Personally, I did very little with 3.2 but actually built Real Stuff only with 3.4.
yegle 8y ago
To be fair: PEP 3333 that addresses WSGI support with Python 3 Unicode was initially drafted in September 2010. I would consider that to be the first day we can write a web server in Python 3.

Godel_unicode 8y ago

Reminder that Python 2 dies in 18 months. https://pythonclock.org https://pythonclock.org

yorwba 8y ago
It will likely remain around for longer, since depending on unsupported software doesn't immediately kill a project. When old Python 2 code hits a bug in the CPython interpreter and nobody is around to fix it, there'll finally be an incentive to migrate away (maybe to Python 3, maybe to another language), but not before that.
- Narkov 8y ago
  True, however lots of large corporates won't use an unsupported version if the vendor drops it.
  - candiodari 8y ago
    There, the vendor will be redhat or ... who will be supporting it longer. What enterprises mean by support isn't provided by the PSF anyway. They support their products 10 years, or indefinitely if you pay extra for it, so Redhat will support Python 2 until June 30, 2024. If I understand it correctly, that means CentOS will too.
- u801e 8y ago
  > It will likely remain around for longer, since depending on unsupported software doesn't immediately kill a project. It also depends on how long companies like Redhat support it (CentOS 6 comes with Python 2.6 and CentOS 7 comes with Python 2.7).
acqq 8y ago
I wouldn't be surprised that vendors will still have to do the fixes to Python 2 for many years past that date, and that a kind of "fork" will keep the 2.7 branch decently alive much longer. It's just the current "official" maintainers that announce the cutoff date for their work.

berti 8y ago

For anyone confused, currently you have a choice of either "Standard Environment", which is Python 2.7 only, or "Flexible Environment" which allows you to choose Python 2.7, Python 3.6, or your own Python runtime [1][2]. This is about supporting Python 3 in the "Standard Environment". It's unclear whether the intention is to replace Python 2 or allow a choice? [1] https://cloud.google.com/appengine/docs/python/ https://cloud.google.com/appengine/docs/python/ [2] https://cloud.google.com/appengine/docs/flexible/python/runtime https://cloud.google.com/appengine/docs/flexible/python/runt...

ognyankulev 8y ago
tl;dr Standard environment apps "scale nearly instantaneously to handle huge traffic spikes", "cost-effective for applications that have significant periods where they are not serving traffic", use reasonably limited standard API + proprietary API
- dudus 8y ago
  More importantly standard scales down to 0 while flexible needs a minimum of 1 instance running at all times. So when just prototyping/developing or creating a small app only you or a few users will use it's very cost effective to use standard.

cheriot 8y ago

I love the new movement in on GAE Standard. With GAE Flexible, GKE, Cloud SQL, etc there's actually a growth path for something that out grows it. When Standard first launched, that additional complexity meant a rewrite. Can the Java runtime implement the full Servlet 3.1 spec now? It's 5 years old (and most Scala http libs don't support blocking Servlets anymore).

privateSFacct 8y ago

I never get this about google. They had a great product here (I and I know others actively interested) but they just drop development on these things for the new shiny toy. We're not asking for a ton, but if you could assign even a few folks to maintain the non-cool stuff, I think a lot more folks would build on GCP. I'm on AWS currently. They keep their old / uncool stuff around in a nice way (and totally seem to drop ball on other stuff that matters less like their new Amazon Linux 2 effort - python 3 support in that is totally wonky etc).

yegle 8y ago
Feature development of GAE (especially the Standard Environment) never stopped. Some of the features require multi quarters of work (e.g. https://cloudplatform.googleblog.com/2018/05/Increase-performance-while-reducing-costs-with-the-new-App-Engine-scheduler.html https://cloudplatform.googleblog.com/2018/05/Increase-perfor...), some are more enterprise-focused, some are not user visible but we improve the reliability with a new stack behind the curtain. We are changing the engine while flying (pun not intended). For various reasons, supporting new runtimes was never an easy task, even for small version bumps. This has changed drastically, starting with Java8 runtime, to the recently announced NodeJS beta, and to the closed alpha of Python 3 (remember this is Python 3.7). If you followed recent announcements from Google you probably can connect the dots. I think we are pretty open on what we are working on to our customers, as long as an NDA is signed. If you are interested be sure to let our cloud support team know! Disclaimer: I'm on App Engine SRE team.
- mijamo 8y ago
  I mean things have been pretty bad for GAE standard in Python for a very long time. It looks so dead I was actually expecting Google to completely remove it in 2 years or so. I work on a project on GAE standard and the platform was chosen before I started. I would say this was the worst possible solution to chose and the biggest cost for us right now. The most annoying point is that it is actually not easy at all to switch to the Flexible environment, because libraries are completely different. Examples of troubles : - new google-cloud libraries run in the flexible environment but not in the standard one (ex: google-cloud-pubsub doesn't run in the standard env after 0.27, and the doc for 0.27 is dead: https://google-cloud-python.readthedocs.io/en/0.27.0/pubsub/index.html https://google-cloud-python.readthedocs.io/en/0.27.0/pubsub/...) - traditional standard env libraries do not run in the flexible env. The most typical example is NDB for the Google datastore. Which means that migrating from the standard to the flexible env requires you to rebuild EVERYTHING that touches Google services... Great, then I'd better just switch to AWS instead! What's moer, flexible env is nice but for some reason it is not possible to have basic scaling anymore, nor is it possible to scale down to 0 when needed, which makes modular apps much more costly (previously we would have several services and when unused they just scaled down to 0). On top of that we just watched services get deprecated one after the other (mail service, then the datastore backup etc.) and old troubles not getting fixed (like memory problems with queries).

olskool 8y ago

I tried Appengine when it first came out but Google never seemed to support it much so I gave up on it.

k__ 8y ago

What lead to this whole Python 2 and 3 divide? I mean, other languages have new versions too...

ptx 8y ago
Python 1 didn't know about Unicode, so characters were bytes and things were simple (for latin1 users). Python 2 bolted on Unicode support, so now you had two kinds of strings (old-style strings and Unicode strings) with implicit conversions between them (that would blow up spectacularly when you least expected it) and overall a huge mess. Python 3 removes the old-style strings (replacing them with the "bytes" type and explicit conversions), so that we once again have only one type of string, and once again things are simple. This clean-up unfortunately means that code relying the Python 2 string type confusion will break.
- k__ 8y ago
  Well, at least someone tried to clean things up. Other languages simply carry these bugs around forever.
dwheeler 8y ago
The transition from Python 2 into Python 3 was badly botched, and for many years. First of all, they confused language with implementation, so the Python 3 interpreter cannot handle the Python 2 language while the Python2 interpreter cannot handle the Python 3 language (I refer to the main C interpreter, not others like PyPy). The automated tool "2to3" wasn't reliable on real-world programs, either. For many years it was impractical to write programs or libraries that worked on both 2 and 3. To write any Python 3 code, ALL of your transitively-dependent libraries had to convert to Python 3, and effectively simultaneously. By itself, that was rediculous. In addition, Python 3 doesn't add any radically new capabilities over Python 2, so it was basically all pain and no gain. Python 3's new approach to strings is conceptually clean, but it struggles with the messiness of the real world and the Python 3 developers had to add a lot of hacks to make it sort-of work. For example, file names are not strings of characters on Unix/Linux, there is no standard encoding for stdin, and real-world data doesn't always follow any particular encoding and there's often no need to require it. Python 3 was effectively ignored by most programmers for at least 5 years after its release, because it created more problems than it solved. What changed is that Python 3 was modified, and libraries were added to both Python 2 and 3, so that it started to became practical to write code that could work on both 2 and 3. The result: instead of having to transition your entire program and all its dependencies simultaneously, people could work incrementally, making little improvements here and there so that programs would keep working in 2, and eventually work in 3 as well. Various hacks were also created so that Python 3 could handle stuff like filenames (!). It would have been much better had the Python 3 developers made it easier to transition from 2 to 3 from the start, but it's finally easier. It's still somewhat painful; mercurial still hasn't transitioned, and it's not clear that they ever will. I'm unhappy about what the Python developers did earlier, but I think they've made good strides in trying to finally make it much easier to transition from Python 2 to Python 3. The main takeaway here should be, "make it as PAINLESS as POSSIBLE for your existing users to transition to a new version". It's okay to change APIs, but use different names or use versioned APIs. It's okay to change other things, but make it easy for people to modify parts of their program at a time. You'd think that'd be obvious, but it's clear that really smart people don't always think about that.

savrajsingh 8y ago

Appengine standard is amazing. I hope they continue to support 2.7 as well.

oooooof 8y ago

Google also committed to releasing an update to its App Engine Gw Basic and Turbo Pascal environments, “some time in the 2021 timeframe”.

mikece 8y ago

Just in case you're like me and had to Bing what the difference is between Docker and Google App Engine, it's summed up really well in this Stack Overflow answer: https://stackoverflow.com/a/49950687/3799 https://stackoverflow.com/a/49950687/3799 I wonder, though, if future development should target Docker/K8S and GAE should only be the domain of legacy applications.

deklerk 8y ago
> the difference is between Docker and Google App Engine Neither the question nor the answer in your link compare Docker and App Engine. Docker is a containerization platform, whereas App Engine is a PaaS (platform as a service). Their functions are wholly different. Very, very loosely; Docker is about building and running an individual package of some software. Basically, bundling software and running bundles. PaaS are platforms for developing, running, and managing (often pre-bundled) software.
- mikece 8y ago
  Would it be more accurate to compare Google App Engine (PaaS) to Azure Web Apps or App Service (also PaaS)? Docker is in a weird spot of not really being PaaS but yet too light to be considered IaaS... I tend to think of it as sufficiently PaaS-like except it runs on everyone's cloud as well as my laptop.

IshKebab 8y ago

Yet another example to throw in the face of people who say that the Python 2/3 problem is solved.