6 ms·

Hi all, Product Manager for App Engine here. Happy to answer any questions about Python 3 on the App Engine standard environment.

peacetreefrog 8y ago
Can I run pandas in it?
- unixhero 8y ago
  May I follow up with: pandas and numpy
  - thesandlord 8y ago
    Yup, both should work fine. Let us know if you run into issues. (I work for GCP)
- justhelpingout 8y ago
  Article says "YES!" > This new runtime allows you to take advantage of Python's vibrant ecosystem of open-source libraries and frameworks. While the Python 2 runtime only allowed the use of specific versions of whitelisted libraries, Python 3 supports arbitrary third-party libraries, including those that rely on C code and native extensions. Just add Django 2.0, NumPy, scikit-learn or your library of choice to a requirements.txt file. App Engine will install these libraries in the cloud when you deploy your app.
- stewart27 8y ago
  Yes, this runtime supports arbitrary dependencies specified via a `requirements.txt` file. If you find a library that doesn't work, please let us know. In general, anything that works using an open source Python 3.7 distribution should be supported.
  - peacetreefrog 8y ago
    Awesome!
  - aleaxit 8y ago
    Incidentally, the best way to "let us know" about bugs in GCP (App Engine or whatever else) is via the Issue Trackers, as per https://cloud.google.com/support/docs/issue-trackers https://cloud.google.com/support/docs/issue-trackers (disclaimer: making this and similar processes work is my current job at Google).
sidmitra 8y ago
Would the python3 runtime support things like ndb, taskqueue, memcache modules? We have a python2.7 standard app in production. It would be a considerable re-write if we had to move away from ndb and taskqueue.
- zinclozenge 8y ago
  https://cloud.google.com/appengine/docs/standard/python3/python-differences https://cloud.google.com/appengine/docs/standard/python3/pyt... it doesn't. "The ndb ORM library is not available for Python 3. You can access Cloud Datastore through the Cloud Datastore API. You can use the Google Cloud client libraries to store and retrieve data from Cloud Datastore."
- stewart27 8y ago
  We're working on suitable replacements for these services. In some cases, e.g., Cloud Tasks, we're pretty close. Others will take a little longer. We'll share updates as we make more of these services available.
  - sidmitra 8y ago
    Great! Thanks for your teams work so far. Also I did find that someone wrote a ndb-like wrapper around the cloud datastore API. https://github.com/Bogdanp/anom-py https://github.com/Bogdanp/anom-py I haven't tried it out yet. Although i think any reasonable path to upgrading to 3.7 might need atleast some wrapper like that around the datastore(even if it's not 100% compatible), otherwise forcing people relying on that to re-write the entire data layer. At this point i might just start re-writing the app, to move to something with less lock-in like Mongodb. But i'd love to hear what other people's plans are for this.
bootloop 8y ago
The (datastore) client libraries take up the http request quota and the build-in mechanism of the Standard Environment is not. Google recommends going forward with client libraries everywhere in the documentation but what's the solution on the introduced request limitation which isn't mentioned anywhere? (Or is this finally fixed in the meanwhile?)
- stewart27 8y ago
  I'm not aware of this issue. Can you point me to a public bug/GitHub issue?
  - groestl 8y ago
    We hit that HTTP request limit as well, and (driven by a discussion with Google support) migrated our backend away from the REST-based Google Cloud Datastore API to the internal (ApiProxy-based) Appengine API. Reduced the latency as well. I can send you the internal support ticket nr if interested.
    - stewart27 8y ago
      Sure, feel free to send it to me directly. stewartr at google.com.
skybrian 8y ago
Are there any billing changes for apps previously running within the free tier in standard environment?
- stewart27 8y ago
  There are no changes to the free tier[1] or to apps running using the existing Python 2.7 runtime in the standard environment. [1] You can use the free tier with the new Python 3.7 runtime.
grouseway 8y ago
What is the access to your hosted postgres like? Do you still have to use that proxy thing? I'm wondering how much effort is require to move a sqlalchemy based app over to this.
- stewart27 8y ago
  You would use Cloud SQL to access PostgreSQL. Here's how you configure your App Engine standard environment project to use Cloud SQL: https://cloud.google.com/appengine/docs/standard/python3/using-cloud-sql-postgres https://cloud.google.com/appengine/docs/standard/python3/usi... The Cloud SQL for PostgreSQL docs contain code samples to get you started: https://cloud.google.com/sql/docs/postgres/ https://cloud.google.com/sql/docs/postgres/ This should all work with SQLAlchemy.
- optimusclimb 8y ago
  It is not possible to access Postgres based CloudSQL from Python on AppEngine standard. See here: https://cloud.google.com/sql/docs/postgres/connect-app-engine https://cloud.google.com/sql/docs/postgres/connect-app-engin... Note that only node and Java are listed under non-Flex. I got burned by this and had to start over with MySQL.
  - dragonwriter 8y ago
    > It is not possible to access Postgres based CloudSQL from Python on AppEngine standard. It is possible on the new (beta) Python 3. 7 runtime on AppEngine standard: https://cloud.google.com/appengine/docs/standard/python3/using-cloud-sql-postgres https://cloud.google.com/appengine/docs/standard/python3/usi...
    - optimusclimb 8y ago
      Then they should update their docs.
      - deleted 8y ago
        [deleted]
rhodysurf 8y ago
Is GRPC supported? My gut feelign is no, but I hope I am wrong Edit: Holy shit this is so bittersweet. I was relying heavily on the batteries included user management, logging, admin only handlers, and especially the super easy cron job definitions. IDK what to think about this. It kinda takes away all the things that made App Engine worth it for me as a programmer who doesnt want to do dev ops.
- thesandlord 8y ago
  AFAIK you can't run a gRPC server on App Engine, but you can access other gRPC services (like all the GCP APIs) as a client. I might be wrong about the server part. All those things you mentioned in the edit should be supported in the new runtime. Edit: I see what you mean. The deeply integrated services are removed in favor of more portable solutions, like Logging moving to Stackdriver Logging (though stdout is still logged automatically), Users moving to Firebase Auth, etc. Cron should still work fine though. (I work for GCP)
  - rhodysurf 8y ago
    Thanks for the reply. Yeah I'm bummed about that because id rather use GRPC client libraries for my apps than the endpoints discovery doc generated ones. Also according to this they are not supported directly anymore except logging might still work but the wording is unclear. https://cloud.google.com/appengine/docs/standard/python3/python-differences https://cloud.google.com/appengine/docs/standard/python3/pyt... .
    - funkjunky 8y ago
      That page mentions that it supports the google-cloud-python libraries, which are the primary GCP gRPC client libraries for Python. The Apiary service-discovery library you are referring to is called google-api. The new GAE version doesn't support the old GAE-only libraries, like memcache, datastore, etc. These older services only worked with GAE, and therefore weren't portable to other services, which they are in the process of replacing. If I recall, the old version didn't support the google-cloud-python libraries either, so it's good to see that they are moving in that direction. I could be wrong about that though. In short, it looks like you have access to the main GCP gRPC libraries. You don't have access to the GAE-only service libraries. Disclaimer: was former GCP customer support.
neves 8y ago
Which were the technological problems that you had to update App Engine to Python 3? I thought it were upgraded a lot of time ago.
iamgopal 8y ago
Does this means, DJango can be supported natively ?
- stewart27 8y ago
  Yes, Django and Django 2.0 (which only supports Python 3) should work.
jkaplowitz 8y ago
Nice launch! I have some Python code which I've been meaning to convert to App Engine standard to take advantage of the free tier, and I hadn't yet adapted it to the proprietary APIs. Now I don't have to. Thanks! One Q: Does Google Cloud IAP work with this as an option for user auth? Would one follow the App Engine flexible environment pattern in their docs? The Cloud IAP docs still mention the Users API as the recommended approach for the App Engine standard environment without an exception for the Python 3.7 runtime.
simonw 8y ago
Just FYI - this page still only mentions Python 2.7, presumably it needs updating: https://cloud.google.com/appengine/docs/standard/ https://cloud.google.com/appengine/docs/standard/

peacetreefrog 8y ago

Can I run pandas in it?

unixhero 8y ago
May I follow up with: pandas and numpy
- thesandlord 8y ago
  Yup, both should work fine. Let us know if you run into issues. (I work for GCP)
justhelpingout 8y ago
Article says "YES!" > This new runtime allows you to take advantage of Python's vibrant ecosystem of open-source libraries and frameworks. While the Python 2 runtime only allowed the use of specific versions of whitelisted libraries, Python 3 supports arbitrary third-party libraries, including those that rely on C code and native extensions. Just add Django 2.0, NumPy, scikit-learn or your library of choice to a requirements.txt file. App Engine will install these libraries in the cloud when you deploy your app.
stewart27 8y ago
Yes, this runtime supports arbitrary dependencies specified via a `requirements.txt` file. If you find a library that doesn't work, please let us know. In general, anything that works using an open source Python 3.7 distribution should be supported.
- peacetreefrog 8y ago
  Awesome!
- aleaxit 8y ago
  Incidentally, the best way to "let us know" about bugs in GCP (App Engine or whatever else) is via the Issue Trackers, as per https://cloud.google.com/support/docs/issue-trackers https://cloud.google.com/support/docs/issue-trackers (disclaimer: making this and similar processes work is my current job at Google).

sidmitra 8y ago

Would the python3 runtime support things like ndb, taskqueue, memcache modules? We have a python2.7 standard app in production. It would be a considerable re-write if we had to move away from ndb and taskqueue.

zinclozenge 8y ago
https://cloud.google.com/appengine/docs/standard/python3/python-differences https://cloud.google.com/appengine/docs/standard/python3/pyt... it doesn't. "The ndb ORM library is not available for Python 3. You can access Cloud Datastore through the Cloud Datastore API. You can use the Google Cloud client libraries to store and retrieve data from Cloud Datastore."
stewart27 8y ago
We're working on suitable replacements for these services. In some cases, e.g., Cloud Tasks, we're pretty close. Others will take a little longer. We'll share updates as we make more of these services available.
- sidmitra 8y ago
  Great! Thanks for your teams work so far. Also I did find that someone wrote a ndb-like wrapper around the cloud datastore API. https://github.com/Bogdanp/anom-py https://github.com/Bogdanp/anom-py I haven't tried it out yet. Although i think any reasonable path to upgrading to 3.7 might need atleast some wrapper like that around the datastore(even if it's not 100% compatible), otherwise forcing people relying on that to re-write the entire data layer. At this point i might just start re-writing the app, to move to something with less lock-in like Mongodb. But i'd love to hear what other people's plans are for this.

bootloop 8y ago

The (datastore) client libraries take up the http request quota and the build-in mechanism of the Standard Environment is not. Google recommends going forward with client libraries everywhere in the documentation but what's the solution on the introduced request limitation which isn't mentioned anywhere? (Or is this finally fixed in the meanwhile?)

stewart27 8y ago
I'm not aware of this issue. Can you point me to a public bug/GitHub issue?
- groestl 8y ago
  We hit that HTTP request limit as well, and (driven by a discussion with Google support) migrated our backend away from the REST-based Google Cloud Datastore API to the internal (ApiProxy-based) Appengine API. Reduced the latency as well. I can send you the internal support ticket nr if interested.
  - stewart27 8y ago
    Sure, feel free to send it to me directly. stewartr at google.com.

skybrian 8y ago

Are there any billing changes for apps previously running within the free tier in standard environment?

stewart27 8y ago
There are no changes to the free tier[1] or to apps running using the existing Python 2.7 runtime in the standard environment. [1] You can use the free tier with the new Python 3.7 runtime.

grouseway 8y ago

What is the access to your hosted postgres like? Do you still have to use that proxy thing? I'm wondering how much effort is require to move a sqlalchemy based app over to this.

stewart27 8y ago
You would use Cloud SQL to access PostgreSQL. Here's how you configure your App Engine standard environment project to use Cloud SQL: https://cloud.google.com/appengine/docs/standard/python3/using-cloud-sql-postgres https://cloud.google.com/appengine/docs/standard/python3/usi... The Cloud SQL for PostgreSQL docs contain code samples to get you started: https://cloud.google.com/sql/docs/postgres/ https://cloud.google.com/sql/docs/postgres/ This should all work with SQLAlchemy.
optimusclimb 8y ago
It is not possible to access Postgres based CloudSQL from Python on AppEngine standard. See here: https://cloud.google.com/sql/docs/postgres/connect-app-engine https://cloud.google.com/sql/docs/postgres/connect-app-engin... Note that only node and Java are listed under non-Flex. I got burned by this and had to start over with MySQL.
- dragonwriter 8y ago
  > It is not possible to access Postgres based CloudSQL from Python on AppEngine standard. It is possible on the new (beta) Python 3. 7 runtime on AppEngine standard: https://cloud.google.com/appengine/docs/standard/python3/using-cloud-sql-postgres https://cloud.google.com/appengine/docs/standard/python3/usi...
  - optimusclimb 8y ago
    Then they should update their docs.
    - deleted 8y ago
      [deleted]

rhodysurf 8y ago

Is GRPC supported? My gut feelign is no, but I hope I am wrong Edit: Holy shit this is so bittersweet. I was relying heavily on the batteries included user management, logging, admin only handlers, and especially the super easy cron job definitions. IDK what to think about this. It kinda takes away all the things that made App Engine worth it for me as a programmer who doesnt want to do dev ops.

thesandlord 8y ago
AFAIK you can't run a gRPC server on App Engine, but you can access other gRPC services (like all the GCP APIs) as a client. I might be wrong about the server part. All those things you mentioned in the edit should be supported in the new runtime. Edit: I see what you mean. The deeply integrated services are removed in favor of more portable solutions, like Logging moving to Stackdriver Logging (though stdout is still logged automatically), Users moving to Firebase Auth, etc. Cron should still work fine though. (I work for GCP)
- rhodysurf 8y ago
  Thanks for the reply. Yeah I'm bummed about that because id rather use GRPC client libraries for my apps than the endpoints discovery doc generated ones. Also according to this they are not supported directly anymore except logging might still work but the wording is unclear. https://cloud.google.com/appengine/docs/standard/python3/python-differences https://cloud.google.com/appengine/docs/standard/python3/pyt... .
  - funkjunky 8y ago
    That page mentions that it supports the google-cloud-python libraries, which are the primary GCP gRPC client libraries for Python. The Apiary service-discovery library you are referring to is called google-api. The new GAE version doesn't support the old GAE-only libraries, like memcache, datastore, etc. These older services only worked with GAE, and therefore weren't portable to other services, which they are in the process of replacing. If I recall, the old version didn't support the google-cloud-python libraries either, so it's good to see that they are moving in that direction. I could be wrong about that though. In short, it looks like you have access to the main GCP gRPC libraries. You don't have access to the GAE-only service libraries. Disclaimer: was former GCP customer support.

neves 8y ago

Which were the technological problems that you had to update App Engine to Python 3? I thought it were upgraded a lot of time ago.

iamgopal 8y ago

Does this means, DJango can be supported natively ?

stewart27 8y ago
Yes, Django and Django 2.0 (which only supports Python 3) should work.

jkaplowitz 8y ago

Nice launch! I have some Python code which I've been meaning to convert to App Engine standard to take advantage of the free tier, and I hadn't yet adapted it to the proprietary APIs. Now I don't have to. Thanks! One Q: Does Google Cloud IAP work with this as an option for user auth? Would one follow the App Engine flexible environment pattern in their docs? The Cloud IAP docs still mention the Users API as the recommended approach for the App Engine standard environment without an exception for the Python 3.7 runtime.

simonw 8y ago

Just FYI - this page still only mentions Python 2.7, presumably it needs updating: https://cloud.google.com/appengine/docs/standard/ https://cloud.google.com/appengine/docs/standard/