3 ms·

We would definitely seriously consider switching to GCS more if your cloud functions were as powerful as AWS Lambda (trigger from an S3 event) and supported Pyt

We would definitely seriously consider switching to GCS more if your cloud functions were as powerful as AWS Lambda (trigger from an S3 event) and supported Python 3.6 with serious control over the environment.

vikiomega9 10y ago
On a curious note, how do you guys use lambda?
- scrollaway 10y ago
  It's a little outdated now, but this post details our pipeline: https://hearthsim.info/blog/how-we-process-replays/ https://hearthsim.info/blog/how-we-process-replays/
  - RulerOf 10y ago
    As someone who's literally just starting to look at Lambda, thanks for that quick read. I had a lot of "chicken and egg"-type questions about using it, and seeing that critical step of bootstrapping the whole thing via the API Gateway was really informative.
boulos 10y ago
Is there something about the GCS trigger that doesn't work for you? I hear you on Python 3, but I'm also curious about "serious control over the environment". Can you be more specific?
- scrollaway 10y ago
  Here are our main issues with Lambda, from highest-to-lowest priority: - It supports Python 2.7 only. We need Python 3.4+ support. - We can't increase CPU allocation without increasing RAM allocation, making them far more expensive than we need. - Using psycopg2 on it is a PITA due to their handling of system dependencies. - The system is entirely proprietary, making it impossible to run it locally for testing. - Cloudwatch sucks for finding errors in the functions and is atrociously expensive. - API gateway is an extremely crufty system, and used not to let you pass around binary data (this has changed) - We can't disable/change the retry-on-error policy. We have a pretty hard tie-in to S3 and Redshift, but when GCF can do better on a majority of these points, we'll begin moving to it. But yes, Python 3 at a minimum would be a requirement.
  - mypalmike 10y ago
    > The system is entirely proprietary, making it impossible to run it locally for testing. I assume that you are referring to emulating the triggering of lambdas behind API gateway...? I've found a project that sets up a node environment to do this. Very handy for js/lambda development. A google search suggests similar options may exist for python.

vikiomega9 10y ago

On a curious note, how do you guys use lambda?

scrollaway 10y ago
It's a little outdated now, but this post details our pipeline: https://hearthsim.info/blog/how-we-process-replays/ https://hearthsim.info/blog/how-we-process-replays/
- RulerOf 10y ago
  As someone who's literally just starting to look at Lambda, thanks for that quick read. I had a lot of "chicken and egg"-type questions about using it, and seeing that critical step of bootstrapping the whole thing via the API Gateway was really informative.

boulos 10y ago

Is there something about the GCS trigger that doesn't work for you? I hear you on Python 3, but I'm also curious about "serious control over the environment". Can you be more specific?

scrollaway 10y ago
Here are our main issues with Lambda, from highest-to-lowest priority: - It supports Python 2.7 only. We need Python 3.4+ support. - We can't increase CPU allocation without increasing RAM allocation, making them far more expensive than we need. - Using psycopg2 on it is a PITA due to their handling of system dependencies. - The system is entirely proprietary, making it impossible to run it locally for testing. - Cloudwatch sucks for finding errors in the functions and is atrociously expensive. - API gateway is an extremely crufty system, and used not to let you pass around binary data (this has changed) - We can't disable/change the retry-on-error policy. We have a pretty hard tie-in to S3 and Redshift, but when GCF can do better on a majority of these points, we'll begin moving to it. But yes, Python 3 at a minimum would be a requirement.
- mypalmike 10y ago
  > The system is entirely proprietary, making it impossible to run it locally for testing. I assume that you are referring to emulating the triggering of lambdas behind API gateway...? I've found a project that sets up a node environment to do this. Very handy for js/lambda development. A google search suggests similar options may exist for python.