3 ms·

Elastic PyPI: your own Python Package Index on AWS Lambda

70 points by di 10y ago

CaliforniaKarl 10y ago
This is cool, but it makes me sad that it has to run on Python 2, as Lambda does not natively support Python 3. (And no, I'm not counting tricks like calling into the underlying OS' Python 3)
_wmd 10y ago
This is a really cute use for Lambda, and a great working example of the kinds of limitations one could expect to bump into. I was recently asked to evaluate Lambda for use in hosting a bunch of existing web sites, and just from README.md, I'm glad I backed the hell away from that pandora's box before opening it.
- RhodesianHunter 10y ago
  Yeah, that's not really what it's intended for so that makes sense. Lambdas are great for anything you can wire them up to in AWS, but using them with API Gateway is a pain. For things like processing files or images as they appear in an S3 bucket, feeding or eating from a queue, replacing CRON jobs or small ETLs, and about a dozen other uses I have found them to be amazing.
- Ysx 10y ago
  Binary uploads are a new APIG feature[1] awaiting Serverless support[2]. Surprised WWW-Authenticate headers aren't passed through. May be possible with custom authorizers[3], though it's extra complexity. --- [1] https://aws.amazon.com/blogs/compute/binary-support-for-api-integrations-with-amazon-api-gateway/ https://aws.amazon.com/blogs/compute/binary-support-for-api-... [2] https://github.com/serverless/serverless/issues/2797 https://github.com/serverless/serverless/issues/2797 [3] http://docs.aws.amazon.com/apigateway/latest/developerguide/use-custom-authorizer.html http://docs.aws.amazon.com/apigateway/latest/developerguide/...
saamm 10y ago
Hosting a PyPI index on S3 is simpler, but I admire the effort here!
jamiesonbecker 10y ago
This is a perfect use for Lambda (i.e., low-traffic, once-off tasks that don't require a full server), or at least this would be a good fit if not for immaturity issues like missing WWW-Authenticate. But Lambda at scale is very expensive. A 100 request/second would cost more than $1,500/month, when even a single nano server could easily service that kind of load all by itself. Back of napkin analysis: https://twitter.com/JamiesonBecker/status/802185522139582464 https://twitter.com/JamiesonBecker/status/802185522139582464 Lambda is great for tech demos, small or rare loads, triggered tasks, etc. Not so great as part of your core app pipeline.
- takeda 10y ago
  Is it? I thought S3 would be a better fit.

70 points by di 10y ago

This is cool, but it makes me sad that it has to run on Python 2, as Lambda does not natively support Python 3. (And no, I'm not counting tricks like calling into the underlying OS' Python 3)

_wmd 10y ago

This is a really cute use for Lambda, and a great working example of the kinds of limitations one could expect to bump into. I was recently asked to evaluate Lambda for use in hosting a bunch of existing web sites, and just from README.md, I'm glad I backed the hell away from that pandora's box before opening it.

RhodesianHunter 10y ago
Yeah, that's not really what it's intended for so that makes sense. Lambdas are great for anything you can wire them up to in AWS, but using them with API Gateway is a pain. For things like processing files or images as they appear in an S3 bucket, feeding or eating from a queue, replacing CRON jobs or small ETLs, and about a dozen other uses I have found them to be amazing.
Ysx 10y ago
Binary uploads are a new APIG feature[1] awaiting Serverless support[2]. Surprised WWW-Authenticate headers aren't passed through. May be possible with custom authorizers[3], though it's extra complexity. --- [1] https://aws.amazon.com/blogs/compute/binary-support-for-api-integrations-with-amazon-api-gateway/ https://aws.amazon.com/blogs/compute/binary-support-for-api-... [2] https://github.com/serverless/serverless/issues/2797 https://github.com/serverless/serverless/issues/2797 [3] http://docs.aws.amazon.com/apigateway/latest/developerguide/use-custom-authorizer.html http://docs.aws.amazon.com/apigateway/latest/developerguide/...

saamm 10y ago

Hosting a PyPI index on S3 is simpler, but I admire the effort here!

jamiesonbecker 10y ago

This is a perfect use for Lambda (i.e., low-traffic, once-off tasks that don't require a full server), or at least this would be a good fit if not for immaturity issues like missing WWW-Authenticate. But Lambda at scale is very expensive. A 100 request/second would cost more than $1,500/month, when even a single nano server could easily service that kind of load all by itself. Back of napkin analysis: https://twitter.com/JamiesonBecker/status/802185522139582464 https://twitter.com/JamiesonBecker/status/802185522139582464 Lambda is great for tech demos, small or rare loads, triggered tasks, etc. Not so great as part of your core app pipeline.

takeda 10y ago
Is it? I thought S3 would be a better fit.