4 ms·

Genuine question... Does it support the size of a typical (for data, ml, image proc, etc) Python dependency bundle?

by ramoz 8y ago

icebraining 8y ago
The deployment package size limit seems to be 250MB: https://docs.aws.amazon.com/lambda/latest/dg/limits.html https://docs.aws.amazon.com/lambda/latest/dg/limits.html
- pplonski86 8y ago
  That's the limitation for data related worker tasks. The iron.io can be alternative (they offer 512 MB https://try.iron.io/pricing-worker-monthly/ https://try.iron.io/pricing-worker-monthly/)
- Glowbox 8y ago
  There are workarounds for that, f.e. on startup loading code from an s3 bucket (https://hackernoon.com/exploring-the-aws-lambda-deployment-limits-9a8384b0bec3 https://hackernoon.com/exploring-the-aws-lambda-deployment-l...)
  - icebraining 8y ago
    That workaround is for the 50MB limit on zip files, not for the 250MB limit :)
    - zwily 8y ago
      No, you get another 500 MB of tmp space you can use at runtime. This workaround does work.
- visarga 8y ago
  That would be OK for some ML applications if the framework itself would not be included in the deployment package.
avip 8y ago
I have a deployed lambda with your usual stack ( numpy sklearn scipy)
- lsorber 8y ago
  Did you manage to do that without pulling them from S3 (i.e., get them to fit in the 250MB limit)?
  - avip 8y ago
    I'm using serverless-python-requirements (see the specific example https://github.com/UnitedIncome/serverless-python-requirements/tree/master/example_native_deps https://github.com/UnitedIncome/serverless-python-requiremen...) - it uploads the libs zipped and unzips them on first invocation in the handler.
guitarbill 8y ago
For a lot of data processing and indigestion in a commercial setting, the standard library is often enough. Had a Lambda that parsed a file attached to an incoming email, and it worked very well. Even with complex packages like lxml, it usually isn't an issue. Seems like it might be tricky with some scientific packages or ML stuff though, but Lambdas aren't great for those workloads anyway, although the 15 minute runtime increase has helped.

Genuine question... Does it support the size of a typical (for data, ml, image proc, etc) Python dependency bundle?

by ramoz 8y ago

Genuine question... Does it support the size of a typical (for data, ml, image proc, etc) Python dependency bundle?

The deployment package size limit seems to be 250MB: https://docs.aws.amazon.com/lambda/latest/dg/limits.html https://docs.aws.amazon.com/lambda/latest/dg/limits.html

pplonski86 8y ago
That's the limitation for data related worker tasks. The iron.io can be alternative (they offer 512 MB https://try.iron.io/pricing-worker-monthly/ https://try.iron.io/pricing-worker-monthly/)
Glowbox 8y ago
There are workarounds for that, f.e. on startup loading code from an s3 bucket (https://hackernoon.com/exploring-the-aws-lambda-deployment-limits-9a8384b0bec3 https://hackernoon.com/exploring-the-aws-lambda-deployment-l...)
- icebraining 8y ago
  That workaround is for the 50MB limit on zip files, not for the 250MB limit :)
  - zwily 8y ago
    No, you get another 500 MB of tmp space you can use at runtime. This workaround does work.
visarga 8y ago
That would be OK for some ML applications if the framework itself would not be included in the deployment package.

avip 8y ago

I have a deployed lambda with your usual stack ( numpy sklearn scipy)

lsorber 8y ago
Did you manage to do that without pulling them from S3 (i.e., get them to fit in the 250MB limit)?
- avip 8y ago
  I'm using serverless-python-requirements (see the specific example https://github.com/UnitedIncome/serverless-python-requirements/tree/master/example_native_deps https://github.com/UnitedIncome/serverless-python-requiremen...) - it uploads the libs zipped and unzips them on first invocation in the handler.

guitarbill 8y ago

For a lot of data processing and indigestion in a commercial setting, the standard library is often enough. Had a Lambda that parsed a file attached to an incoming email, and it worked very well. Even with complex packages like lxml, it usually isn't an issue. Seems like it might be tricky with some scientific packages or ML stuff though, but Lambdas aren't great for those workloads anyway, although the 15 minute runtime increase has helped.