4 ms·

Show HN: Build optimized Python AWS Lambda layers

45 points by tobilg 7y ago

zomglings 7y ago
This is really great - I use AWS Lambda extensively and have started using Layers now that I have backends that require enough data to function. Will try your scripts out for sure! Really like that it's just a few bash scripts. I use bash and Python scripts like this in my CI environments to manage my Lambda functions. The integration with AWS is quite useful, and may be a feature you want to consider if you plan to widen the scope of this project. Thank you!
- tobilg 7y ago
  Great, looking forward for your feedback. It’s currently just a weekend project, so it might still have some rough edges...
kichik 7y ago
That's pretty cool. Layers are so much better for Python dependencies than the other solutions. They make code changes much faster to deploy. I also used layers in a recent project of mine. But instead of using Docker, I opted to build the layer in a special builder Lambda function. It's then used as a custom resource which outputs the S3 path and finally a layer resource uses that to create the actual layer.
tedshroyer 7y ago
I believe you could add the -t /some/path option (--target) to your pip install and avoid some of your cleanup steps. Also, if there are binary build steps, you may want to consider using an ec2 instance for the build. (https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes...)
- tobilg 7y ago
  Thanks, l‘ll have a look. Building binaries should be possible with the used lambci/lambda Docker image afaik.
kylebarron 7y ago
> Optimized the zip size by removing .pyc files and unnecessary libraries Why do you remove the .pyc files instead of the .py files? Keeping the .pyc bytecode means that it can skip the bytecode compiling when used. Some bootstrapping code I use from [0] is helpful to remove the uncompiled .py files. [0]: https://github.com/RemotePixel/cogeo-layer/blob/f04d14ebf99dfcfa71ae5584a818956e91e8f0fa/scripts/create-lambda-layer.sh#L20-L23 https://github.com/RemotePixel/cogeo-layer/blob/f04d14ebf99d...
- fulafel 7y ago
  You wouldn't be able to inspect the code anymore or see the plain text stack traces. Also does this actually work with the new model of .pyc files in __pycache__ dir?
  - kylebarron 7y ago
    I mean I've used the code I linked as a lambda layer just fine. I never specifically looked in the zip archive to make sure .py files weren't included, but from that bash script I think they're removed.

45 points by tobilg 7y ago

This is really great - I use AWS Lambda extensively and have started using Layers now that I have backends that require enough data to function. Will try your scripts out for sure! Really like that it's just a few bash scripts. I use bash and Python scripts like this in my CI environments to manage my Lambda functions. The integration with AWS is quite useful, and may be a feature you want to consider if you plan to widen the scope of this project. Thank you!

tobilg 7y ago
Great, looking forward for your feedback. It’s currently just a weekend project, so it might still have some rough edges...

kichik 7y ago

That's pretty cool. Layers are so much better for Python dependencies than the other solutions. They make code changes much faster to deploy. I also used layers in a recent project of mine. But instead of using Docker, I opted to build the layer in a special builder Lambda function. It's then used as a custom resource which outputs the S3 path and finally a layer resource uses that to create the actual layer.

tedshroyer 7y ago

I believe you could add the -t /some/path option (--target) to your pip install and avoid some of your cleanup steps. Also, if there are binary build steps, you may want to consider using an ec2 instance for the build. (https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes...)

tobilg 7y ago
Thanks, l‘ll have a look. Building binaries should be possible with the used lambci/lambda Docker image afaik.

kylebarron 7y ago

> Optimized the zip size by removing .pyc files and unnecessary libraries Why do you remove the .pyc files instead of the .py files? Keeping the .pyc bytecode means that it can skip the bytecode compiling when used. Some bootstrapping code I use from [0] is helpful to remove the uncompiled .py files. [0]: https://github.com/RemotePixel/cogeo-layer/blob/f04d14ebf99dfcfa71ae5584a818956e91e8f0fa/scripts/create-lambda-layer.sh#L20-L23 https://github.com/RemotePixel/cogeo-layer/blob/f04d14ebf99d...

fulafel 7y ago
You wouldn't be able to inspect the code anymore or see the plain text stack traces. Also does this actually work with the new model of .pyc files in __pycache__ dir?
- kylebarron 7y ago
  I mean I've used the code I linked as a lambda layer just fine. I never specifically looked in the zip archive to make sure .py files weren't included, but from that bash script I think they're removed.