3 ms·

We have recently set up a TensorFlow assessment function in AWS lambda, and got very close to the maximum allowed size of a lambda function (250MB) with the tra

We have recently set up a TensorFlow assessment function in AWS lambda, and got very close to the maximum allowed size of a lambda function (250MB) with the trained model currently being 85MB, and the TensorFlow libraries and binaries taking up another 140 or so megabytes by default (<- which we cracked down a bit, but that's quite a hack). I feel like Amazon could do some work in this area to support users to use their own engines and not be bound to AWS AI Platforms and Services. This could be as simple as publicly documenting the time lambda's stay 'warm' for and retain data in /tmp persisting through multiple invocations or some other examples on how an AI workflow could be implemented with popular custom engines such as TensorFlow. Does anybody else have any experience in this regard?

sandusk 10y ago
I'm making over $7k a month working part time. I kept hearing other people tell me how much money they can make online so I decided to look into it. Well, it was all true and has totally changed my life. This is what I do... http://bit.ly/2atnA1a http://bit.ly/2atnA1a
djhworld 10y ago
We get around this by using a dependency injection framework (e.g. Guice) and initialising the module statically, so it will only get called once when the JVM loads (i.e. when the Lambda 'warms up') and then inject the dependencies into anything that uses them For example in one Lambda function, we use Guice to get resources from S3.
deanCommie 10y ago
It sounds like you're stretching to make the use case fit inside a Lambda. Have you considered extracting the assessment functionality into an SWF activity, invoked from your Lambda via StepFunctions?
- niklasrde 10y ago
  No, we haven't looked at SWF at all - might do some reading up on that. We considered running it on EC2, but the economics just didn't work out for our needs (hundreds of parallel processed jobs with irregular spikes, <5s runtimes per invocation and some others)
  - illumin8 10y ago
    You might also consider AWS batch. It's designed for this type of use case.
ionwake 10y ago
I have wondered about use cases for tensorflow - can you give me an example of a simple one ?
sorenvrist 10y ago
Knowing nothing about assement and tensor flow: Could the trained model be downloaded via s3 at init instead of being included inside the function zip? I believe the memory limits are more lax than the actual function limits. And the /tmp limit is 512MB.
tekoyaki 10y ago
Just curious, is this way better than using Cloud ML? https://cloud.google.com/ml/ https://cloud.google.com/ml/
- niklasrde 10y ago
  We're (sadly) a bit locked into using AWS. We've been eying the Google offerings for a while though (not just for this), so who knows what we may do in the future.
ranman 10y ago
The keep warm time isn't static AFAIK but some frameworks like Zappa have chosen 5 minutes (I work for AWS). If you raise an issue with support there may be other ways around this. There are also other reasons for not relying on functions being warm (spike in concurrent invocations, AZ outages, latency, etc.). If you open a support ticket and email me the # I'll see what I can find out: randhunt at amazon dot com I'm curious what sort of things you did to shrink your model? Have you considered pulling the model data from S3 outside of the main lambda handler and seeing if that negatively impacts performance -- with a cloud watch event running every 5 minutes or so to keep the function warm?
- niklasrde 10y ago
  Hey, thanks for your reply. Hadn't thought about a CloudWatch even to keep the function warm, I might suggest that to the team. We haven't shrunk the model, we've deleted superfluous files from the TensorFlow python library and dependencies (we don't need tensorboard, for example). It would be nice if you could package TensorFlow up into a minimal component just for assessment and not have any of the 'learning' stuff or other added-on libraries, but we couldn't find a simple way of doing that - we're not pro C++ engineers and even our python is not the greatest. We're managing for now, but if our model grows any bigger we will run into issues, but there have been some good suggestions in here. We had considered the S3 store, but we ruled that out quite quickly for cost & performance at the number of invocations a months we're looking at - but that was before we knew more about the 'keeping warm', so that may be revisited, too.
doppenhe 10y ago
Have you tried using Algorithmia for this? We get around a lot of these limitations that Lambda has (you can also call us from lambda and we can host most of your DL models https://algorithmia.com/developers/algorithm-development/model-guides/ https://algorithmia.com/developers/algorithm-development/mod...). Disclaimer: work @ algorithmia
tcas 10y ago
I've hacked around lambda quite a bit (I think the compressed size of one function is a tad under the max allowed). My hacks I remember are: - Run strip all .so libraries -- many aren't stripped fully - In Python I manually deleted sub packages of numpy/scipy I didn't need - If you're loading large models at initialize, numpy load routines are _much_ faster than cPickle. Have it load at module initialization, not during each invocation. I should really write a blog post about my experience with it. At a certain point I decided I was doing something that lambda really wasn't designed for -- I'm looking at migrating off, but the current implementation makes capacity planning super easy. Provisioning 1000 machines with 1GB of RAM for 15 minutes every day to read off a queue isn't a trivial problem. (Also, if anyone from AWS is reading, being able to limit the max concurrency of a single function vs account level limits would be super useful).
- btown 10y ago
  Would love to see that blog post!
- niklasrde 10y ago
  That sounds quite similar to what we're doing! Would love to see a blog post to compare experiences.. (Speaking of blogposts.. that was somewhere on the todo list..)
- jedberg 10y ago
  Just to throw this out there as another possible optimization, if you find that you're putting a big fat library into every function, one possibility is to run the library as its own lambda function. You'll be slowed down a bit by the network but it might be made up for by not having to constantly initialize the same thing.
mallya16 10y ago
checkout this blog on running predictions on Lambda with MXNet. It has pre-packaged numpy, openCV and MXNet. https://aws.amazon.com/blogs/compute/seamlessly-scale-predictions-with-aws-lambda-and-mxnet/ https://aws.amazon.com/blogs/compute/seamlessly-scale-predic...

sandusk 10y ago

I'm making over $7k a month working part time. I kept hearing other people tell me how much money they can make online so I decided to look into it. Well, it was all true and has totally changed my life. This is what I do... http://bit.ly/2atnA1a http://bit.ly/2atnA1a

djhworld 10y ago

We get around this by using a dependency injection framework (e.g. Guice) and initialising the module statically, so it will only get called once when the JVM loads (i.e. when the Lambda 'warms up') and then inject the dependencies into anything that uses them For example in one Lambda function, we use Guice to get resources from S3.

deanCommie 10y ago

It sounds like you're stretching to make the use case fit inside a Lambda. Have you considered extracting the assessment functionality into an SWF activity, invoked from your Lambda via StepFunctions?

niklasrde 10y ago
No, we haven't looked at SWF at all - might do some reading up on that. We considered running it on EC2, but the economics just didn't work out for our needs (hundreds of parallel processed jobs with irregular spikes, <5s runtimes per invocation and some others)
- illumin8 10y ago
  You might also consider AWS batch. It's designed for this type of use case.

ionwake 10y ago

I have wondered about use cases for tensorflow - can you give me an example of a simple one ?

sorenvrist 10y ago

Knowing nothing about assement and tensor flow: Could the trained model be downloaded via s3 at init instead of being included inside the function zip? I believe the memory limits are more lax than the actual function limits. And the /tmp limit is 512MB.

tekoyaki 10y ago

Just curious, is this way better than using Cloud ML? https://cloud.google.com/ml/ https://cloud.google.com/ml/

niklasrde 10y ago
We're (sadly) a bit locked into using AWS. We've been eying the Google offerings for a while though (not just for this), so who knows what we may do in the future.

ranman 10y ago

The keep warm time isn't static AFAIK but some frameworks like Zappa have chosen 5 minutes (I work for AWS). If you raise an issue with support there may be other ways around this. There are also other reasons for not relying on functions being warm (spike in concurrent invocations, AZ outages, latency, etc.). If you open a support ticket and email me the # I'll see what I can find out: randhunt at amazon dot com I'm curious what sort of things you did to shrink your model? Have you considered pulling the model data from S3 outside of the main lambda handler and seeing if that negatively impacts performance -- with a cloud watch event running every 5 minutes or so to keep the function warm?

niklasrde 10y ago
Hey, thanks for your reply. Hadn't thought about a CloudWatch even to keep the function warm, I might suggest that to the team. We haven't shrunk the model, we've deleted superfluous files from the TensorFlow python library and dependencies (we don't need tensorboard, for example). It would be nice if you could package TensorFlow up into a minimal component just for assessment and not have any of the 'learning' stuff or other added-on libraries, but we couldn't find a simple way of doing that - we're not pro C++ engineers and even our python is not the greatest. We're managing for now, but if our model grows any bigger we will run into issues, but there have been some good suggestions in here. We had considered the S3 store, but we ruled that out quite quickly for cost & performance at the number of invocations a months we're looking at - but that was before we knew more about the 'keeping warm', so that may be revisited, too.

doppenhe 10y ago

Have you tried using Algorithmia for this? We get around a lot of these limitations that Lambda has (you can also call us from lambda and we can host most of your DL models https://algorithmia.com/developers/algorithm-development/model-guides/ https://algorithmia.com/developers/algorithm-development/mod...). Disclaimer: work @ algorithmia

tcas 10y ago

I've hacked around lambda quite a bit (I think the compressed size of one function is a tad under the max allowed). My hacks I remember are: - Run strip all .so libraries -- many aren't stripped fully - In Python I manually deleted sub packages of numpy/scipy I didn't need - If you're loading large models at initialize, numpy load routines are _much_ faster than cPickle. Have it load at module initialization, not during each invocation. I should really write a blog post about my experience with it. At a certain point I decided I was doing something that lambda really wasn't designed for -- I'm looking at migrating off, but the current implementation makes capacity planning super easy. Provisioning 1000 machines with 1GB of RAM for 15 minutes every day to read off a queue isn't a trivial problem. (Also, if anyone from AWS is reading, being able to limit the max concurrency of a single function vs account level limits would be super useful).

btown 10y ago
Would love to see that blog post!
niklasrde 10y ago
That sounds quite similar to what we're doing! Would love to see a blog post to compare experiences.. (Speaking of blogposts.. that was somewhere on the todo list..)
jedberg 10y ago
Just to throw this out there as another possible optimization, if you find that you're putting a big fat library into every function, one possibility is to run the library as its own lambda function. You'll be slowed down a bit by the network but it might be made up for by not having to constantly initialize the same thing.

mallya16 10y ago

checkout this blog on running predictions on Lambda with MXNet. It has pre-packaged numpy, openCV and MXNet. https://aws.amazon.com/blogs/compute/seamlessly-scale-predictions-with-aws-lambda-and-mxnet/ https://aws.amazon.com/blogs/compute/seamlessly-scale-predic...