4 ms·
I just restructured a job to be parallel and run on Lambda and I couldn't be happier. Glad to see this idea is gaining mindshare. I found that it was easy to te
by teej 10y ago
I just restructured a job to be parallel and run on Lambda and I couldn't be happier. Glad to see this idea is gaining mindshare. I found that it was easy to test, easy to debug, easy to maintain, not very expensive, and just kinda worked.
The only caveat for those considering taking this path - AWS has an account-wide limit of 100 concurrent lambdas running at once. There's no way to know how many are currently running unless you keep track of it yourself. You'll only find out when you try to kick off a job and it get's throttled. I haven't contacted support yet to find out how hard it is for the limit to get raised or what they'll raise it to.
- ronack 10y agoWe had it raised to several thousand concurrent requests with no questions asked. YMMV based on age of account, amount of spend, etc.
- ceejayoz 10y agoYeah, most of the limits are there just to prevent "whoops, we fired up 500 $2/hour servers and left them running all month, can you refund us?" sort of newbie situations. Very few are technological and AWS has been happy to bump them up for me in each occasion I've had a need.
- teej 10y agoSome are, some aren't. For example, my experience has been that it's harder to get the 100 S3 bucket-per-account limit raised.
- jedberg 10y agoThat used to be a hard technological limitation but they fixed it a few months back. You should be able to get more now (but you have to have a good justification for it).
- chrisseaton 10y ago100 seems an odd number to be a technological limitation, doesn't it? It's not a power of two or anything so it's not the size of an integer. How could technology have been limiting it?
- ma2rten 10y agoWhat would be a good justification? Is there anything you can do with 101 buckets that you can't do with folders one bucket?
- ceejayoz 10y agoRequester-pays billing (http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketBilling.html http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketBilling...) for more than a hundred clients, static sites served through CloudFront, etc.
- ceejayoz 10y agoIIRC that was one of the few the AWS docs specifically called out as unraisable.
- vidarh 10y agoI've had plenty of cases where it's taken time, and required escalation and justification to get even relatively modest instance limits raised, depending on the data centre and instance types. It's not technological - it's about ensuring they have a reasonable handle on capacity and what they happen to have plenty of. It's not a problem as long as you're not asking for astronomical limits, but people need to be aware of it and request increases in advance of starting to hit limits rather than assume it'll be near immediate to get them lifted.
- teej 10y agoVery useful, thanks!
- Jdam 10y agoI once wanted to raise the limit of g2 instances from 2 to 100 in us-east-1 and the reply from the customer agent was he first "has to check if that does not affect overall region stability" WTF Anyway, it was raised a couple of days later.
- wnevets 10y agoraising the limit is a pretty easy request to aws support.