6 ms·

This is a screenshot of my google search from 2 days ago: http://i.imgur.com/BNAcSsn.png http://i.imgur.com/BNAcSsn.png I've been using Lambda quite a bit, I

by ralusek 10y ago

This is a screenshot of my google search from 2 days ago: http://i.imgur.com/BNAcSsn.png http://i.imgur.com/BNAcSsn.png I've been using Lambda quite a bit, I think it's SO amazingly useful. Tasks that are highly parallelized and CPU intensive can literally be infinitely scaled out. I find it weird that their poster child use case is still always a reactive event like watching S3 and formatting images. There are so many use cases for directly invoking a lambda directly from your code. Imagine a case where you had to parse a million documents with a relatively expensive computation, let's say 250MS per document. Maybe you have a solid machine with a few cores that's running your server, but even then you can't have the server cpu locked for so long, so naturally you'd need some sort of worker server set up. With a good machine and multiple cores, maybe you get 8 running at once. With a lambda, you can forego the worker server altogether. Just invoke a million lambdas directly from your application server, completely parallelized. Theoretically, you've taken something that would take 70 hours and had it run in 250ms without having to set up any additional infrastructure.

teej 10y ago
I just restructured a job to be parallel and run on Lambda and I couldn't be happier. Glad to see this idea is gaining mindshare. I found that it was easy to test, easy to debug, easy to maintain, not very expensive, and just kinda worked. The only caveat for those considering taking this path - AWS has an account-wide limit of 100 concurrent lambdas running at once. There's no way to know how many are currently running unless you keep track of it yourself. You'll only find out when you try to kick off a job and it get's throttled. I haven't contacted support yet to find out how hard it is for the limit to get raised or what they'll raise it to.
- ronack 10y ago
  We had it raised to several thousand concurrent requests with no questions asked. YMMV based on age of account, amount of spend, etc.
  - ceejayoz 10y ago
    Yeah, most of the limits are there just to prevent "whoops, we fired up 500 $2/hour servers and left them running all month, can you refund us?" sort of newbie situations. Very few are technological and AWS has been happy to bump them up for me in each occasion I've had a need.
    - teej 10y ago
      Some are, some aren't. For example, my experience has been that it's harder to get the 100 S3 bucket-per-account limit raised.
      - jedberg 10y ago
        That used to be a hard technological limitation but they fixed it a few months back. You should be able to get more now (but you have to have a good justification for it).
        chrisseaton 10y ago
        100 seems an odd number to be a technological limitation, doesn't it? It's not a power of two or anything so it's not the size of an integer. How could technology have been limiting it?
        sharpy 10y ago
        S3 bucket names are globally scoped - ie. they are literally scarce resources.
        jedberg 10y ago
        Technology in the sense that it was a hard coded variable, but it was picked arbitrarily.
        ma2rten 10y ago
        What would be a good justification? Is there anything you can do with 101 buckets that you can't do with folders one bucket?
- wnevets 10y ago
  raising the limit is a pretty easy request to aws support.
hueving 10y ago
>Theoretically, you've taken something that would take 70 hours and had it run in 250ms without having to set up any additional infrastructure. And you've spent the cost of building out that 8 server infrastructure in one batch.
- etrain 10y ago
  One of the nice things about lambda is the billing is super granular - you get billed at 100ms intervals. Assuming 70 hours at $0.000000834/100ms [1] The whole job costs $2.10. [1] - https://aws.amazon.com/lambda/pricing/ https://aws.amazon.com/lambda/pricing/
  - teej 10y ago
    70 * 60 * 60 * 10 * $0.000000834 = $2.1
  - nostrademons 10y ago
    By comparison (since I was curious), 70 hours on an m3.medium spot instance will run around $0.70. On an on-demand, it's about $5.39. EMR will cost you about $7.00 on top of the EC2 costs. If you can peg the CPU and don't mind getting interrupted, spot instances are still a fair bit cheaper. But Lambda looks pretty attractive for any other use-case where the statelessness of Lambda doesn't bite you.
    - deleted 10y ago
      [deleted]
    - etrain 10y ago
      Keep in mind that with EC2 you're billed hourly so the fastest a 70 cpu-hour job could finish on m3.medium for $0.70 is 1 hour, and that's ignoring setup time, etc. Meanwhile, on Lambda, you can actually run 1600 60s jobs (or 27 CPU-hours) in 3 minutes. This is inclusive of setup time, job submission, stragglers, etc. [1] Of course, if you've got sustained load, it's cheaper to go with spot instances, but the "occasionally I need a buttload of compute," model is well-served by Lambda. [1] http://ericjonas.com/pywren.html http://ericjonas.com/pywren.html
brianwawok 10y ago
So google compute setup I did a while back with preemptible instances + a celery queue + some autoscale based on load... The guts to make all that work was 50 or so lines of config. I think my auto scale script was 20 lines or so of Python. I guess the biggest downside was spinning up the new server took about 2 minutes, so for big load spikes it took a bit for it to level out... but with GCE per minute billing, all the napkin math I did says it is a fair bit cheaper per cpu unit to do it this way. In conclusion, someone good with devops + a normal work queue system could do this years ago. I guess it's cool to lower the barrier to entry, but not it does not seem like a game changer. It totally IS cool to scale out backend work to a job farm. Just seems like not the only way to do it.
- vgt 10y ago
  That's essentially what Google Cloud Dataproc gives you (managed Hadoop/Spark): - Per-minute billing - 0-to-cluster in under 90 seconds (aim for 30 seconds) - Pre-emptiblem VMs - Custom VMs Now you start with a job, pay a 30 second penalty, and execute it on an entirely ephemeral cluster. The "get a cluster and fill it with jobs and round up to an hour" model is indeed outdated IMHO. (work at Google Cloud)
  - brianwawok 10y ago
    Well it depends on my workload. The setup above on GCE had a stable load of X thousand jobs per minute, then burst loads up to 100x for short times. So for me it made sense to have a 24/7 celery cluster for the base load and add and remove nodes for the variable node. There was never a point shutting down the cluster made sense.
qaq 10y ago
if it's one of you will not be able to get the limit if you do this even at a rate of 1 req/sec you will be paying close to 1.4 million USD per month just for compute.
edblarney 10y ago
" I find it weird that their poster child use case is still always a reactive event like watching S3 and formatting images." Because those are the things that need to be done in the real world, 99% of the time. Sadly, we don't all get to work on cool stuff :) But I think as more people get access to the power and find uses cases, it will get used more. The 'next' step for Lambda is serverless hosting for web-sites, something which Lambda is 'almost but not quite' ready for, the limiting factor being transactional latency, especially in some weird cases. Once they have that sorted out, I will never want to see a server again :)
PetahNZ 10y ago
Except for the fact Lambda is limited to 100 concurrent executions.
- shawn-butler 10y ago
  This isn't a fact. The default limit is 100 concurrent per region. Also a simple conversation with AWS increases the limit[0]. They obviously don't want people to write absurd runaway structures and 100 is an arbitrarily chosen value to guard against that. [0]: https://aws.amazon.com/lambda/faqs/ https://aws.amazon.com/lambda/faqs/

by ralusek 10y ago

teej 10y ago

I just restructured a job to be parallel and run on Lambda and I couldn't be happier. Glad to see this idea is gaining mindshare. I found that it was easy to test, easy to debug, easy to maintain, not very expensive, and just kinda worked. The only caveat for those considering taking this path - AWS has an account-wide limit of 100 concurrent lambdas running at once. There's no way to know how many are currently running unless you keep track of it yourself. You'll only find out when you try to kick off a job and it get's throttled. I haven't contacted support yet to find out how hard it is for the limit to get raised or what they'll raise it to.

ronack 10y ago
We had it raised to several thousand concurrent requests with no questions asked. YMMV based on age of account, amount of spend, etc.
- ceejayoz 10y ago
  Yeah, most of the limits are there just to prevent "whoops, we fired up 500 $2/hour servers and left them running all month, can you refund us?" sort of newbie situations. Very few are technological and AWS has been happy to bump them up for me in each occasion I've had a need.
  - teej 10y ago
    Some are, some aren't. For example, my experience has been that it's harder to get the 100 S3 bucket-per-account limit raised.
    - jedberg 10y ago
      That used to be a hard technological limitation but they fixed it a few months back. You should be able to get more now (but you have to have a good justification for it).
      - chrisseaton 10y ago
        100 seems an odd number to be a technological limitation, doesn't it? It's not a power of two or anything so it's not the size of an integer. How could technology have been limiting it?
        sharpy 10y ago
        S3 bucket names are globally scoped - ie. they are literally scarce resources.
        jedberg 10y ago
        Technology in the sense that it was a hard coded variable, but it was picked arbitrarily.
      - ma2rten 10y ago
        What would be a good justification? Is there anything you can do with 101 buckets that you can't do with folders one bucket?
wnevets 10y ago
raising the limit is a pretty easy request to aws support.

hueving 10y ago

>Theoretically, you've taken something that would take 70 hours and had it run in 250ms without having to set up any additional infrastructure. And you've spent the cost of building out that 8 server infrastructure in one batch.

etrain 10y ago
One of the nice things about lambda is the billing is super granular - you get billed at 100ms intervals. Assuming 70 hours at $0.000000834/100ms [1] The whole job costs $2.10. [1] - https://aws.amazon.com/lambda/pricing/ https://aws.amazon.com/lambda/pricing/
- teej 10y ago
  70 * 60 * 60 * 10 * $0.000000834 = $2.1
- nostrademons 10y ago
  By comparison (since I was curious), 70 hours on an m3.medium spot instance will run around $0.70. On an on-demand, it's about $5.39. EMR will cost you about $7.00 on top of the EC2 costs. If you can peg the CPU and don't mind getting interrupted, spot instances are still a fair bit cheaper. But Lambda looks pretty attractive for any other use-case where the statelessness of Lambda doesn't bite you.
  - deleted 10y ago
    [deleted]
  - etrain 10y ago
    Keep in mind that with EC2 you're billed hourly so the fastest a 70 cpu-hour job could finish on m3.medium for $0.70 is 1 hour, and that's ignoring setup time, etc. Meanwhile, on Lambda, you can actually run 1600 60s jobs (or 27 CPU-hours) in 3 minutes. This is inclusive of setup time, job submission, stragglers, etc. [1] Of course, if you've got sustained load, it's cheaper to go with spot instances, but the "occasionally I need a buttload of compute," model is well-served by Lambda. [1] http://ericjonas.com/pywren.html http://ericjonas.com/pywren.html

brianwawok 10y ago

So google compute setup I did a while back with preemptible instances + a celery queue + some autoscale based on load... The guts to make all that work was 50 or so lines of config. I think my auto scale script was 20 lines or so of Python. I guess the biggest downside was spinning up the new server took about 2 minutes, so for big load spikes it took a bit for it to level out... but with GCE per minute billing, all the napkin math I did says it is a fair bit cheaper per cpu unit to do it this way. In conclusion, someone good with devops + a normal work queue system could do this years ago. I guess it's cool to lower the barrier to entry, but not it does not seem like a game changer. It totally IS cool to scale out backend work to a job farm. Just seems like not the only way to do it.

vgt 10y ago
That's essentially what Google Cloud Dataproc gives you (managed Hadoop/Spark): - Per-minute billing - 0-to-cluster in under 90 seconds (aim for 30 seconds) - Pre-emptiblem VMs - Custom VMs Now you start with a job, pay a 30 second penalty, and execute it on an entirely ephemeral cluster. The "get a cluster and fill it with jobs and round up to an hour" model is indeed outdated IMHO. (work at Google Cloud)
- brianwawok 10y ago
  Well it depends on my workload. The setup above on GCE had a stable load of X thousand jobs per minute, then burst loads up to 100x for short times. So for me it made sense to have a 24/7 celery cluster for the base load and add and remove nodes for the variable node. There was never a point shutting down the cluster made sense.

qaq 10y ago

if it's one of you will not be able to get the limit if you do this even at a rate of 1 req/sec you will be paying close to 1.4 million USD per month just for compute.

edblarney 10y ago

" I find it weird that their poster child use case is still always a reactive event like watching S3 and formatting images." Because those are the things that need to be done in the real world, 99% of the time. Sadly, we don't all get to work on cool stuff :) But I think as more people get access to the power and find uses cases, it will get used more. The 'next' step for Lambda is serverless hosting for web-sites, something which Lambda is 'almost but not quite' ready for, the limiting factor being transactional latency, especially in some weird cases. Once they have that sorted out, I will never want to see a server again :)

PetahNZ 10y ago

Except for the fact Lambda is limited to 100 concurrent executions.

shawn-butler 10y ago
This isn't a fact. The default limit is 100 concurrent per region. Also a simple conversation with AWS increases the limit[0]. They obviously don't want people to write absurd runaway structures and 100 is an arbitrarily chosen value to guard against that. [0]: https://aws.amazon.com/lambda/faqs/ https://aws.amazon.com/lambda/faqs/