5 ms·

> I've hacked a few lambdas together but never dug deep Then why comment? You clearly don't understand the use-case that AWS fits. I've had jobs that took 18

> I've hacked a few lambdas together but never dug deep Then why comment? You clearly don't understand the use-case that AWS fits. I've had jobs that took 18 hours to run on single machine finish in 12 minutes on Lambda. I could run that process 4 times a month and still stay within AWS's free tier limits. For the right workloads it is 100% worth realigning your code to fit the stack.

foxtr0t 7y ago
>Then why comment? Because the instruction list from above isn't backed with any solid reasoning and because commenting is what people do on HN. >You clearly don't understand the use-case that AWS fits. Pray tell, what is this enviable use case that I so clearly do not grasp?
ses1984 7y ago
>I've had jobs that took 18 hours to run on single machine finish in 12 minutes on Lambda. I could run that process 4 times a month and still stay within AWS's free tier limits. Ok I'll bite. What takes 18 hours to run on a single machine but finishes in 12 minutes on Lambda.
- mycall 7y ago
  An unoptimized query?
  - kovek 7y ago
    Why would the processing time differ? Would you have multiple lambdas running different subsets of the unoptimized query?
    - kahnjw 7y ago
      You could achieve the same with basically any decent concurrency model on a single machine.
    - mycall 7y ago
      Maybe one lambda is timing out to fallback, slowing sagas down. I don't know.
- akoumis 7y ago
  I worked on a service a year ago that would stream a video from a source and upload it to a video hosting service. A few concurrent transfers would saturate the NIC. Putting each transfer job in a separate lambda allowed running any number of them in parallel, much faster than queuing up jobs on standalone instances
  - ses1984 7y ago
    If you throw more resources at a bottlenecked problem, it will go faster.
    - akoumis 7y ago
      Right, but without the lambda infrastructure it would be infeasible from infrastructure and cost perspective to spin up, let’s say 10,000 instances, complete a 10 minute job on each of them, and then turn them off to save money, on a regular basis
      - curryst 7y ago
        Isn't that also possible with EC2? Just set the startup script to something that installs your software (or build an AMI with it). Dump videos to be processed into SQS, have your software pull videos from that. You'd need some logic to shut down the instances once it's done, but the simplest logic would be to have the software do a self-destruct on the EC2 VM if it's unable to pull a video to process for X time, where X is something sensible like 5 minutes.
  - drieddust 7y ago
    Yes but running multiple lambda jobs in parallel would still add upto more time than 12 minutes. What am I missing?
    - akoumis 7y ago
      If I was running 10,000 transfer jobs in parallel and the longest of them took 12 minutes, the job would take 12 minutes
- hallman76 7y ago
  We developed a web-based tool that described water quality based on your location. We generated screenshots of every outcome so the results could be shared to FB/titter. It was something on the order of 40k screenshots. Our process used headless chrome to generate a screenshot then it was uploaded to S3 for hosting. Doing that in a series took forever. It took something like 14 hours to generate the screenshots, then 4 hours to upload them all. Spreading that load across lambda functions allowed us to basically run the job in parallel. Each individual lambda process took longer to generate a screenshot than on our initial desktop process, but the overall process was dramatically faster.
  - foxtr0t 7y ago
    The parallelism argument doesn’t pass muster because you can do the same thing with a cluster of free tier t2.micro machines with any good orchestration platform, not just lambda. This argument is basically: no counterpoint to the original post, but you can do things that are also easy on any other comparable platform. Tell me again what I don’t understand?
    - weberc2 7y ago
      > Tell me again what I don’t understand? As someone who has done both, it's far, far easier to stand up a lambda than it is to manage a cluster of servers.
      - foxtr0t 7y ago
        This still doesn’t make sense. There are portable systems that do the same, and have fully managed options, such as kubernetes. In my mind the thing that makes lambda “easier” is they make a bunch of decisions for you, for better or worse. For real applications probably for the worse. If you have the knowledge to make those decisions for yourself you’re probably better off doing that.
        weberc2 7y ago
        > This still doesn’t make sense. There are portable systems that do the same, and have fully managed options, such as kubernetes. The whole value proposition behind AWS is that they can do it better than your business due (directly or indirectly) to economies of scale. I think Kubernetes is super cool, but rebuilding AWS on top of Kubernetes is not cost effective for most companies--they're better off using AWS-managed offerings. Of course, you can mix and match via EKS or similar, but there are lots of gotchas there as well (how do I integrate Kubernetes' permissions model with IAM? how do I get Kubernetes logs into CloudWatch? how do I use CloudWatch to monitor Kubernetes events? etc).

foxtr0t 7y ago

>Then why comment? Because the instruction list from above isn't backed with any solid reasoning and because commenting is what people do on HN. >You clearly don't understand the use-case that AWS fits. Pray tell, what is this enviable use case that I so clearly do not grasp?

ses1984 7y ago

>I've had jobs that took 18 hours to run on single machine finish in 12 minutes on Lambda. I could run that process 4 times a month and still stay within AWS's free tier limits. Ok I'll bite. What takes 18 hours to run on a single machine but finishes in 12 minutes on Lambda.

mycall 7y ago
An unoptimized query?
- kovek 7y ago
  Why would the processing time differ? Would you have multiple lambdas running different subsets of the unoptimized query?
  - kahnjw 7y ago
    You could achieve the same with basically any decent concurrency model on a single machine.
  - mycall 7y ago
    Maybe one lambda is timing out to fallback, slowing sagas down. I don't know.
akoumis 7y ago
I worked on a service a year ago that would stream a video from a source and upload it to a video hosting service. A few concurrent transfers would saturate the NIC. Putting each transfer job in a separate lambda allowed running any number of them in parallel, much faster than queuing up jobs on standalone instances
- ses1984 7y ago
  If you throw more resources at a bottlenecked problem, it will go faster.
  - akoumis 7y ago
    Right, but without the lambda infrastructure it would be infeasible from infrastructure and cost perspective to spin up, let’s say 10,000 instances, complete a 10 minute job on each of them, and then turn them off to save money, on a regular basis
    - curryst 7y ago
      Isn't that also possible with EC2? Just set the startup script to something that installs your software (or build an AMI with it). Dump videos to be processed into SQS, have your software pull videos from that. You'd need some logic to shut down the instances once it's done, but the simplest logic would be to have the software do a self-destruct on the EC2 VM if it's unable to pull a video to process for X time, where X is something sensible like 5 minutes.
- drieddust 7y ago
  Yes but running multiple lambda jobs in parallel would still add upto more time than 12 minutes. What am I missing?
  - akoumis 7y ago
    If I was running 10,000 transfer jobs in parallel and the longest of them took 12 minutes, the job would take 12 minutes
hallman76 7y ago
We developed a web-based tool that described water quality based on your location. We generated screenshots of every outcome so the results could be shared to FB/titter. It was something on the order of 40k screenshots. Our process used headless chrome to generate a screenshot then it was uploaded to S3 for hosting. Doing that in a series took forever. It took something like 14 hours to generate the screenshots, then 4 hours to upload them all. Spreading that load across lambda functions allowed us to basically run the job in parallel. Each individual lambda process took longer to generate a screenshot than on our initial desktop process, but the overall process was dramatically faster.
- foxtr0t 7y ago
  The parallelism argument doesn’t pass muster because you can do the same thing with a cluster of free tier t2.micro machines with any good orchestration platform, not just lambda. This argument is basically: no counterpoint to the original post, but you can do things that are also easy on any other comparable platform. Tell me again what I don’t understand?
  - weberc2 7y ago
    > Tell me again what I don’t understand? As someone who has done both, it's far, far easier to stand up a lambda than it is to manage a cluster of servers.
    - foxtr0t 7y ago
      This still doesn’t make sense. There are portable systems that do the same, and have fully managed options, such as kubernetes. In my mind the thing that makes lambda “easier” is they make a bunch of decisions for you, for better or worse. For real applications probably for the worse. If you have the knowledge to make those decisions for yourself you’re probably better off doing that.
      - weberc2 7y ago
        > This still doesn’t make sense. There are portable systems that do the same, and have fully managed options, such as kubernetes. The whole value proposition behind AWS is that they can do it better than your business due (directly or indirectly) to economies of scale. I think Kubernetes is super cool, but rebuilding AWS on top of Kubernetes is not cost effective for most companies--they're better off using AWS-managed offerings. Of course, you can mix and match via EKS or similar, but there are lots of gotchas there as well (how do I integrate Kubernetes' permissions model with IAM? how do I get Kubernetes logs into CloudWatch? how do I use CloudWatch to monitor Kubernetes events? etc).