3 ms·

Thanks for sharing! I ended up building a similar featureset in Python but using invocation type Event instead of RequestResponse. This ended up being a pain in

by teej 9y ago

Thanks for sharing! I ended up building a similar featureset in Python but using invocation type Event instead of RequestResponse. This ended up being a pain in the ass because Event type has these awful baked in automatic retries that you cannot configure or disable. It ended up being worth it because we wanted to use up to the full 5 minute runtime limit. How is this working for you? Where do you see it going?

ak217 9y ago
FYI, I believe you can disable retries and replace them with a DLQ (which you can then ignore, if you don't care about failures). I would be interested, though, if anyone has figured out a way to raise the 30 second RequestResponse timeout. Both that and the 300s Lambda timeout are annoyingly low. Google is slightly better, they go to 600s!
- teej 9y ago
  At my company we created a queue for Lambdas. The pattern we use is to monitor the execution time left and queue up a follow-up once there's <30s left. Not every job we do can be composed this way but it helps a lot.
  - madamelic 9y ago
    >The pattern we use is to monitor the execution time left and queue up a follow-up once there's <30s left. Not every job we do can be composed this way but it helps a lot. Hmm. Are you saying you can queue additional jobs on the same worker to use up remaining execution time?
    - teej 9y ago
      Amazon doesnt really give you any direct access to the underlying "worker" so it doesnt work like that. It's really as simple as: - the active lambda decides to stop performing work once there's only 30 seconds left - it switches to perform whatever cleanup is necessary - it queues up a new lambda to be executed by pushing the lambda name and event payload to redis - I have a special lambda that's scheduled for once a minute that scans certain keys in redis, pops the data out, and invokes the lambdas To give an example, I have a job where I need to read ~50 files in S3 line by line and do some work over them. I spin up 50 lambdas, one for each file. Each lambda chugs along until it is about to run out of time. It then queues up a followup lambda with the byte # to start at in the event payload. The followup lambda then picks up where the last one left off. I have some cases where instead of queueing up the followup lambda I just invoke it directly. The value in having a queue step in between is that you can more cleanly handle throttling and you can avoid event payload limits by just putting the payload into Redis.
      - madamelic 9y ago
        That's... really awesome. I think that may have to be a follow-up feature for this project. Thanks!
madamelic 9y ago
>How is this working for you? Where do you see it going? It actually works quite well. I use it for processing large amounts of data quickly. I originally built this to process some data that would've taken ~14 days and shortened to a few minutes (so nice!). I enjoy using it when I can find a way to fit it into a project. I've had it internally for a month so not a whole lot of time to see it in a variety of projects. In terms of where I see it going, I'm not entirely sure yet. It is pretty capable for what I am using it for currently, it can use node.js libraries so it fits the current needs (which are pretty basic). I'll see what happens when more people try it out. :) Do you have any tips on what should be added, since you've been down this road before?
- teej 9y ago
  Most of the things I've added are for logging and orchestration. The most important unlock for us was the ability for us to enable chaining of Lambdas that worked seamlessly with our orchestration. So Orchestrator kicks off 25 lambdas (A) that are reading files off S3. Each Lambda A reads 100 lines of that file and invokes a Lambda B to parse it. Lambda B parses and filters each line in its batch and sends it to Lambda C to be persisted or Lambda D if it needs to be discarded. We end up with hundreds or thousands of Lambda invocations and the orchestrator is aware of all of them. If there's a failure in any lambda, I can stop the entire job and retry from scratch. The orchestrator knows exactly when the job is done so I can easily chain it together with some other non-Lambda job in my system. Been working fine for us for ~200k invocations and 8k CPU minutes per day baseline with the ability to scale up to 10x that just by running more stuff. Our nightly job processes about 50gb of data in 3 min.

by teej 9y ago

ak217 9y ago

FYI, I believe you can disable retries and replace them with a DLQ (which you can then ignore, if you don't care about failures). I would be interested, though, if anyone has figured out a way to raise the 30 second RequestResponse timeout. Both that and the 300s Lambda timeout are annoyingly low. Google is slightly better, they go to 600s!

teej 9y ago
At my company we created a queue for Lambdas. The pattern we use is to monitor the execution time left and queue up a follow-up once there's <30s left. Not every job we do can be composed this way but it helps a lot.
- madamelic 9y ago
  >The pattern we use is to monitor the execution time left and queue up a follow-up once there's <30s left. Not every job we do can be composed this way but it helps a lot. Hmm. Are you saying you can queue additional jobs on the same worker to use up remaining execution time?
  - teej 9y ago
    Amazon doesnt really give you any direct access to the underlying "worker" so it doesnt work like that. It's really as simple as: - the active lambda decides to stop performing work once there's only 30 seconds left - it switches to perform whatever cleanup is necessary - it queues up a new lambda to be executed by pushing the lambda name and event payload to redis - I have a special lambda that's scheduled for once a minute that scans certain keys in redis, pops the data out, and invokes the lambdas To give an example, I have a job where I need to read ~50 files in S3 line by line and do some work over them. I spin up 50 lambdas, one for each file. Each lambda chugs along until it is about to run out of time. It then queues up a followup lambda with the byte # to start at in the event payload. The followup lambda then picks up where the last one left off. I have some cases where instead of queueing up the followup lambda I just invoke it directly. The value in having a queue step in between is that you can more cleanly handle throttling and you can avoid event payload limits by just putting the payload into Redis.
    - madamelic 9y ago
      That's... really awesome. I think that may have to be a follow-up feature for this project. Thanks!

madamelic 9y ago

>How is this working for you? Where do you see it going? It actually works quite well. I use it for processing large amounts of data quickly. I originally built this to process some data that would've taken ~14 days and shortened to a few minutes (so nice!). I enjoy using it when I can find a way to fit it into a project. I've had it internally for a month so not a whole lot of time to see it in a variety of projects. In terms of where I see it going, I'm not entirely sure yet. It is pretty capable for what I am using it for currently, it can use node.js libraries so it fits the current needs (which are pretty basic). I'll see what happens when more people try it out. :) Do you have any tips on what should be added, since you've been down this road before?

teej 9y ago
Most of the things I've added are for logging and orchestration. The most important unlock for us was the ability for us to enable chaining of Lambdas that worked seamlessly with our orchestration. So Orchestrator kicks off 25 lambdas (A) that are reading files off S3. Each Lambda A reads 100 lines of that file and invokes a Lambda B to parse it. Lambda B parses and filters each line in its batch and sends it to Lambda C to be persisted or Lambda D if it needs to be discarded. We end up with hundreds or thousands of Lambda invocations and the orchestrator is aware of all of them. If there's a failure in any lambda, I can stop the entire job and retry from scratch. The orchestrator knows exactly when the job is done so I can easily chain it together with some other non-Lambda job in my system. Been working fine for us for ~200k invocations and 8k CPU minutes per day baseline with the ability to scale up to 10x that just by running more stuff. Our nightly job processes about 50gb of data in 3 min.