3 ms·
Wouldn’t it be better to break your tasks into smaller, discrete units instead of a 15 minute job, such as using a message queue (SQS)?
by jasonjei 7y ago
Wouldn’t it be better to break your tasks into smaller, discrete units instead of a 15 minute job, such as using a message queue (SQS)?
- koolba 7y agoIf your unit of work is a transaction that takes 15+ minutes that's not possible. A common example is nightly ETL jobs that update the world.
- notyourwork 7y agoThat’s not a good use case for lambda.
- koolba 7y agoWell duh. That's why the top level comment is saying, "That's where Fargate comes in...".
- notyourwork 7y ago> If your unit of work is a transaction that takes 15+ minutes that's not possible. A common example is nightly ETL jobs that update the world. So what are you suggesting here?
- chadash 7y agoYes, that's ideal. But: * Sometimes it's not efficient to break a task down or it simply can't be done. * Sometimes it just makes the code hard to maintain if breaking it down into different components is a difficult task An example I can think of is loading a very large CSV file into a database. Lambda doesn't have enough memory to read the whole file and even if it did, it might take more than 16 minutes to run. I can try to break it out into multiple tasks and it'll probably be faster, but reading that code is (generally) gonna be much harder.