5 ms·
We have a process where we strip out the text of PDFs and shove in to elastic. The lambda starts by counting the pages and if it’s 250 or less it handles the jo
by social_quotient 4y ago
We have a process where we strip out the text of PDFs and shove in to elastic. The lambda starts by counting the pages and if it’s 250 or less it handles the job. If it’s larger than that we make the lambda kick the job to a temp ec2 instance which takes over the job. Our cutoff is around 250 pages but it’s highly dependent on text density.
It would be great if the lambda could handle running long. Id probably even be fine if the duration was punitive in that the longer you run over X time it becomes progressively more expensive. This would create a disincentive for using the service wrongly but would allow for oddball tasks.
- philliphaydon 4y agoUse a step function and run a parallel task?
- scarface74 4y agoSince Lambda is just one of at least three ways to “launch a Firecracker task to do something”, I choose the right one. For something like that, I would use CodeBuild. In essence all CodeBuild is a method to run a list of bash commands in a Linux Docker container. Standard disclaimer: I work at AWS in Professional Services.
- social_quotient 4y agoThanks for this! I’m not sure I would have looked at CodeBuild
- scarface74 4y agoI meant to mention. You can run and rest CodeBuild locally. Just download the Docker image and run the shell script. https://docs.aws.amazon.com/codebuild/latest/userguide/use-codebuild-agent.html https://docs.aws.amazon.com/codebuild/latest/userguide/use-c...
- moltar 4y agoWhy not count the pages and then create a separate task for batches of N pages? You could orchestrate the batch thru parallel map in step function and bring together the results on the final step.