4 ms·
I tried to spread large dataset into thousands of files on S3 and use StepFunctions Distributed Map to launch thousands of Lambda instances to process those fil
by DiscreteTom 2y ago
I tried to spread large dataset into thousands of files on S3 and use StepFunctions Distributed Map to launch thousands of Lambda instances to process those files in parallel, using DuckDB (or other libs) in Lambda. The parallel loading and processing is way faster than doing this in a single big EC2 instance.
- ramraj07 2y agoLambda isn’t infinitely parallel. I thought it doesn’t do more than 100 parallel runners? I4i.metal has 96 cores and can be faster than that.
- DiscreteTom 2y agoAs per AWS said in https://aws.amazon.com/cn/blogs/aws/aws-lambda-functions-now-scale-12-times-faster-when-handling-high-volume-requests/ https://aws.amazon.com/cn/blogs/aws/aws-lambda-functions-now... > Each synchronously invoked Lambda function now scales by 1,000 concurrent executions every 10 seconds.