Interesting, I have found AWS Lambda quite poor for data pipelines. Batch jobs can't exceed 15 minutes. Memory limit is 3GB. Payload sizes are 256kb max. If y

Interesting, I have found AWS Lambda quite poor for data pipelines. Batch jobs can't exceed 15 minutes. Memory limit is 3GB. Payload sizes are 256kb max. If you are used to batch processing large amounts of data, Lambda seems to be the exact opposite of this. I think it is good for "small data, highly concurrent event processing" but this is a very different use case from batch processing data pipelines.