reading/writing Arrow is zero serde overhead operation to/from memory to disk. I think of lambda as a thread, and you can put a trigger on S3 bucket on each in

reading/writing Arrow is zero serde overhead operation to/from memory to disk. I think of lambda as a thread, and you can put a trigger on S3 bucket on each incoming file - to get processed. This allows you to get around GIL, and lets you invoke your lambda for each mini-batch. assuming you have high volume and frequency of data - you will need to "cool down" your high frequency data, and switch from row-basis (like millions of rows per second) to mini-batch basis (like one batch file per 100Mb). This can be achieved by having kafka with high partition number on the ingestion side, and sink to s3. from S3 for each new file your lambda will be invoked and minibatch will be processed by your python code, and you can right size your lambda's RAM, but usually I reserve 2-3x size of a batch file for lambda. the killer feature is zero ops. Just by tuning your minibatch size you can regulate how many times your lambda will be invoked