3 ms·
Do you have any benchmarks about how much data a given lambda can search/process after loading Arrow data? Not trying to argue, I'm curious because I never thou
by Karrot_Kream 3y ago
Do you have any benchmarks about how much data a given lambda can search/process after loading Arrow data? Not trying to argue, I'm curious because I never thought of this architecture myself, because I would think that the time it takes to ingest the Arrow data and then search through it would be too long for a lambda but I may be totally off base here. I've not played around in detail with lambdas so I don't have particularly robust mental model on their limitations.
- slt2021 3y agoreading/writing Arrow is zero serde overhead operation to/from memory to disk. I think of lambda as a thread, and you can put a trigger on S3 bucket on each incoming file - to get processed. This allows you to get around GIL, and lets you invoke your lambda for each mini-batch. assuming you have high volume and frequency of data - you will need to "cool down" your high frequency data, and switch from row-basis (like millions of rows per second) to mini-batch basis (like one batch file per 100Mb). This can be achieved by having kafka with high partition number on the ingestion side, and sink to s3. from S3 for each new file your lambda will be invoked and minibatch will be processed by your python code, and you can right size your lambda's RAM, but usually I reserve 2-3x size of a batch file for lambda. the killer feature is zero ops. Just by tuning your minibatch size you can regulate how many times your lambda will be invoked
- Karrot_Kream 3y agoVery cool. Do you then further aggregate and load into a DB or vector store or something?