3 ms·
First impression: this is a brilliant piece of software design. The ability to compose a map/filter chain and execute it in parallel against every object in a
by simonw 10y ago
First impression: this is a brilliant piece of software design.
The ability to compose a map/filter chain and execute it in parallel against every object in an S3 bucket that matches a specific prefix - wow.
The set of problems that can be quickly and cheaply solved with this thing is enormous. My biggest problem with lambda functions is that they are a bit of a pain to actually write - for transforming data in S3 this looks like my ideal abstraction.
- wellsjohnston 10y agoThank you :) Writing this was a necessity for me, being a 1-person data team coming from a Node.js background.
- rpedela 10y agoYes, I concur! I am definitely trying this out. I have a couple use cases where I think lambda functions would be useful but I don't currently have the time to figure out how to write and execute them.
- koolba 10y ago... Except it's not! The "lambda" here isn't AWS Lambda. It's a locally executed function. Now if this scheduled a bunch of real Lambdas to execute the work for each bucket then yes that'd be awesome.
- glogla 10y agoThat would be glorious.
- kot-behemoth 10y agoIt should be fairly easily doable with Gordon (https://github.com/jorgebastida/gordon https://github.com/jorgebastida/gordon), and scheduling via CloudWatch Events. Or Airflow.
- simonw 10y agoBah. My first impression was totally wrong in that case. Here's hoping someone builds a version of this that executes magically in the lambda cloud.
- illumin8 10y agoWell, you could run it on a large EC2 instance (x1.32xlarge?!:O) and it would be running the lambdas on the cloud, technically... ;-)