3 ms·
Can any of these ETL frameworks kick off an ETL script without a rewrite? Something that would handle scheduling, retries, emit metrics around those actions, bu
by cheriot 5y ago
Can any of these ETL frameworks kick off an ETL script without a rewrite? Something that would handle scheduling, retries, emit metrics around those actions, but let me use my own tools for the actual data manipulation.
- johanneskanybal 5y agoOne of the standard patterns is airflow and kubernetes executor and then the pod can do whatever. There's the occasional side effect and some setup but more or less it works as advertised.
- timwis 5y agoCelery?
- jasonpbecker 5y agoWe use Jenkins for essentially this right now, and it works fine (especially if you have someone who knows Jenkins already).
- slotrans 5y agoSecond this. Jenkins is far and away the best generalized tool for running scheduled jobs, including ETL jobs.