3 ms·
Is Airflow good for an ETL pipeline? Right now a client uses Jenkins, but it is quite clunky and difficult to automate, though they've managed to. Cloud not a
by digisign 4y ago
Is Airflow good for an ETL pipeline? Right now a client uses Jenkins, but it is quite clunky and difficult to automate, though they've managed to. Cloud not an option.
- theptip 4y agoAirflow is generally brought in when you have a DAG of jobs with many edges, and where you might want to re-run a sub-graph, or have sub-graphs run on different cadences. In a simplistic ETL/ELT pipeline you can model things as "Extract everything, then Load everything, then Transform everything", in which case you'll add a bunch of unnecessary complexity with Airflow. If you're looking for a framework to make the plumbing of ELT itself easier, but don't need sub-graph dependency modeling, Meltano is a good option to consider.
- digisign 4y agoThanks. Really just looking for a new routine/on-demand scheduler to run jobs with a nice interface. There might be a dependency or two, but not a lot. Also the jobs themselves are thousands of lines of code and not going to be substantially changed.