4 ms·
Would you recommend Airflow? I've been researching pipeline management libraries compatible with Python to replace our fragile cron/bash setup. From what I've r
by cpburns2009 9y ago
Would you recommend Airflow? I've been researching pipeline management libraries compatible with Python to replace our fragile cron/bash setup. From what I've read Airflow appears to be the best contender in the Python space.
- mattbillenstein 9y agoI do, I tried Luigi before Airflow and I rather liked the simplicity of Luigi's checkpointing of jobs (files vs a db with airflow), but Airflow has a nice UI, charts and graphs that are handy and nice to look at. Both describe DAGs using python, Airflow has more options and executors as I recall, but we only really run bash to run python jobs. The trouble with Airflow is it uses a database for state, and you can get in situations where a task is failing, so the dagrun fails; so you clear the state after fixing the task using the web ui and it'll usually run and everything is fine. But sometimes you get into a state where it says the dag is deadlocked and doing this doesn't work. But, steady-state, it works pretty well; I think among the things I've tried (cron + python, luigi, airflow), it's the best I've come across.
- jackgolding 9y agoI used airflow a few jobs ago where I was a 1 man data engineer (single server doing ETL) - honestly wish I just used CRON/bash.
- mattbillenstein 9y agoWhen you have a few things to run, yeah -- and we started this way. When you have a bunch of multi-step things with various dependencies and you want to visually see status -- airflow is worth the headache.