3 ms·
I always felt neither Airflow nor Superset solved any of the foundational problems with data analytics today. If we take Airflow, it is relatively easy to sched
by kfk 6y ago
I always felt neither Airflow nor Superset solved any of the foundational problems with data analytics today. If we take Airflow, it is relatively easy to schedule runs of scripts using cron (or more fancy Nomad jobs with a period stanza). What else does Airflow give me that cron doesn't? Is the parallelization stuff working? Dask is built from the ground up with parallelization in mind, sure it seems to solve a more foundational problem than Airflow. Is triggering and listening to events working? Doesn't look like. Is collaboration working? Doesn't seem to be the case since after writing your python script you need to basically rewrite it into an Airflow dag.
- naveedn 6y agoAirflow is 100% better at chaining jobs together than cron. The Airflow scheduler makes it so you don’t have to worry about putting “sleep” calls in your bash scripts to wait for some conditions to be met, and allows for non-linear orchestration of jobs. You might have different flows that need to run on different schedules, and with airflow, you can wait until one flow is done before the other starts.
- bosie 6y agodependency management. if task #3 fails, any task depending on it shouldn't run. not easy to do with cron based triggers
- kfk 6y agoYes but that is a feature of parallel and Dask does it well.
- bosie 6y agoI don't know too much about dask, how would you build a node in a Dask DAG to execute a java app and analyse its results (i.e. say database entries) to evalutate the success of that step?
- pgwhalen 6y agoAirflow is generally not aware of data flow, it just runs something once its upstream dependencies have finished. Less useful in a way, but also much more broadly applicable.