3 ms·
This seems like a space that could always use an improvement. What are the benefits of a tool like this over something more popular like Airflow?
by j_kao 8y ago
This seems like a space that could always use an improvement. What are the benefits of a tool like this over something more popular like Airflow?
- tracer4201 8y agoI lead an initiative in building a data engineering program and randomly picked Airflow to orchestrate our work loads before doing a deeper dive into discovering the right solution. We’re still running Airflow, and not because we had invested too much into it but because We found it to be the most easy to use and manageable solution.
- josmek 8y agoThere are a few reasons why this could be chosen over something like Airflow. They are (but not limited to): modern workflow semantics including both functional and imperative API and the ability to work with dynamically mapped tasks, a stricter / richer notion of state, data passage between tasks as a first class operation, better scheduling semantics (workflows aren’t required to have a schedule, scheduled flows can still be run off schedule, and it’s possible to have multiple runs of the same flow occurring at the exact same time), etc…