4 ms·
More recent tools such as Dagster and Prefect have much more to offer. One simple example is communication between tasks. Airflow has a clunky system for that c
by edumucelli 4y ago
More recent tools such as Dagster and Prefect have much more to offer. One simple example is communication between tasks. Airflow has a clunky system for that called XCom. The actual author of XCom says you probably should not use it due to the level of hackery it has under the hood [1]:
On Dagster and Prefect you communicate between tasks as if you were writing pure Python. On Airflow on the other hand ...
[1] https://www.youtube.com/watch?v=TlawR_gi8-Y&t=740s https://www.youtube.com/watch?v=TlawR_gi8-Y&t=740s
- marcinzm 4y agoI'm not sure if linking to a talk from 2018 in 2022 for a project that is being actively worked on (a bunch of Python abstractions for xcom were added in 2.0) is fair.
- byteflip 4y agoYes, curious if the Taskflow API introduced as part of Airflow 2.0 reduces this pain. It appears much easier/saner than working with XCOMs directly - less coupling and removes the need for lots of boilerplate code.
- karog 4y agoThere's also Flyte, which is natively data aware and schedules tasks around data dependencies. The syntax is essentially pure python too.