3 ms·
(Newbie Airflow user here). I believe one easy way to do it is by using Airflow's 'sensors'. Sensors are operators which poke continuously with an action until
by tpaschalis 8y ago
(Newbie Airflow user here). I believe one easy way to do it is by using Airflow's 'sensors'.
Sensors are operators which poke continuously with an action until it returns True (eg. until a file exists, an API gives a specific response, a process/query has finished).
Another way to do it would be to 'XComs', small pieces of information flying between DAGs, or 'Triggers', but these require some more setup, and IMO depend more on the way you're setting up your tasks.
- xkcd-sucks 8y agoYeah the issue is once a sensor fires once, it doesn't reset and keep firing on new data My homegrown solution is a sensor at the beginning, and at the end an airflow api call to trigger a dag run of the same dag. Not DagRunOperator because then no dag would never finish due to infinite recursion It seems kinda sketchy so I'm considering a lower level Celery implementation or even GenStage