3 ms·

What strategies do people use to make Airflow behave like an "event-driven" scheduler versus a "time-driven" scheduler? Like, for example, processing data as it

What strategies do people use to make Airflow behave like an "event-driven" scheduler versus a "time-driven" scheduler? Like, for example, processing data as it is received versus processing data at set time intervals

tedmiston 8y ago
One way to achieve that is to use externally triggered DAGs [1][2]. Instead of trying to make the Airflow scheduler work as event-driven though, I've more tried to adapt my approach. I've used Airflow as a bridge between a micro batch streaming system (Sparking Streaming --> S3, Airflow: S3 --> ...) successfully, but there are some challenges around the edge cases like files landing later than expected. One approach you can take is to have the Airflow sensor hit a file that's a proxy for all of the files being present. For example, if your process could write hundreds of S3 files, once it's finished the last write for that hour (even if that happens late for whatever reason), then it could write a top-level OK file that the sensor hits. Can you elaborate on your specific use case? [1]: https://stackoverflow.com/a/50592573/149428 https://stackoverflow.com/a/50592573/149428 [2]: https://airflow.readthedocs.io/en/stable/scheduler.html#external-triggers https://airflow.readthedocs.io/en/stable/scheduler.html#exte...
tpaschalis 8y ago
(Newbie Airflow user here). I believe one easy way to do it is by using Airflow's 'sensors'. Sensors are operators which poke continuously with an action until it returns True (eg. until a file exists, an API gives a specific response, a process/query has finished). Another way to do it would be to 'XComs', small pieces of information flying between DAGs, or 'Triggers', but these require some more setup, and IMO depend more on the way you're setting up your tasks.
- xkcd-sucks 8y ago
  Yeah the issue is once a sensor fires once, it doesn't reset and keep firing on new data My homegrown solution is a sensor at the beginning, and at the end an airflow api call to trigger a dag run of the same dag. Not DagRunOperator because then no dag would never finish due to infinite recursion It seems kinda sketchy so I'm considering a lower level Celery implementation or even GenStage
yzmtf2008 8y ago
For "event-driven" scheduler, you may want to have a streaming system like Flink or Spark Streaming instead. You can schedule the startup of these pipelines with Airflow (e.g., using BashOperator), and handle the actual streaming with the streaming systems directly. We use Airflow with this pattern extensively.

tedmiston 8y ago

One way to achieve that is to use externally triggered DAGs [1][2]. Instead of trying to make the Airflow scheduler work as event-driven though, I've more tried to adapt my approach. I've used Airflow as a bridge between a micro batch streaming system (Sparking Streaming --> S3, Airflow: S3 --> ...) successfully, but there are some challenges around the edge cases like files landing later than expected. One approach you can take is to have the Airflow sensor hit a file that's a proxy for all of the files being present. For example, if your process could write hundreds of S3 files, once it's finished the last write for that hour (even if that happens late for whatever reason), then it could write a top-level OK file that the sensor hits. Can you elaborate on your specific use case? [1]: https://stackoverflow.com/a/50592573/149428 https://stackoverflow.com/a/50592573/149428 [2]: https://airflow.readthedocs.io/en/stable/scheduler.html#external-triggers https://airflow.readthedocs.io/en/stable/scheduler.html#exte...

tpaschalis 8y ago

(Newbie Airflow user here). I believe one easy way to do it is by using Airflow's 'sensors'. Sensors are operators which poke continuously with an action until it returns True (eg. until a file exists, an API gives a specific response, a process/query has finished). Another way to do it would be to 'XComs', small pieces of information flying between DAGs, or 'Triggers', but these require some more setup, and IMO depend more on the way you're setting up your tasks.

xkcd-sucks 8y ago
Yeah the issue is once a sensor fires once, it doesn't reset and keep firing on new data My homegrown solution is a sensor at the beginning, and at the end an airflow api call to trigger a dag run of the same dag. Not DagRunOperator because then no dag would never finish due to infinite recursion It seems kinda sketchy so I'm considering a lower level Celery implementation or even GenStage

yzmtf2008 8y ago

For "event-driven" scheduler, you may want to have a streaming system like Flink or Spark Streaming instead. You can schedule the startup of these pipelines with Airflow (e.g., using BashOperator), and handle the actual streaming with the streaming systems directly. We use Airflow with this pattern extensively.