One way to achieve that is to use externally triggered DAGs [1][2]. Instead of trying to make the Airflow scheduler work as event-driven though, I've more trie

One way to achieve that is to use externally triggered DAGs [1][2]. Instead of trying to make the Airflow scheduler work as event-driven though, I've more tried to adapt my approach. I've used Airflow as a bridge between a micro batch streaming system (Sparking Streaming --> S3, Airflow: S3 --> ...) successfully, but there are some challenges around the edge cases like files landing later than expected. One approach you can take is to have the Airflow sensor hit a file that's a proxy for all of the files being present. For example, if your process could write hundreds of S3 files, once it's finished the last write for that hour (even if that happens late for whatever reason), then it could write a top-level OK file that the sensor hits. Can you elaborate on your specific use case? [1]: https://stackoverflow.com/a/50592573/149428 https://stackoverflow.com/a/50592573/149428 [2]: https://airflow.readthedocs.io/en/stable/scheduler.html#external-triggers https://airflow.readthedocs.io/en/stable/scheduler.html#exte...