For "event-driven" scheduler, you may want to have a streaming system like Flink or Spark Streaming instead. You can schedule the startup of these pipelines wit

For "event-driven" scheduler, you may want to have a streaming system like Flink or Spark Streaming instead. You can schedule the startup of these pipelines with Airflow (e.g., using BashOperator), and handle the actual streaming with the streaming systems directly. We use Airflow with this pattern extensively.