4 ms·
I looked at Airflow a long time ago, and it seemed focus on running tasks based on a time-based schedule. For example, if I needed to create a daily sales repor
by Buttons840 6y ago
I looked at Airflow a long time ago, and it seemed focus on running tasks based on a time-based schedule. For example, if I needed to create a daily sales report every night, Airflow would be great because I could set up a timer to kick off a reporting process every night. Then I could do things like rerun the last 5 days of reports if needed, etc.
I was more interested in event-based tasks though. For example, a sale happens, and then a report about that individual sale gets generated. At the time, this seemed to require some Airflow hacks and would cause me to miss out on some of its features. Is this still the case?
- f311a 6y agoI think you need an airflow sensor.
- dimberman 6y agoA sensor would also work here. Especially with the new SmartSensor feature, sensors are basically free so you can set them up for event-based DAG executions.
- matthewbauer 6y agoWhen I looked at it, it looked like sensors were all polling based. Things like wait until another task completes or this file appears. Is there a way to make them push/event based?
- dimberman 6y agoWe didn't separate task instances from timestamps YET purely because there was already so much to release that we didn't want to add more potential for bugs/upgrade difficulties. I believe this is on the docket for 2.1 or 3.0 depending on whether it requires a breaking change (that said we plan to release much more frequently going forward so we're planning to have this feature in 2021)
- recov 6y agoAnother option besides sensors is using airflows API, which with 2.0 is much improved - http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/stable-rest-api-ref.html http://apache-airflow-docs.s3-website.eu-central-1.amazonaws...