3 ms·

Does Airflow support streaming the outputs to downstream tasks? I tried to do something like this with Prefect but with Prefect you have to wait for the upstrea

• shmoogy 5y ago
  My understanding is each task is supposed to idempotent, so I don't believe this is a valid use for Airflow.
  • arthurcolle 5y ago
    Yeah your understanding is correct but if you relax the idempotency constraint you can achieve a lot more with just a little bit extra logic in your interface layer with other internal services or potentially other mechanisms to ensure consistency. YMMV
• arthurcolle 5y ago
  Hit me up at my email in my profile if you want to chat about this stuff, I have a lot of thoughts on this but it's probably off topic for this post and I usually am just hacking stuff together to get my systems up and running. What you're talking about is very sensible and I was equally surprised that Airflow didn't support long running tasks but you can layer over the workflow orchestration system a kind of ad-hoc higher order system that enables what you speak of. It kind of feels ugly but can get a lot done. There are definitely ways to accomplish what you are saying using a combination of DockerOperators + ephemeral WebSocket servers running within containers as semi-long running tasks, and basically just have a dumb/heavy Redis container that persists to run streaming between the coordination architecture across these data flow jobs. "Work in progress" lol! EDIT: updating Airflow from 1.10.10 to 2.1.2 recently was a huge pain in the ass for what it's worth, good luck to all our fellow protagonists that are dealing with multi tens of thousands of task DAG setups... big ooooff