4 ms·
> - Don't use it for tasks that don't require idempotency (eg. a job that uses a bookmark). You can totally design your tasks to be idempotent - but its up to
by prions 6y ago
> - Don't use it for tasks that don't require idempotency (eg. a job that uses a bookmark).
You can totally design your tasks to be idempotent - but its up to you to make them that way. The scheduler or executor doesn't have any context into your job.
This is why I encourage people to use a unified base operator and then pass their own docker containers to it. Aka like how https://medium.com/bluecore-engineering/were-all-using-airflow-wrong-and-how-to-fix-it-a56f14cb0753 https://medium.com/bluecore-engineering/were-all-using-airfl... outlines it.
> - Don't use it for latency-sensitive jobs (this one should be obvious).
IIRC this is being addressed in Airflow 2.0
> - Don't use sensors or cross-DAG dependencies.
This is a little extreme. I've never ran into issues with cross dag dependencies or sensors. They make managing my DAGs way easier because wee can separate computation dags from loading dags.
context: I built/manage my company's Airflow platform. Everything is managed on k8s.
- simo7 6y ago> You can totally design your tasks to be idempotent Yes of course, I mean Airflow is not a good fit for the tasks you don't want to be idempotent (I think most but not all tasks should be idempotent). > I've never ran into issues with cross dag dependencies I believe Airflow docs advice against them when possible. I see why from my experience: less visibility and more complexity, especially for backfills.
- domenp 6y ago> context: I built/manage my company's Airflow platform. Everything is managed on k8s My team is running Airflow on a single node but we're slowly outgrowing this setup. We're considering running jobs on k8s. Curious what's your setup like? Is your cluster of a fixed size or does it scale with the load?
- ForHackernews 6y agoUsing the KubernetesPodOperator for everything adds a huge amount of overhead. You still need Airflow worker nodes, but they're just babysitting the K8S pods doing the real work. I know it's 2020 and memory is cheap or whatever, but Airflow is shockingly wasteful of system resources.