2 ms·
I think it's better to place everything in one DAG if that solves the problem. If it doesn't then sensors are ok I guess, but I would try to avoid them otherwis
by simo7 6y ago
I think it's better to place everything in one DAG if that solves the problem. If it doesn't then sensors are ok I guess, but I would try to avoid them otherwise.
- prions 6y agoI really disagree here. Monolithic dags are a bigger pain to manage. Breaking them out into smaller dags makes retrying/backfilling/etc a lot more straightforward. It also lets you reuse those pieces more easily. We have compute DAGs that are the upstream dependency for many other dags. Originally, this dag was monolithic and loaded data into one table. But because the dag is split into computation and loading we can easily add more downstream dags without changing how the first one operated.