At Airbnb, we were using SubDAGs to try to manage large number of tasks in a single DAG. This allowed organizing tasks and drilling down into failures more easi

At Airbnb, we were using SubDAGs to try to manage large number of tasks in a single DAG. This allowed organizing tasks and drilling down into failures more easily but came with its own challenges. In more recent versions of Airflow, TaskGroups (https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html#taskgroups-vs-subdags https://airflow.apache.org/docs/apache-airflow/stable/concep..., https://www.astronomer.io/guides/task-groups/ https://www.astronomer.io/guides/task-groups/ ) were made to help this a little bit. Hopefully that helps a bit. At ~1k nodes in the graph introspection becomes hard anyway, as others have suggested, breaking it down if possible might be a good idea.