When it comes to scale and DS work I'd use the ploomber open-source (https://github.com/ploomber/ploomber https://github.com/ploomber/ploomber). It allows an ea

When it comes to scale and DS work I'd use the ploomber open-source (https://github.com/ploomber/ploomber https://github.com/ploomber/ploomber). It allows an easy transition between dev and production, incrementally building the DAG so you avoid expensive compute time and costs. It's easier to maintain and integrates seamlessly with Airflow, generating the DAGs for you.