as a daily jupyter user, i like this .py DAG approach a lot - more so for visualizing complex workflows than addressing a potentially corrupt hidden state. the

as a daily jupyter user, i like this .py DAG approach a lot - more so for visualizing complex workflows than addressing a potentially corrupt hidden state. there isn't much *data science functionality* integrated into jupyter core apart from displaying static dataframes (nor is their internal desire to add it), and jupyter *extensions/widgets* should really be built with dash/streamlit, so i'd encourage you to make an ecosystem. having spent a lot of time with airflow recently, it's designed for scheduling recurring jobs, not *batch processing.* meanwhile dagster is not meant for computationally intensive/ long-running jobs and the dataset-centric approach is offputting overengineering = since you made the cells functions already, it would be cool if you could make multiprocessing/multithreading an option in the cell UI and help keep track of multi-state variables/execution happy to give feedback/ user interview = linkedin.com/in/laynesadler/