My process is to separate numbered "scratch" notebooks in which I do one-off analysis from "pipeline" notebooks/code that process data for use in later analysis

My process is to separate numbered "scratch" notebooks in which I do one-off analysis from "pipeline" notebooks/code that process data for use in later analysis/modeling. Pipeline notebooks are made to take input/output locations from environment variables and run with nbconvert. The pipeline code is run by a DAG-like runner (sort of like Make) that is aware of the parameters and what level of the pipeline they are used at so that I can trace back outputs to the parameters and files used at every stage of the model. For stages that are notebooks I can also see the HTML generated by that run of the notebook. Unfortunately I've looked far and wide and there's no real open source version of what I've described, but I'm hoping to open source it once I've worked through the kinks.