This is something I often wonder about. In my own experience the perceived productivity gains in prototyping often come with externalized technical debt that le

This is something I often wonder about. In my own experience the perceived productivity gains in prototyping often come with externalized technical debt that lead to an overall loss of productivity. Notebook-driven development tends to produce models that are too tightly coupled with an underlying data pipeline, as the workflow very much facilitates this. There’s a measured loss of productivity when a model and pipeline need to be decoupled, e.g., to split the pipeline or repurpose the model. Obviously coupling components is not unique to notebooks, but there just seems to be a culture in notebook-driven development to disproportionately push poorly designed code artifacts. The MLEs and data scientist I’ve worked with that write simple functional modules tend to introduce less technical debt, at the expense sometimes of taking longer to push out a demo prototype. All that said, I think the aforementioned issues are not truly intrinsic to a notebook-driven development, but rather an emerging culture that has rather aptly been called Potemkin data science: https://mcorrell.medium.com/potemkin-data-science-fba2b5ba5cc6 https://mcorrell.medium.com/potemkin-data-science-fba2b5ba5c...