Thanks for your question! Good points -- those are not easy problems! I'd be really curious about what the community thinks, but here are my opinions: While Ha

Thanks for your question! Good points -- those are not easy problems! I'd be really curious about what the community thinks, but here are my opinions: While Hamilton/DAGWorks is mainly for expressing the pipeline/abstracting away the infrastructure, DAGWorks can help make the model lifecycle easy as well: - Hamilton pipelines can be run anywhere, including in an online setting - Breaking into functions can make it modular/easy to annotate and gather post-hoc analysis - Hamilton pipelines specify the data movement in code -- providing a source of truth That said, I think the ecosystem for doing this is much cleaner/easier to manage than it used to be -- the MLOps stack is far more sophisticated. Scalable/reliable compute (spark, Modin), easy storage (snowflake, new feature store technology), and more model experiment-as-a-service type systems (mlflow, model-db, etc...) have made these less of a difficult problem than it was in the past. As these permeate the industry and we develop more standards, I think that it pushes the problem up a level -- rather than figuring out exactly how to solve these, the difficult part is looping together a bunch of systems that all do it fairly well but (a) require significant expertise to manage and (b) often result in code that's super coupled to the systems themselves (making it hard to test). DAGWorks wants to decouple these from the pipeline code, enabling you to choose which systems you delegate to and not have to worry about it. Furthermore, we think that smaller pipelines are actually super underserved in ML/data science community -- E.G. pipelines that don't have a lot of the "moving data around" problems but can be run on a single machine. I've seen these suffer from getting too complex/being difficult to manage, and we think Hamilton can solve this out of the box. Thoughts?