Versioning is indeed a hard topic, especially for data science/engineering projects in production. When you have a pipleine defined as a complex DAG of operati

Versioning is indeed a hard topic, especially for data science/engineering projects in production. When you have a pipleine defined as a complex DAG of operations, you can't just version the entire thing, unless you have enough resources to re-compute from scratch with every change, which is wasteful. So then, you have to keep track of data dependencies and their versions if you would like to ensure reproducibility. Versioning code isn't enough when you have runtime parameters that affects output data, and you want to stay flexible by allowing experimenting and re-running computations with different parameters, to be able to iterate quickly. Which poses a lot of challenges. And there doesn't seem to be a framework that easily solves those issues out of the box. I'm closely watching Dagster (https://dagster.io https://dagster.io), as they seem to be aware of those challenges (for example for versioning: https://docs.dagster.io/guides/dagster/memoization https://docs.dagster.io/guides/dagster/memoization), but I didn't try it yet; it introduces a lot of concepts and has a steep learning curve.