I wonder how to use a devops flow for machine learning workloads. Something we are working on at my current job is how to balance the speed of releasing togethe

I wonder how to use a devops flow for machine learning workloads. Something we are working on at my current job is how to balance the speed of releasing together with keeping data, pipeline and model versions aligned (where it for instance takes multiple days to release a new model). Releasing new features which need a model retraining is currently a very jarring experience.