That's a great point! Machine Learning reproducibility is a huge problem, both in Academia and within companies. Comet.ml tracks every run of your script, the h

That's a great point! Machine Learning reproducibility is a huge problem, both in Academia and within companies. Comet.ml tracks every run of your script, the hyper-params (pulled automatically from the ML library when possible) and your results. So in the example you gave we would be able to track the DS loading pre-trained weights, manually changing hyperparams and the datasets hash. In such complex cases there's still a little discipline required on the user behalf but overall it's just including a few calls to our Experiment object.