3 ms·

Can you talk about how CometML fits into the real world state of ML training tracking, which is a pretty terrible Wild West of non-reproducible practices and pr

by yodon 9y ago

Can you talk about how CometML fits into the real world state of ML training tracking, which is a pretty terrible Wild West of non-reproducible practices and processes? There have been articles and comments here on HN about the sorry state of ML trackability, with papers being published on models whose training is not reproducible because no one really knows how it was trained. One in particular (I apologize for not having retained the link) described researchers starting with partially trained models they had lying around (with undocumented and unknown prior training applied), manually changing hyper parameters mid training while watching the learning progress, swapping different training sets in and out, and etc. From what I see, the problems in ML reproducibility aren’t in the code, they are in the external human processes that are used to drive and train the models (essentially bad DevOps practices more than bad dev practices). Do you help with these kind of real-world trackability and reproducabilty scenarios?

gidim 9y ago
That's a great point! Machine Learning reproducibility is a huge problem, both in Academia and within companies. Comet.ml tracks every run of your script, the hyper-params (pulled automatically from the ML library when possible) and your results. So in the example you gave we would be able to track the DS loading pre-trained weights, manually changing hyperparams and the datasets hash. In such complex cases there's still a little discipline required on the user behalf but overall it's just including a few calls to our Experiment object.
laythea 9y ago
Im no expert but reading that it sounds like a bunch of people poking at knobs and dials in order to see which dials work and which don't, and then not understanding why...?
jdoliner 9y ago
My sense is that CometML is tackling some but not all of the problems you're talking about here. Specifically I think it's pretty focused on tracking ML models so you can keep track of how models are performing, how they're improving etc. But I think, and the other comment seems to suggest, that external systems and practices will be required to make it so these results can be reproduced. Making ML experiments reproducible is a different and fairly challenging problem. You need to have a way to essentially snapshot everything that goes into training a model, training data, code, tuning parameters maybe even specific hardware it needs to run on. As you mention, this is as much a DevOps problem as it is an ML problem, maybe even more so. I think you'd have good luck pairing CometML with a system like Pachyderm [0] (which, full disclosure, is a system I helped write). Pachyderm can handle the Devopsy reproducibility piece and be paired with CometML to get insight into what's actually going on with the models. [0] https://github.com/pachyderm https://github.com/pachyderm

by yodon 9y ago

gidim 9y ago

That's a great point! Machine Learning reproducibility is a huge problem, both in Academia and within companies. Comet.ml tracks every run of your script, the hyper-params (pulled automatically from the ML library when possible) and your results. So in the example you gave we would be able to track the DS loading pre-trained weights, manually changing hyperparams and the datasets hash. In such complex cases there's still a little discipline required on the user behalf but overall it's just including a few calls to our Experiment object.

laythea 9y ago

Im no expert but reading that it sounds like a bunch of people poking at knobs and dials in order to see which dials work and which don't, and then not understanding why...?

jdoliner 9y ago

My sense is that CometML is tackling some but not all of the problems you're talking about here. Specifically I think it's pretty focused on tracking ML models so you can keep track of how models are performing, how they're improving etc. But I think, and the other comment seems to suggest, that external systems and practices will be required to make it so these results can be reproduced. Making ML experiments reproducible is a different and fairly challenging problem. You need to have a way to essentially snapshot everything that goes into training a model, training data, code, tuning parameters maybe even specific hardware it needs to run on. As you mention, this is as much a DevOps problem as it is an ML problem, maybe even more so. I think you'd have good luck pairing CometML with a system like Pachyderm [0] (which, full disclosure, is a system I helped write). Pachyderm can handle the Devopsy reproducibility piece and be paired with CometML to get insight into what's actually going on with the models. [0] https://github.com/pachyderm https://github.com/pachyderm