4 ms·
And if the first "slug of data" contains something you need to remove? If you need to remove something you would need to roll-back to the last checkpoint that d
by supermatt 3y ago
And if the first "slug of data" contains something you need to remove? If you need to remove something you would need to roll-back to the last checkpoint that didnt include it.
- mynameisvlad 3y agoI mean, sure, but the point of frequent checkpoints is that these issues would be rooted out sooner rather than later. It would have been caught far sooner than “we have to redo everything from scratch”.
- Turing_Machine 3y agoWell, that's why you checkpoint a lot and feed in the new training data a little at a time, rather than dumping in a massive slug all at once. Right?
- supermatt 3y agoNot really. If they are ingesting peoples data to train their model, at what point are they checkpointing? Is data I submit 2 months ago that I would want removed not present in a recent checkpoint or would they have to "unwind" the last 2 months of training to remove it? What about people that would want something removed 6 months prior? What if something in the core model is to be removed? Checkpointing may help, sure, but it isnt going to allow you to remove a single piece of training data without headache, and potential substantial retraining. If that 3% from the GPs comment comment is uniformly distributed throughout the training history, the only way it can be reliably removed is to retrain from scratch.
- Turing_Machine 3y ago> If they are ingesting peoples data to train their model, at what point are they checkpointing? Multiple times per day if they're not incompetent. And I don't think they're incompetent.
- supermatt 3y agoAre you implying that removing the diff between checkpoints achieves the same effect? Ive never heard of this, but I suppose it may be possible. I suppose the "ghost" of the removed weights would also have shaped subsequent training though... Interesting idea...