4 ms·

Well, that's why you checkpoint a lot and feed in the new training data a little at a time, rather than dumping in a massive slug all at once. Right?

• supermatt 3y ago
  Not really. If they are ingesting peoples data to train their model, at what point are they checkpointing? Is data I submit 2 months ago that I would want removed not present in a recent checkpoint or would they have to "unwind" the last 2 months of training to remove it? What about people that would want something removed 6 months prior? What if something in the core model is to be removed? Checkpointing may help, sure, but it isnt going to allow you to remove a single piece of training data without headache, and potential substantial retraining. If that 3% from the GPs comment comment is uniformly distributed throughout the training history, the only way it can be reliably removed is to retrain from scratch.
  • Turing_Machine 3y ago
    > If they are ingesting peoples data to train their model, at what point are they checkpointing? Multiple times per day if they're not incompetent. And I don't think they're incompetent.
    • supermatt 3y ago
      Are you implying that removing the diff between checkpoints achieves the same effect? Ive never heard of this, but I suppose it may be possible. I suppose the "ghost" of the removed weights would also have shaped subsequent training though... Interesting idea...