3 ms·

They’re assuredly checkpointing, which is SOP for any long training. Lots of things can go wrong besides power loss.

• esseph 2mo ago
  If yout backup power is unreliable it could slow down your model release by days or weeks.
  • mjhay 2mo ago
    Sure. Reasonably reliable backup power is a very good thing, but losing power isn’t necessarily a catastrophe.