4 ms·

Thanks for this, I wasn't aware of it. From reading the docs, I don't see anywhere if Grid automatically handles spot interruptions to resume from last checkpoi

Thanks for this, I wasn't aware of it. From reading the docs, I don't see anywhere if Grid automatically handles spot interruptions to resume from last checkpoint which was the main focus of our internal tool. Have you used it btw ? and what has your experience been with Grid ?

barefeg 5y ago
Ah good point. They do have persistent storage so I’m guessing you can add code to do the same. I haven’t personally tried it since we developed our own solution in house. It basically does everything grid.ai does but with checkpoints. Though our solution does require adding a few lines of sdk code to handle the check pointing
- barefeg 5y ago
  I should add that where our solution falls short is in multi node training which grid.ai seems to have out of the box