2 ms·
Oh I didn't see much in Nimbo with a quick glance but reading more closely > We immediately resume training after interruptions, using the last model checkpoint
by JZL003 5y ago
Oh I didn't see much in Nimbo with a quick glance but reading more closely
> We immediately resume training after interruptions, using the last model checkpoint via persistent EBS volume.
Makes sense, just save checkpoints to disk. What I'm doing is more CPU bound and not straight ML so less easily check-pointed, sadly. Cool though, it's worth jumping through hoops for 70% reduction