3 ms·
Are there no incremental ways to verify that it's working?
by Jeff_Brown 3y ago
Are there no incremental ways to verify that it's working?
- olalonde 3y agoYes, you can. After every training batch, the network's loss function is updated and you can track it over time to verify that it's going down. Sometimes, the loss can get stuck on a plateau, or even go up, and it's not clear whether it's worthwhile to keep training or not.
- brucethemoose2 3y agoTraditionally, training runs can "explode" and fail, but there are methods to incrementally back them up and resume when that happens, see https://www.mosaicml.com/blog/mpt-7b https://www.mosaicml.com/blog/mpt-7b
- mark_l_watson 3y agoOf course, you can measure loss on a dev set, etc. However, LLMs are so complex that until they are done training and many people evaluate them, there will be misses.