3 ms·
You make an evaluation set and only apply the new model when the score improves. In time you extend your evaluation set to include all the edge cases you discov
by visarga 4y ago
You make an evaluation set and only apply the new model when the score improves. In time you extend your evaluation set to include all the edge cases you discover in production. If the evaluation set is large enough you can have a degree of confidence. It's more like working with people than working with software - we have exams for people, but exams can only sample knowledge in a limited way.
- teraflop 4y agoBy "score", do you mean manually reviewing and rating the outputs from the evaluation set every time the model changes? Because it's hard to imagine an automated evaluation system that would be sophisticated enough to tell whether the outputs were getting better or worse -- unless, again, it was as smart as GPT-3.