5 ms·
Here is a summary: - Training on a mixture of fill-in-the-gaps (a few missing words) and denoising (every word slightly corrupted) produces better LLMs than ei
by whatshisface 4y ago
Here is a summary:
- Training on a mixture of fill-in-the-gaps (a few missing words) and denoising (every word slightly corrupted) produces better LLMs than either one alone.
- This advantage (that of using both metrics at once) can be gained with just a little extra training on a model previously trained with only one of them.
This results in 2-4% improvements on most tasks, with a couple really big improvements (+20%) and one quite surprising (+60%) on a few BigBench tasks. The large percent improvements on the BigBench tasks seem to have more to do with the low initial performance than the new performance being outstanding; the 60% was from 7.6% right to 12.5% right.
- mcint 4y agoThank you for the summary! I quite like this idea, train on a mixture of simple critics, or really here, simple sources of noise.
- deleted 4y ago[deleted]
- sebzim4500 4y ago>the 60% was from 7.6% right to 12.5% right. That seems like an absurd metric for judging the improvement in a test. It is wrong on both extremes, 1% to 5% is not really a 400% improvement and 99% to 100% is a drastic improvement despite only being a ~1% improvement by this metric.
- vinaychandranp 4y agoRelative error reduction might be a better way to talk about this. 1% to 5% would result in a relative error reduction of ~4% and 99% to 100% would be a 100% reduction in the error.