32 ms·
> it appears that someone managed to train a BERT model[1] to 90% of the GLUE score reported in the original BERT paper on a single GPU in ~100 hours Wondering
by lloeki 3y ago
> it appears that someone managed to train a BERT model[1] to 90% of the GLUE score reported in the original BERT paper on a single GPU in ~100 hours
Wondering what the GLUE score/training time curve would look like.
- londons_explore 3y agoindeed - 90% of the GLUE score might be nothing to get excited about if the last 10% is the difficult bit...
- nighthawk454 3y agoThat's not what's impressive here. > While we can see that BERT-Base performed better at every task; the results for this model would have been very good (possibly SOTA for a few tasks) in early 2018. SOTA - meaning no models of _any_ size at that time could do this. And they're achieving that now on a single consumer GPU with 1/30th the data and 1/40th the epochs. That's huge for improvements to training efficiency.