4 ms·
It worked well enough for BERT: https://en.m.wikipedia.org/wiki/BERT_(language_model) https://en.m.wikipedia.org/wiki/BERT_(language_model)
by riwsky 4y ago
It worked well enough for BERT: https://en.m.wikipedia.org/wiki/BERT_(language_model) https://en.m.wikipedia.org/wiki/BERT_(language_model)
- tivert 4y ago> It worked well enough for BERT: https://en.m.wikipedia.org/wiki/BERT_(language_model) https://en.m.wikipedia.org/wiki/BERT_(language_model) Great, the next education fad will be trying to teach kids like some software engineer trained his algorithm. "Hey kids, start your integration over this new 1TB data set. Remember to adjust your weights after each iteration!"
- amluto 4y agoBERT wasn’t learning how to read. It was learning how to predict (or generate or understand or whatever) text.
- DonsDiscountGas 4y agoIt's true, masked-language-modelling has been hugely successful in natural language processing. So a priori I wouldn't have guessed it would work so poorly for humans and that phonics-based teaching is so much better. But the evidence is clear. Just because MLM can work doesn't mean it does work in all contexts. We get to change the software (and hardware although that has been less relevant IMHO) behind ML models as much as we want, for humans we need to adapt the training methods.