3 ms·
I have yet to see anyone explain or explore why text could not be trained on multiple times. And is there any reason a text passage wouldn’t be just as good run
by dougmwne 4y ago
I have yet to see anyone explain or explore why text could not be trained on multiple times. And is there any reason a text passage wouldn’t be just as good run backwards as forwards? Predicting the previous word in a sentence seems just as relevant to grasping semantic meaning as the next word.
- blurbleblurble 4y agoOne of the first transformers, BERT, was and still is bidirectional.
- visarga 4y agoThey might fear unwanted memorsation/regurgitation of training content.
- ALittleLight 4y agoBut shouldn't they try it? Has someone tried it?
- thomashop 4y agoThis is already how it works. When you train a deep learning model you show the samples of the training data hundreds of times. Of course depends on the size of your training set and your budget, etc. You often train for 100s of epochs. One epooch means one pass over the training set.
- navjordj 4y agoThe original T5 models were trained with masked language modelling similar to BERT and afterwards also trained on a autoregressive task similar to GPT.