3 ms·
> They said within 6 months, however by that time there will likely be something better. There has. BERT has come out and is better.
by speedplane 7y ago
> They said within 6 months, however by that time there will likely be something better.
There has. BERT has come out and is better.
- cthalupa 7y agoErm, isn't a big part of this whole story that GPT-2 > BERT -> GPT?
- gwern 7y agoBERT is bidirectional. How do you use that for language generation?
- visarga 7y agoIt will generate words for every [UNK] in its input sequence.
- gwern 7y agoIt'll generate one token because it's trained to predict one missing UNK, as I understood it. What is the scaffolding? Do you generate random sentences and iterate repeatedly? And how does that get you whole coherent paragraphs? (Has anyone demonstrated that this actually works with BERT?)
- speedplane 7y agoBERT can pretty easily be used to generate text. It's intended to be used as a base model and fine-tuned with an additional model on top. The fine-tuning model could then be trained to generate sentences with the underlying language model powered by BERT.