2 ms·
They said within 6 months, however by that time there will likely be something better.
by p1esk 8y ago
They said within 6 months, however by that time there will likely be something better.
- marvin 8y agoReleasing publicly AI tech that is six months behind the state of the art, might be a reasonable compromise against the fear of nefarious use. That way, more people know that the tech exists before it's available to everyone, and get a 6-month window to adapt. At the same time, the risk is lower that a different team develops similar capabilities and widely deploy it without anyone even knowing that what they do is possible. Wouldn't be surprised if this pattern is followed even more in the future. It's an acknowledgement that there are many angry cats that will (almost unstoppably) be let out of the bag in the next decades, but attempting at releasing the cats in the most responsible manner possible.
- speedplane 8y ago> They said within 6 months, however by that time there will likely be something better. There has. BERT has come out and is better.
- cthalupa 8y agoErm, isn't a big part of this whole story that GPT-2 > BERT -> GPT?
- gwern 8y agoBERT is bidirectional. How do you use that for language generation?
- visarga 8y agoIt will generate words for every [UNK] in its input sequence.
- gwern 8y agoIt'll generate one token because it's trained to predict one missing UNK, as I understood it. What is the scaffolding? Do you generate random sentences and iterate repeatedly? And how does that get you whole coherent paragraphs? (Has anyone demonstrated that this actually works with BERT?)
- speedplane 8y agoBERT can pretty easily be used to generate text. It's intended to be used as a base model and fine-tuned with an additional model on top. The fine-tuning model could then be trained to generate sentences with the underlying language model powered by BERT.