3 ms·
Or maybe multimodal models. If Google used YouTube as training data, they wouldn't run out of data in a while. Just feed it video once you run out of text.
by macrolime 4y ago
Or maybe multimodal models. If Google used YouTube as training data, they wouldn't run out of data in a while. Just feed it video once you run out of text.
- brian_cloutier 4y agoOr maybe hallucinating additional training data: my impression is that AlphaFold was able to get around a lack of sufficient data by training an initial model and having it make a bunch of predictions, then the high-confidence predictions from that set were used to train the final model. Given that language models seem to mostly know when they're making correct predictions [1] this method might be useful for stretching the available datasets into something larger without falling into the same pitfalls repeating data would give you. And if you squint it kind of looks like daydreaming? When I'm learning a new skill I often find myself playing back scenes and internally running some simulations of what I would have done differently. [1] https://arxiv.org/abs/2207.05221 https://arxiv.org/abs/2207.05221
- macrolime 4y agoInteresting idea. If a person sits down and writes a bunch of essays, that usually leads to improved language skills. Maybe hallucinating additional training data could be something somewhat similar for transformer networks.