4 ms·
The article seems to build on several foundations, but one is critical: That the AIs need enormous amount of data, and that that data has to be from a pool that
by Arnt 3y ago
The article seems to build on several foundations, but one is critical: That the AIs need enormous amount of data, and that that data has to be from a pool that people can now poison with AI-generated data. If that doesn't hold, nothing in the article holds.
And it doesn't seem clear to me. It may be true, but far from obvious.
For example, DALL-E, which was better than its predecessors. Was it better because of mode input data, that is, was more input data a necessary condition for its improvement? Or even the biggest reason? Reading the OpenAI blog makes it sound as if they had new kinds of AI models, new ideas about models, and that the use of data from a web crawler was little more than a cost optimisation. If that's true, then then it should be possible to build another generation of image AI by combining more new insights with, say, the picture archives of Reuters and other companies with archives of known provenance.
Maybe I'm an elitist snob, but the idea that you can generate amazing pictures using the Reuters archive sounds more plausible than that you could do the same using a picture archive from all the world's SEOspam pages. SEOspam just doesn't look intelligent or amazing.
- spacebanana7 3y ago> That the AIs need enormous amount of data I was surprised to learn that it didn’t take an enormous amount of data to train Llama. Meta used 1.4 trillion tokens training Llama, but that’s only about a million times the Harry Potter collection [1]. Given that the kindle store has 12 million books [2], it’s credible to get 1.4T tokens just from that. Twitter has 200 billion new tweets per year, so it’d only take 7 tokens per Tweet for that to produce a Llama sized dataset every year. [1] https://en.wikipedia.org/wiki/LLaMA https://en.wikipedia.org/wiki/LLaMA [2] https://blog.fostergrant.co.uk/2017/08/03/word-counts-popular-books-world https://blog.fostergrant.co.uk/2017/08/03/word-counts-popula... (1 million words) [3] https://www.omnicoreagency.com/twitter-statistics https://www.omnicoreagency.com/twitter-statistics