2 ms·
Synthetic datasets are useless (other than for very specific purposes, such as enforcing known strong priors, and even then it's way better to do it directly by
by zozbot234 2y ago
Synthetic datasets are useless (other than for very specific purposes, such as enforcing known strong priors, and even then it's way better to do it directly by changing the architecture). You're better off spending that compute by making multiple passes over the data you do have.
- HeatrayEnjoyer 2y agoThis is contrary to what the big AI labs have found. Synthetic data is the new game in town.
- kranke155 2y agoIlya is saying it doesn’t work in this talk apparently.
- seanhunter 2y agoIlya could be wrong. I don’t think the question is decided yet in general. We already know that in lots of fields fake data can be used in ways that are as useful as or even more useful than the real thing[1], but in my understanding that tends to be situations where we have an objective function that is unambiguous and known beforehand. Meta has some very impressive work on synthetic data for training and my (uninformed) read was that is the state of the art in eg voice recognition at the moment.[2] [1] Eg Sobel sequences in a monte carlo simulation instead of real random numbers. They allow better coverage of the space of a simulation from fewer paths. https://www.sciencedirect.com/science/article/abs/pii/0041555367901449 https://www.sciencedirect.com/science/article/abs/pii/004155... [2] Seems a good overview is https://arxiv.org/html/2404.07503 https://arxiv.org/html/2404.07503
- toxik 2y agoMost priors are not encodable as architecture though, or only partially.