4 ms·
The paper that Gary cited is just so stupid. If you're drawing training data from a fixed distribution randomly, of course as the size of your training data se
by CuriouslyC 3y ago
The paper that Gary cited is just so stupid. If you're drawing training data from a fixed distribution randomly, of course as the size of your training data set increases, the odds that you will get a random draw that is very similar to an existing piece of data increases, until you have to do a lot of random draws to see a new part of the distribution. That is the cause of the exponential increase in data requirements for linear increase in performance.
We just need to methodically sample the under sampled portions of the source distribution. Not more data, better data.
- hartator 3y agoHumans were able to get better data themselves. Which I think it is what we mean by actual AI.
- philipwhiuk 3y agoAh yes, "just sample under sampled portions" The problem is that they don't exist in discrete missed data sets they only exist in larger data sets that also contain large amounts of well sampled data. If you could separate out the existing data you could cherry-pick the under sampled data, but do that requires the very classifier you're trying to build in the first place. This reaction is like saying "if we knew everything we'd be able to answer every question"
- naasking 3y agoHis suggestion sounds more like human curated data sets rather than automated curation like you're assuming, although this is a good distinction to make.
- CuriouslyC 3y agoThat's not really true. You don't need to actually solve the problem, you just need to create a first order solution for optimizing your data set, which can enable better models that can be used to create a solution that is second order accurate, and so on iteratively.
- brookst 3y agoIsn’t it possible for out of band systems to improve data distribution? You don’t need a fancy model trained on all languages to tell you its training set underrepresented Finnish. You just need the training set curators to say “hey there’s no Finnish in the inputs”. If the goal is high quality models, why be a purist about only using the resulting model to measure deficiencies in the training set?
- nick7376182 3y agoAt some point autoencoding during training will take care of that, it seems. We kind of already it with 'tutor' training and gpt-generated training data, as well as synthetic logic and math problem sets.
- CuriouslyC 3y agoOh, having language models sit on top of autoencoders is 100% the right way to go, even if we weren't moving towards multi-modal models, just because right now LLMs can be brilliant in one language and retarded in another based on the training data set. Putting an autoencoder in front and using "language agnostic" encoding would solve that problem.
- optimalsolver 3y ago>autoencoding during training Could you elaborate on this?