5 ms·

>> The solution is to just have more data and better data. Nah, sorry, that is just trying to put out the fire by throwing fuel at it. The big, big weakness of

>> The solution is to just have more data and better data. Nah, sorry, that is just trying to put out the fire by throwing fuel at it. The big, big weakness of neural networks right now is their reliance on gigantic datasets that require gigantic computational resources. Neural nets need those because they can't generalise to unseen data. So people try to get them to see as much data as possible during training. They still overfit, but if they can overfit to a diverse enough dataset, then they can be useful in practice, even if that's only to solve narrow, specific instances of a problem (like in the domino recognition system in the article). To address this weakness what is needed is to find ways to make neural nets less reliant on data, not to find ways to make more data. Make neural nets capable of generalising robustly to unseen data, from few training instances. Then you don't need to train in a simulation. Of course that would require a radical rethink of how deep neural nets are trained (even perhaps whether they remain "deep", or whether they are trained using gradient descent, the sources of their data-hunger). Trying to make more data by simulation is only kicking the can down the road and the only effect it can possibly have is to push the time at which the real limitation must be really addressed even further down the line so that an other generation of researchers has to deal with it while the current generation can keep getting their papers published and their grants granted. See Vladimir Vapnik's challenge to the machine vision community: https://youtu.be/bQa7hpUpMzM?t=492 https://youtu.be/bQa7hpUpMzM?t=492 To summarise: learn to identify MNIST digits from 60 examples of each class, rather than 6000, while retaining current accuracy. (My words now:) Improve sample efficiency to improve neural nets. Neural nets have shown a remarkable ability to work well when large amount of resources are available. Now, do like everyone else does in computer science and try to make them (sample) efficient.

teruakohatu 5y ago
> learn to identify MNIST digits from 60 examples of each class, rather than 6000 More like 60,000 per class once augmentation has generated a bunch of new samples from each orginal!
treesprite82 5y ago
> To summarise: learn to identify MNIST digits from 60 examples of each class, rather than 6000, SOTA accuracy on a similar but more challenging problem (5-shot 20-way rather than 60-shot 10-way) appears to be around 99.6%: https://paperswithcode.com/sota/few-shot-image-classification-on-omniglot-5-1 https://paperswithcode.com/sota/few-shot-image-classificatio... > while retaining current accuracy. Depends how strict you're being with this. There's room for the gap to shrink, but I think on average classifiers (whether organic or machine) with a large number of examples to go off of will always perform at least marginally better than classifiers with fewer examples.
- YeGoblynQueenne 5y ago
  >> SOTA accuracy on a similar but more challenging problem (5-shot 20-way rather than 60-shot 10-way) appears to be around 99.6%: https://paperswithcode.com/sota/few-shot-image-classificatio https://paperswithcode.com/sota/few-shot-image-classificatio... I'm aware of results like that but they're just kicking the can down the road with the other foot: they push the problem of training with big data to the pre-training stage and then claim to do "few-" or "one-shot" learning at the end, or even "zero-shot" which is egregious abuse of terminology (and it's very sad that it's accepted terminology). It's like the Aesop's fable where the sparrow hid in the eagle's feathers and jumped up at the last moment to claim "I'm the bird that flies the highest!". Vapnik's point in the interview I linked above is that you should not need a lot of data for anything, including pre-training. His challenge is for the community to find what he calls good "predicates" which are primitive functions (I think of them as feature detectors) that can be composed into a good statistical invariant, a function representing a high-level concept while having good out-of-dataset generalisation ability. His claim is that if you have a bunch of good predicates and good invariants, then you don't need a lot of data, because the generalisation ability of the good predicates makes up for it. Or, seen another way, lots of data is needed _in the case when_ a good predicate is not known. In a certain way, transfer learning, or meta-learning in the case of the paper you link, is a step towards the right direction, but the reliance on big data for pre-training suggests that the models are still not learning good representations that generalise well - so they still need big data to make up for it.
twayt 5y ago
As long as (stochastic) gradient descent and random initialization is the state of the art, more data will be better.
- kd5bjo 5y ago
  Random initialization is already starting to give way. One of the more promising approaches right now is pretraining with synthetic data to obtain “good” initial weights. Once that converges, you’ve got a network that internally recognizes interesting features which is a bettter-than-random starting point for training with real-world data. cf. https://arxiv.org/abs/2106.05963 https://arxiv.org/abs/2106.05963
zwaps 5y ago
This is always an interesting discussion. I think what you say is probably right "in practice" - that is, it is where ML should move now to improve its ability to generalize. However, from a more statistical viewpoint, this approach is not "theoretically correct". The issue of unseen data arises for one simple reason: the model does not capture causal mechanisms in the DGP. The model instead learns a reduced form that represents the training sample. If the training sample is large, and the model expressive, this leads to incredible results. However, it fundamentally differs from a causal model in that it could not do true out-of-sample predicitions if these are counterfactual to the data. That is, it may work, but we can not rely on it. It may randomly fail (and not in dependence on robustness of smoothness of the model, but in joint dependence to the counterfactual DGP, if that makes sense) This issue stands independent of the amount of data. Indeed, if you had a model that identified all relevant causal mechanisms, then of course more data is still better. In some areas, ideas from causal statistical analyses seep in. Some physical systems are well described by ODE and PDEs - which can be learned by DNNs but retain their causal structure. I see no general way forward, though. The approach you propose moves along the general idea of ML/AI - pose better challenges to the model and see if it works. This has worked well, perhaps it is the way to go.
- YeGoblynQueenne 5y ago
  >> The issue of unseen data arises for one simple reason: the model does not capture causal mechanisms in the DGP. The model instead learns a reduced form that represents the training sample. Thanks for articulating it so clearly, that's very much how I think of it! >> This issue stands independent of the amount of data. Indeed, if you had a model that identified all relevant causal mechanisms, then of course more data is still better. Well, my point -and, as far as I understand it, Vapnik's point, in my link above- is that if a model can generalise well, then it doesn't need a lot of data, even if a lot of data is available. In terms of a model that can identify causal relations, then it should be able to do so without lots of data. My intuition for that is that if the model needs lots of data it will inevitably learn to represent the sampling error in the data generation process (DGP?). A model that can learn good generalisations from few data on the other hand is in a sense immunised against sampling error because it can ignore most of the data. Vapnik makes an analogy about a good teacher. He asks, what is it that a good teacher does that helps his students learn better? He answers it by saying that the good teacher gives his students "good predicates", what I would call background knowledge, from which the students can build good statistical invariants. But the students must also be able to build good statistical invariants _on their own_, otherwise it doesn't matter what good predicates the teacher can give them, the student's can't learn good invariants. I think that's the point of the challenge - Vapnik is asking for learning from few examples (not unreasonably few, I think) as a way to demonstrate that a system is learning good underlying principles- his good predicates and statistical invariants. In the past the machine learning community has sidestepped the issue by finding ways to augment data where it was scarce and then claim progress in certain problems for which there were initially few examples (for example, Bongard problems and Winograd schemas) but Vapnik is not only calling for few data, but also for good predicates, which is I think a better defense than just asking for few-shot learning. In a sense I think if we find that more data is better, then that means the system is still not learning the good bits that make more data unnecessary.

teruakohatu 5y ago

> learn to identify MNIST digits from 60 examples of each class, rather than 6000 More like 60,000 per class once augmentation has generated a bunch of new samples from each orginal!

treesprite82 5y ago

> To summarise: learn to identify MNIST digits from 60 examples of each class, rather than 6000, SOTA accuracy on a similar but more challenging problem (5-shot 20-way rather than 60-shot 10-way) appears to be around 99.6%: https://paperswithcode.com/sota/few-shot-image-classification-on-omniglot-5-1 https://paperswithcode.com/sota/few-shot-image-classificatio... > while retaining current accuracy. Depends how strict you're being with this. There's room for the gap to shrink, but I think on average classifiers (whether organic or machine) with a large number of examples to go off of will always perform at least marginally better than classifiers with fewer examples.

YeGoblynQueenne 5y ago
>> SOTA accuracy on a similar but more challenging problem (5-shot 20-way rather than 60-shot 10-way) appears to be around 99.6%: https://paperswithcode.com/sota/few-shot-image-classificatio https://paperswithcode.com/sota/few-shot-image-classificatio... I'm aware of results like that but they're just kicking the can down the road with the other foot: they push the problem of training with big data to the pre-training stage and then claim to do "few-" or "one-shot" learning at the end, or even "zero-shot" which is egregious abuse of terminology (and it's very sad that it's accepted terminology). It's like the Aesop's fable where the sparrow hid in the eagle's feathers and jumped up at the last moment to claim "I'm the bird that flies the highest!". Vapnik's point in the interview I linked above is that you should not need a lot of data for anything, including pre-training. His challenge is for the community to find what he calls good "predicates" which are primitive functions (I think of them as feature detectors) that can be composed into a good statistical invariant, a function representing a high-level concept while having good out-of-dataset generalisation ability. His claim is that if you have a bunch of good predicates and good invariants, then you don't need a lot of data, because the generalisation ability of the good predicates makes up for it. Or, seen another way, lots of data is needed _in the case when_ a good predicate is not known. In a certain way, transfer learning, or meta-learning in the case of the paper you link, is a step towards the right direction, but the reliance on big data for pre-training suggests that the models are still not learning good representations that generalise well - so they still need big data to make up for it.

twayt 5y ago

As long as (stochastic) gradient descent and random initialization is the state of the art, more data will be better.

kd5bjo 5y ago
Random initialization is already starting to give way. One of the more promising approaches right now is pretraining with synthetic data to obtain “good” initial weights. Once that converges, you’ve got a network that internally recognizes interesting features which is a bettter-than-random starting point for training with real-world data. cf. https://arxiv.org/abs/2106.05963 https://arxiv.org/abs/2106.05963

zwaps 5y ago

This is always an interesting discussion. I think what you say is probably right "in practice" - that is, it is where ML should move now to improve its ability to generalize. However, from a more statistical viewpoint, this approach is not "theoretically correct". The issue of unseen data arises for one simple reason: the model does not capture causal mechanisms in the DGP. The model instead learns a reduced form that represents the training sample. If the training sample is large, and the model expressive, this leads to incredible results. However, it fundamentally differs from a causal model in that it could not do true out-of-sample predicitions if these are counterfactual to the data. That is, it may work, but we can not rely on it. It may randomly fail (and not in dependence on robustness of smoothness of the model, but in joint dependence to the counterfactual DGP, if that makes sense) This issue stands independent of the amount of data. Indeed, if you had a model that identified all relevant causal mechanisms, then of course more data is still better. In some areas, ideas from causal statistical analyses seep in. Some physical systems are well described by ODE and PDEs - which can be learned by DNNs but retain their causal structure. I see no general way forward, though. The approach you propose moves along the general idea of ML/AI - pose better challenges to the model and see if it works. This has worked well, perhaps it is the way to go.

YeGoblynQueenne 5y ago
>> The issue of unseen data arises for one simple reason: the model does not capture causal mechanisms in the DGP. The model instead learns a reduced form that represents the training sample. Thanks for articulating it so clearly, that's very much how I think of it! >> This issue stands independent of the amount of data. Indeed, if you had a model that identified all relevant causal mechanisms, then of course more data is still better. Well, my point -and, as far as I understand it, Vapnik's point, in my link above- is that if a model can generalise well, then it doesn't need a lot of data, even if a lot of data is available. In terms of a model that can identify causal relations, then it should be able to do so without lots of data. My intuition for that is that if the model needs lots of data it will inevitably learn to represent the sampling error in the data generation process (DGP?). A model that can learn good generalisations from few data on the other hand is in a sense immunised against sampling error because it can ignore most of the data. Vapnik makes an analogy about a good teacher. He asks, what is it that a good teacher does that helps his students learn better? He answers it by saying that the good teacher gives his students "good predicates", what I would call background knowledge, from which the students can build good statistical invariants. But the students must also be able to build good statistical invariants _on their own_, otherwise it doesn't matter what good predicates the teacher can give them, the student's can't learn good invariants. I think that's the point of the challenge - Vapnik is asking for learning from few examples (not unreasonably few, I think) as a way to demonstrate that a system is learning good underlying principles- his good predicates and statistical invariants. In the past the machine learning community has sidestepped the issue by finding ways to augment data where it was scarce and then claim progress in certain problems for which there were initially few examples (for example, Bongard problems and Winograd schemas) but Vapnik is not only calling for few data, but also for good predicates, which is I think a better defense than just asking for few-shot learning. In a sense I think if we find that more data is better, then that means the system is still not learning the good bits that make more data unnecessary.