8 ms·

“Less than one”-shot learning

94 points by monsieurpng 6y ago

alquemist 6y ago
The title is misleading. The core technique still uses 60,000 images from MNIST, but 'distills' them into 10 images that contain the information from the original 60,000. The 10 'distilled' images look nothing like digits. Learning a complex model from 10 (later reduced to 2) 'distilled' number arrays is an interesting research idea, but it has little to do with reducing the size of the input dataset. Arguably the heavy lifting part of the learning process moved from training the model to generating the distilled dataset. There is also some unconvincing discussion around synthetic datasets, though it remains fully unclear how these synthetic datasets have anything to do with real world scenarios. > In a previous paper, MIT researchers had introduced a technique to “distill” giant data sets into tiny ones, and as a proof of concept, they had compressed MNIST down to only 10 images.
- vikramkr 6y ago
  So this is more like data storage/some sort of extreme compression than anything? Would it be accurate to interpret this as basically saving the trained neural network into 10 images and reading it back to retrain a model? Or, what really does this do/accomplish?
  - currymj 6y ago
    mostly it seems that it tells you something sort of weird about how neural networks train. it’s not obvious that this should work, and that it can be made to work is interesting.
    - alquemist 6y ago
      Agreed, the idea is interesting and worthy of further exploration. For starters, does it scales beyond MNIST or 'tiny synthetic datasets'?
      - freeone3000 6y ago
        The approach appears similar to model distillation, performed on the input data instead of on the weights. Model distillation scales well in classification but less well in generation.
- gameswithgo 6y ago
  Is this process net faster, or requiring of less memory?
- ampdepolymerase 6y ago
  It may mean we might finally have a method for reliably updating big neural networks instead of having to do continuous on the fly retraining. (Imagine future neural networks on your smart car having "upgrade packs" or country specific data that can be used to fine tune the main network in a matter of minutes) A high-level form of patch-and-diff for networks. There is probably a ML Ops startup opportunity somewhere in this.
  - bitL 6y ago
    "Upgrade packs" might be already possible using transformer adapters, i.e. tiny networks trained on customized data plugged into a large fixed pretrained transformer, providing whatever custom functionality you require.
  - heyitsguay 6y ago
    I don't think this changes anything. Deployed networks typically use only inference from pretrained weights, and those weights are what get transferred for model "updates". You can have all your devices using weight array W0 for a neural net architecture, spend a million compute hours training that net on cutting edge systems to produce a much better weight array W1, then upgrade all the deployed devices by sending them W1 which will be the same size as W0.
    - ampdepolymerase 6y ago
      For aerospace applications where bandwidth is severely limited, a hundred megabyte NN is a lot.
- ufo 6y ago
  At least I'm glad that the press article included the actual images. Oftentimes we have to track the original paper to see that kind of thing.
- mota7 6y ago
  It's still interesting as it reflects that despite 60,000 images, there's a very small amount of data that the network actually learns. The total entropy in 10 images (even carefully engineered ones) is very low in comparison to the full data-set.
- dang 6y ago
  Ok, we've replaced the title with the first part of the subtitle.
- sdenton4 6y ago
  It's interesting for localized learning... You have some local data that you want the model to incorporate, but don't want it to forget the main dataset. So you can create one of these distilled datasets and include it in a batch with the local data when you updated the model.
- jsharf 6y ago
  To me, this seems similar to training a neural network on 10,000 images and then publishing the weights themselves as a distilled "image" (of course, it's not the weights themselves but a "distillation"). I feel like you could get something similar by taking the trained network and then applying deep-dream techniques for each classification label on a blank canvas.
m101010 6y ago
if I understand correctly the key benefit would be that models could be trained on smaller datasets and therefore reduce the time spent computing the models? I am not convinced that this time saving is more than the time spent to engineer the combined and synthesised data.
iandanforth 6y ago
"carefully engineered their soft labels" is the same thing as training the network. Just because you encode information outside of the weights doesn't mean you're not encoding information from training data. It's like saying here's the ideal partitioning scheme, memorize this.
nharada 6y ago
Direct link to paper: https://arxiv.org/pdf/2009.08449.pdf https://arxiv.org/pdf/2009.08449.pdf Interesting paper, although the headline is of course sensational. The crux of the paper is that by using "soft labels" (for example a probability distribution rather than one-hot), it's possible to create a decision boundary that encodes more classes than you have examples. In fact, only two examples can be used to encode any finite number of classes. This is interesting because it means that, in theory, ML models should be able to learn decision spaces that are far more complex than the input data has traditionally been thought to encode. Maybe one day we can create complex, generalizable models using a small amount of data. As written, this paper does not provide much actionable information. The problem is a toy problem, and is far from being useful in "modern" AI techniques (especially things like deep learning or boosted trees). The paper also is not practical in the sense that in real life you don't know what your decision boundary should look like (that's what you learn after all), and there's no obvious way to know which data to collect to get a decision boundary you want. In other words, this paper has said "this representation is mathematically possible" and is hoping that future work can actually make it useful in practice.
- ummonk 6y ago
  Right it's just creating a 2D basis space and placing the various labels inside that space. (And the previous paper was about compositing elements of multiple dataset images into a single training example - likely overfitting).
etaioinshrdlu 6y ago
I like to think of this as adversarial training data. Adversarial inputs in general trick a NN to producing a specific output -- Adversarial training data tricks the NN into learning specific weights. Note that the distilled data is not even from the same "domain" of input data any more. They're basically adversarial inputs.
echelon 6y ago
I've started to view Technology Review as a PR puff piece for MIT. They often overstate claims or leave out critical details. As an example, the media lab is still citing innovation with deep fakes, claiming entirely novel results people are shocked to see. They hype their own researchers even though there are kids on YouTube who that have been making similar content up to a year prior to Technology Review's publication. I suspect they do the same with fields I'm less familiar with.
- bonoboTP 6y ago
  In this case it's clear since this is a publication of MIT and they hype up their own research of course. But even when you're note reading a university publication/PR piece, you still see this effect. One big player like a famous lab at Stanford or MIT publishes an incremental paper that is a followup on a well known existing research direction, where several groups are working in parallel on very similar things, and then it's presented as if it was some breakthrough and the whole subfield was invented by them right now. It's very, very hard for outsiders to recognize this and to really understand what the actual incremental step in a particular paper is. Necessarily, when explaining to laypeople, you can only scratch the surface and present the rought idea of a whole big research field, and it gets really murky what is part of the established, pre-existing research field and what is the novel contribution. I'm sure I fall victim to this when reading outside my expertise as well, i.e. when reading about genetics stuff or quantum computing.
  - dmurray 6y ago
    > I'm sure I fall victim to this when reading outside my expertise as well, i.e. when reading about genetics stuff or quantum computing. I try to avoid this by focusing on the content, not who the researchers are. If they can do some cool new thing with mosquito genetics, now I know that. I don't need to know whether the specific paper being hyped is novel or whether 95% of the content was proved by someone else: that's a task for the Nobel committee and I wouldn't recognise the researchers' names again anyway.
codelord 6y ago
The title is click-bait. This has been known for several years[1], the technique has little practical value, and the assertion that you can learn from no data is completely false and misleading. The training data was compressed to a few examples. To the journalist: it's OK not to maximize for click-bait when you write an article. [1]: https://www.ttic.edu/dl/dark14.pdf https://www.ttic.edu/dl/dark14.pdf
- bonoboTP 6y ago
  This is not a magazine article written by a journalist, but a press release written by MIT marketing people to hype up research from their university.
- jerry80 6y ago
  Why not just make one image where the pixel values are the trained network weights? Then you can say you can distill any training set into just one image.
  - vonwoodson 6y ago
    You just nailed it. The whole thing feels like reinventing the wheel of ML.
supernova87a 6y ago
Well, I'm sure nothing will be societally objectionable about this!
breck 6y ago
> ...very different from human learning. A child often needs to see just a few examples of an object, or even only one, before being able to recognize it for life. I see this a lot. It's completely wrong. I'm not trying to pick on the author here, I think 95%+ of people share this misunderstanding of deep learning. If you see "only one" horse, say for even a second, you really are seeing a huge number of horses, from various angles, with various shades of lighting. The motions of the horse; the motions of your head (even if slight); the undulations of the light; are generating a much larger number of basically augmented training data. If you look at a horse for a minute it could be the equivalent of training on 1 million images of a horse. I'm not sure the exact OOM, but it's certainly orders of magnitude more than "one" horse. (Relatedly: Some people say there is an experiment you can conduct at home to see the actual images your brain is training on).
- ummonk 6y ago
  Not really. A child can see a single static picture of an unfamiliar animal and learn to recognize the animal from that picture alone.
  - breck 6y ago
    No, they can’t. Flash an image of an animal for a microsecond in front of a child. They won’t “see” anything. One needs to “see” many, many versions of something before you “see” anything at all, nevermind learn something.
    - ummonk 6y ago
      That isn’t seeing many versions of the animal. It is merely processing the one training sample over several seconds, because it takes a few hundred milliseconds for the human brain and visual system to process an image, and several more seconds for the brain to analyze and memorize key features of the sample. Standard machine learning systems don’t work like this. No matter how much time they’re allowed to process a single sample image, they are unable to learn to classify it. There are of course machine learning systems that work more like humans (facial recognition is a notable example), but they’re the exception not the norm.
ummonk 6y ago
Am I correct that they don't use a test-train split for generating these distilled images? Until you test on new images outside of what is inputted to the distiller, it seems to be a way to just overfit specific images, probably by combining unique elements of each into a single composite image. There are plenty of classical signal processing ways to do this (including just building a composite patchwork quilt).
smt1 6y ago
Remember that any meta-virtualization (aka software-defined anything, but is really figuring how how to use less energy to control hardware with better regularity). Remember all 1-bit ReLU (linear non-linear rectifier) are just are one way to think about the same thing as say a dirac function using the language of mathmatical physics. Better to pretend that something like all ad networks or dynamic pieces of web browsers could be influenced to do stuff using other pieces of hardware influenced software that apparently already coded to do that. Solve the "well known" abc or apple google microsoft (lua) = use the wisdom of the crowds to look at symbolic differentiation. Or look at it in 2 elementary bit boolean l-r modular problems so solve the calculus of arbitrary excitations/fluxtions in inductive-coinductive in the calculus of categories of ends and coends using modern elementary mathmatical physics just by using noetherian algebras (understsnding why things don't commute rather than associative properties create transitive closures). Understand why noether's theorem and abelian categories are more powerful aways of computing and energy saving by energy scavenging power. Sorry for the sameless plugin and hit me up at sashmit@gmail.com if you want to know more or collaborate. All this can be understood more clearly by treating neural networks as more elementary computer vision applied to more mature algos with more ethical ways of using it.

94 points by monsieurpng 6y ago

The title is misleading. The core technique still uses 60,000 images from MNIST, but 'distills' them into 10 images that contain the information from the original 60,000. The 10 'distilled' images look nothing like digits. Learning a complex model from 10 (later reduced to 2) 'distilled' number arrays is an interesting research idea, but it has little to do with reducing the size of the input dataset. Arguably the heavy lifting part of the learning process moved from training the model to generating the distilled dataset. There is also some unconvincing discussion around synthetic datasets, though it remains fully unclear how these synthetic datasets have anything to do with real world scenarios. > In a previous paper, MIT researchers had introduced a technique to “distill” giant data sets into tiny ones, and as a proof of concept, they had compressed MNIST down to only 10 images.

vikramkr 6y ago
So this is more like data storage/some sort of extreme compression than anything? Would it be accurate to interpret this as basically saving the trained neural network into 10 images and reading it back to retrain a model? Or, what really does this do/accomplish?
- currymj 6y ago
  mostly it seems that it tells you something sort of weird about how neural networks train. it’s not obvious that this should work, and that it can be made to work is interesting.
  - alquemist 6y ago
    Agreed, the idea is interesting and worthy of further exploration. For starters, does it scales beyond MNIST or 'tiny synthetic datasets'?
    - freeone3000 6y ago
      The approach appears similar to model distillation, performed on the input data instead of on the weights. Model distillation scales well in classification but less well in generation.
gameswithgo 6y ago
Is this process net faster, or requiring of less memory?
ampdepolymerase 6y ago
It may mean we might finally have a method for reliably updating big neural networks instead of having to do continuous on the fly retraining. (Imagine future neural networks on your smart car having "upgrade packs" or country specific data that can be used to fine tune the main network in a matter of minutes) A high-level form of patch-and-diff for networks. There is probably a ML Ops startup opportunity somewhere in this.
- bitL 6y ago
  "Upgrade packs" might be already possible using transformer adapters, i.e. tiny networks trained on customized data plugged into a large fixed pretrained transformer, providing whatever custom functionality you require.
- heyitsguay 6y ago
  I don't think this changes anything. Deployed networks typically use only inference from pretrained weights, and those weights are what get transferred for model "updates". You can have all your devices using weight array W0 for a neural net architecture, spend a million compute hours training that net on cutting edge systems to produce a much better weight array W1, then upgrade all the deployed devices by sending them W1 which will be the same size as W0.
  - ampdepolymerase 6y ago
    For aerospace applications where bandwidth is severely limited, a hundred megabyte NN is a lot.
ufo 6y ago
At least I'm glad that the press article included the actual images. Oftentimes we have to track the original paper to see that kind of thing.
mota7 6y ago
It's still interesting as it reflects that despite 60,000 images, there's a very small amount of data that the network actually learns. The total entropy in 10 images (even carefully engineered ones) is very low in comparison to the full data-set.
dang 6y ago
Ok, we've replaced the title with the first part of the subtitle.
sdenton4 6y ago
It's interesting for localized learning... You have some local data that you want the model to incorporate, but don't want it to forget the main dataset. So you can create one of these distilled datasets and include it in a batch with the local data when you updated the model.
jsharf 6y ago
To me, this seems similar to training a neural network on 10,000 images and then publishing the weights themselves as a distilled "image" (of course, it's not the weights themselves but a "distillation"). I feel like you could get something similar by taking the trained network and then applying deep-dream techniques for each classification label on a blank canvas.

m101010 6y ago

if I understand correctly the key benefit would be that models could be trained on smaller datasets and therefore reduce the time spent computing the models? I am not convinced that this time saving is more than the time spent to engineer the combined and synthesised data.

iandanforth 6y ago

"carefully engineered their soft labels" is the same thing as training the network. Just because you encode information outside of the weights doesn't mean you're not encoding information from training data. It's like saying here's the ideal partitioning scheme, memorize this.

nharada 6y ago

Direct link to paper: https://arxiv.org/pdf/2009.08449.pdf https://arxiv.org/pdf/2009.08449.pdf Interesting paper, although the headline is of course sensational. The crux of the paper is that by using "soft labels" (for example a probability distribution rather than one-hot), it's possible to create a decision boundary that encodes more classes than you have examples. In fact, only two examples can be used to encode any finite number of classes. This is interesting because it means that, in theory, ML models should be able to learn decision spaces that are far more complex than the input data has traditionally been thought to encode. Maybe one day we can create complex, generalizable models using a small amount of data. As written, this paper does not provide much actionable information. The problem is a toy problem, and is far from being useful in "modern" AI techniques (especially things like deep learning or boosted trees). The paper also is not practical in the sense that in real life you don't know what your decision boundary should look like (that's what you learn after all), and there's no obvious way to know which data to collect to get a decision boundary you want. In other words, this paper has said "this representation is mathematically possible" and is hoping that future work can actually make it useful in practice.

ummonk 6y ago
Right it's just creating a 2D basis space and placing the various labels inside that space. (And the previous paper was about compositing elements of multiple dataset images into a single training example - likely overfitting).

etaioinshrdlu 6y ago

I like to think of this as adversarial training data. Adversarial inputs in general trick a NN to producing a specific output -- Adversarial training data tricks the NN into learning specific weights. Note that the distilled data is not even from the same "domain" of input data any more. They're basically adversarial inputs.

echelon 6y ago

I've started to view Technology Review as a PR puff piece for MIT. They often overstate claims or leave out critical details. As an example, the media lab is still citing innovation with deep fakes, claiming entirely novel results people are shocked to see. They hype their own researchers even though there are kids on YouTube who that have been making similar content up to a year prior to Technology Review's publication. I suspect they do the same with fields I'm less familiar with.

bonoboTP 6y ago
In this case it's clear since this is a publication of MIT and they hype up their own research of course. But even when you're note reading a university publication/PR piece, you still see this effect. One big player like a famous lab at Stanford or MIT publishes an incremental paper that is a followup on a well known existing research direction, where several groups are working in parallel on very similar things, and then it's presented as if it was some breakthrough and the whole subfield was invented by them right now. It's very, very hard for outsiders to recognize this and to really understand what the actual incremental step in a particular paper is. Necessarily, when explaining to laypeople, you can only scratch the surface and present the rought idea of a whole big research field, and it gets really murky what is part of the established, pre-existing research field and what is the novel contribution. I'm sure I fall victim to this when reading outside my expertise as well, i.e. when reading about genetics stuff or quantum computing.
- dmurray 6y ago
  > I'm sure I fall victim to this when reading outside my expertise as well, i.e. when reading about genetics stuff or quantum computing. I try to avoid this by focusing on the content, not who the researchers are. If they can do some cool new thing with mosquito genetics, now I know that. I don't need to know whether the specific paper being hyped is novel or whether 95% of the content was proved by someone else: that's a task for the Nobel committee and I wouldn't recognise the researchers' names again anyway.

codelord 6y ago

The title is click-bait. This has been known for several years[1], the technique has little practical value, and the assertion that you can learn from no data is completely false and misleading. The training data was compressed to a few examples. To the journalist: it's OK not to maximize for click-bait when you write an article. [1]: https://www.ttic.edu/dl/dark14.pdf https://www.ttic.edu/dl/dark14.pdf

bonoboTP 6y ago
This is not a magazine article written by a journalist, but a press release written by MIT marketing people to hype up research from their university.
jerry80 6y ago
Why not just make one image where the pixel values are the trained network weights? Then you can say you can distill any training set into just one image.
- vonwoodson 6y ago
  You just nailed it. The whole thing feels like reinventing the wheel of ML.

supernova87a 6y ago

Well, I'm sure nothing will be societally objectionable about this!

breck 6y ago

> ...very different from human learning. A child often needs to see just a few examples of an object, or even only one, before being able to recognize it for life. I see this a lot. It's completely wrong. I'm not trying to pick on the author here, I think 95%+ of people share this misunderstanding of deep learning. If you see "only one" horse, say for even a second, you really are seeing a huge number of horses, from various angles, with various shades of lighting. The motions of the horse; the motions of your head (even if slight); the undulations of the light; are generating a much larger number of basically augmented training data. If you look at a horse for a minute it could be the equivalent of training on 1 million images of a horse. I'm not sure the exact OOM, but it's certainly orders of magnitude more than "one" horse. (Relatedly: Some people say there is an experiment you can conduct at home to see the actual images your brain is training on).

ummonk 6y ago
Not really. A child can see a single static picture of an unfamiliar animal and learn to recognize the animal from that picture alone.
- breck 6y ago
  No, they can’t. Flash an image of an animal for a microsecond in front of a child. They won’t “see” anything. One needs to “see” many, many versions of something before you “see” anything at all, nevermind learn something.
  - ummonk 6y ago
    That isn’t seeing many versions of the animal. It is merely processing the one training sample over several seconds, because it takes a few hundred milliseconds for the human brain and visual system to process an image, and several more seconds for the brain to analyze and memorize key features of the sample. Standard machine learning systems don’t work like this. No matter how much time they’re allowed to process a single sample image, they are unable to learn to classify it. There are of course machine learning systems that work more like humans (facial recognition is a notable example), but they’re the exception not the norm.

ummonk 6y ago

Am I correct that they don't use a test-train split for generating these distilled images? Until you test on new images outside of what is inputted to the distiller, it seems to be a way to just overfit specific images, probably by combining unique elements of each into a single composite image. There are plenty of classical signal processing ways to do this (including just building a composite patchwork quilt).

smt1 6y ago

Remember that any meta-virtualization (aka software-defined anything, but is really figuring how how to use less energy to control hardware with better regularity). Remember all 1-bit ReLU (linear non-linear rectifier) are just are one way to think about the same thing as say a dirac function using the language of mathmatical physics. Better to pretend that something like all ad networks or dynamic pieces of web browsers could be influenced to do stuff using other pieces of hardware influenced software that apparently already coded to do that. Solve the "well known" abc or apple google microsoft (lua) = use the wisdom of the crowds to look at symbolic differentiation. Or look at it in 2 elementary bit boolean l-r modular problems so solve the calculus of arbitrary excitations/fluxtions in inductive-coinductive in the calculus of categories of ends and coends using modern elementary mathmatical physics just by using noetherian algebras (understsnding why things don't commute rather than associative properties create transitive closures). Understand why noether's theorem and abelian categories are more powerful aways of computing and energy saving by energy scavenging power. Sorry for the sameless plugin and hit me up at sashmit@gmail.com if you want to know more or collaborate. All this can be understood more clearly by treating neural networks as more elementary computer vision applied to more mature algos with more ethical ways of using it.