3 ms·

The title is misleading. The core technique still uses 60,000 images from MNIST, but 'distills' them into 10 images that contain the information from the origin

The title is misleading. The core technique still uses 60,000 images from MNIST, but 'distills' them into 10 images that contain the information from the original 60,000. The 10 'distilled' images look nothing like digits. Learning a complex model from 10 (later reduced to 2) 'distilled' number arrays is an interesting research idea, but it has little to do with reducing the size of the input dataset. Arguably the heavy lifting part of the learning process moved from training the model to generating the distilled dataset. There is also some unconvincing discussion around synthetic datasets, though it remains fully unclear how these synthetic datasets have anything to do with real world scenarios. > In a previous paper, MIT researchers had introduced a technique to “distill” giant data sets into tiny ones, and as a proof of concept, they had compressed MNIST down to only 10 images.

vikramkr 6y ago
So this is more like data storage/some sort of extreme compression than anything? Would it be accurate to interpret this as basically saving the trained neural network into 10 images and reading it back to retrain a model? Or, what really does this do/accomplish?
- currymj 6y ago
  mostly it seems that it tells you something sort of weird about how neural networks train. it’s not obvious that this should work, and that it can be made to work is interesting.
  - alquemist 6y ago
    Agreed, the idea is interesting and worthy of further exploration. For starters, does it scales beyond MNIST or 'tiny synthetic datasets'?
    - freeone3000 6y ago
      The approach appears similar to model distillation, performed on the input data instead of on the weights. Model distillation scales well in classification but less well in generation.
gameswithgo 6y ago
Is this process net faster, or requiring of less memory?
ampdepolymerase 6y ago
It may mean we might finally have a method for reliably updating big neural networks instead of having to do continuous on the fly retraining. (Imagine future neural networks on your smart car having "upgrade packs" or country specific data that can be used to fine tune the main network in a matter of minutes) A high-level form of patch-and-diff for networks. There is probably a ML Ops startup opportunity somewhere in this.
- bitL 6y ago
  "Upgrade packs" might be already possible using transformer adapters, i.e. tiny networks trained on customized data plugged into a large fixed pretrained transformer, providing whatever custom functionality you require.
- heyitsguay 6y ago
  I don't think this changes anything. Deployed networks typically use only inference from pretrained weights, and those weights are what get transferred for model "updates". You can have all your devices using weight array W0 for a neural net architecture, spend a million compute hours training that net on cutting edge systems to produce a much better weight array W1, then upgrade all the deployed devices by sending them W1 which will be the same size as W0.
  - ampdepolymerase 6y ago
    For aerospace applications where bandwidth is severely limited, a hundred megabyte NN is a lot.
ufo 6y ago
At least I'm glad that the press article included the actual images. Oftentimes we have to track the original paper to see that kind of thing.
mota7 6y ago
It's still interesting as it reflects that despite 60,000 images, there's a very small amount of data that the network actually learns. The total entropy in 10 images (even carefully engineered ones) is very low in comparison to the full data-set.
dang 6y ago
Ok, we've replaced the title with the first part of the subtitle.
sdenton4 6y ago
It's interesting for localized learning... You have some local data that you want the model to incorporate, but don't want it to forget the main dataset. So you can create one of these distilled datasets and include it in a batch with the local data when you updated the model.
jsharf 6y ago
To me, this seems similar to training a neural network on 10,000 images and then publishing the weights themselves as a distilled "image" (of course, it's not the weights themselves but a "distillation"). I feel like you could get something similar by taking the trained network and then applying deep-dream techniques for each classification label on a blank canvas.

vikramkr 6y ago

So this is more like data storage/some sort of extreme compression than anything? Would it be accurate to interpret this as basically saving the trained neural network into 10 images and reading it back to retrain a model? Or, what really does this do/accomplish?

currymj 6y ago
mostly it seems that it tells you something sort of weird about how neural networks train. it’s not obvious that this should work, and that it can be made to work is interesting.
- alquemist 6y ago
  Agreed, the idea is interesting and worthy of further exploration. For starters, does it scales beyond MNIST or 'tiny synthetic datasets'?
  - freeone3000 6y ago
    The approach appears similar to model distillation, performed on the input data instead of on the weights. Model distillation scales well in classification but less well in generation.

gameswithgo 6y ago

Is this process net faster, or requiring of less memory?

ampdepolymerase 6y ago

It may mean we might finally have a method for reliably updating big neural networks instead of having to do continuous on the fly retraining. (Imagine future neural networks on your smart car having "upgrade packs" or country specific data that can be used to fine tune the main network in a matter of minutes) A high-level form of patch-and-diff for networks. There is probably a ML Ops startup opportunity somewhere in this.

bitL 6y ago
"Upgrade packs" might be already possible using transformer adapters, i.e. tiny networks trained on customized data plugged into a large fixed pretrained transformer, providing whatever custom functionality you require.
heyitsguay 6y ago
I don't think this changes anything. Deployed networks typically use only inference from pretrained weights, and those weights are what get transferred for model "updates". You can have all your devices using weight array W0 for a neural net architecture, spend a million compute hours training that net on cutting edge systems to produce a much better weight array W1, then upgrade all the deployed devices by sending them W1 which will be the same size as W0.
- ampdepolymerase 6y ago
  For aerospace applications where bandwidth is severely limited, a hundred megabyte NN is a lot.

ufo 6y ago

At least I'm glad that the press article included the actual images. Oftentimes we have to track the original paper to see that kind of thing.

mota7 6y ago

It's still interesting as it reflects that despite 60,000 images, there's a very small amount of data that the network actually learns. The total entropy in 10 images (even carefully engineered ones) is very low in comparison to the full data-set.

dang 6y ago

Ok, we've replaced the title with the first part of the subtitle.

sdenton4 6y ago

It's interesting for localized learning... You have some local data that you want the model to incorporate, but don't want it to forget the main dataset. So you can create one of these distilled datasets and include it in a batch with the local data when you updated the model.

jsharf 6y ago

To me, this seems similar to training a neural network on 10,000 images and then publishing the weights themselves as a distilled "image" (of course, it's not the weights themselves but a "distillation"). I feel like you could get something similar by taking the trained network and then applying deep-dream techniques for each classification label on a blank canvas.