4 ms·

deep neural network trained on vast numbers of generated examples could come to learn what a suitable prior should be by learning the patterns used to generate

by p1esk 3y ago

deep neural network trained on vast numbers of generated examples could come to learn what a suitable prior should be by learning the patterns used to generate them. This sounds promising. Do you see any potential issues with this approach?

uoaei 3y ago
This is how NN theory describes the process of training, i.e., that's the mechanism by which all neural networks come to represent the problem they are being applied to solve. GP is using a fancy way to say "train NNs on massive datasets" which is par for the course.
versteegen 3y ago
DNNs are very good at learning features so are a natural fit if you have lots of training data. The main drawback is then you've lost the ability to work with small data. The other drawback is the computational cost of the maxent model on top of it. So you get both the best and the worst of both halves. So: input --DNN--> feature vector --minimax_entropy_optimisation--> maxent model --sampling/inference--> output But minimax entropy learning is just learning a model that matches statistics of a set of features (filters). If you just want to draw a single sample from that model or the maximum likelihood point (e.g. you just want the answer to an IQ question) rather than doing complicated inference (in the DNN era, people have forgotten that "inference" is far more general than classification) you don't actually need the model at all, you can skip learning it, and go straight to computing a result with the desired statistics. This is around 100-1000x faster. input --DNN--> feature vector --statistics_matching--> output This is what I mostly did in my PhD work, and there's a famous paper doing this: A Neural Algorithm of Artistic Style https://arxiv.org/abs/1505.07376 https://arxiv.org/abs/1505.07376 This has a sister paper which is even purer statistics-of-features-matching: Texture Synthesis Using Convolutional Neural Networks https://arxiv.org/abs/1505.07376 https://arxiv.org/abs/1505.07376 However soon people worked out you can just train a DNN to do the statistics matching directly. Initially, with a fixed target style, and later taking the style as a second input. If the results are acceptable, this is vastly faster at inference time. input --DNN--> output Using them in this way would likely mean just replacing the final layer. The original papers on 'style transfer' did this, a DNN plus minimax entropy learning. has been done in texture and image models.
- versteegen 3y ago
  (Whoops, ignore that last line; just fragments of unfinished sentences)

by p1esk 3y ago

uoaei 3y ago

This is how NN theory describes the process of training, i.e., that's the mechanism by which all neural networks come to represent the problem they are being applied to solve. GP is using a fancy way to say "train NNs on massive datasets" which is par for the course.

versteegen 3y ago

DNNs are very good at learning features so are a natural fit if you have lots of training data. The main drawback is then you've lost the ability to work with small data. The other drawback is the computational cost of the maxent model on top of it. So you get both the best and the worst of both halves. So: input --DNN--> feature vector --minimax_entropy_optimisation--> maxent model --sampling/inference--> output But minimax entropy learning is just learning a model that matches statistics of a set of features (filters). If you just want to draw a single sample from that model or the maximum likelihood point (e.g. you just want the answer to an IQ question) rather than doing complicated inference (in the DNN era, people have forgotten that "inference" is far more general than classification) you don't actually need the model at all, you can skip learning it, and go straight to computing a result with the desired statistics. This is around 100-1000x faster. input --DNN--> feature vector --statistics_matching--> output This is what I mostly did in my PhD work, and there's a famous paper doing this: A Neural Algorithm of Artistic Style https://arxiv.org/abs/1505.07376 https://arxiv.org/abs/1505.07376 This has a sister paper which is even purer statistics-of-features-matching: Texture Synthesis Using Convolutional Neural Networks https://arxiv.org/abs/1505.07376 https://arxiv.org/abs/1505.07376 However soon people worked out you can just train a DNN to do the statistics matching directly. Initially, with a fixed target style, and later taking the style as a second input. If the results are acceptable, this is vastly faster at inference time. input --DNN--> output Using them in this way would likely mean just replacing the final layer. The original papers on 'style transfer' did this, a DNN plus minimax entropy learning. has been done in texture and image models.

versteegen 3y ago
(Whoops, ignore that last line; just fragments of unfinished sentences)