3 ms·

If I had to guess I would say that: - It involves generative models (needed to make inferences from very few examples) - It is still a connectionist approach

If I had to guess I would say that: - It involves generative models (needed to make inferences from very few examples) - It is still a connectionist approach (typical probabilistic programming is great if you have a lot of insight into the model, but not if you're trying to solve a general case... unless you're doing program induction but you need to represent programs...) - It doesn't involve MCMC sampling for inference, because that's too slow or even intractable. Some type of variational program induction where programs are represented as differentiable neural networks would be in that corner. Or it could be something totally different, but speculating is fun.

BenoitEssiambre 11y ago
I dunno, I still think it might be possible to make tractable program induction through MCMC. I've been trying (but failing) to do so by clustering similar generative rules of different complexity together then getting the algorithm to search the grammar space in a way that it recursively tries simpler models first then goes on to try generating (and learning) more complex rules that are known to have similar output to the best simpler ones. My intuition is that you have to cluster your generative rules into an taxonomy that your algorithm can navigate from the top. It's exponentially inefficient to try to recognize "dog" until you have recognized that the simpler "animal" is a good approximation. I also think that in the ultimate solution, the output leaf statements of the grammar will be parametrized like a normal programming language, so that for example, the algorithm can generate a color, a radius and then generate 100 circles referencing this single color and radius to represent repeated patterns without having to learn their size and color individually when they are clearly homogeneous or invariant across a bunch. The bayesian, occam's razor, solution to a bunch of similar things that you don't have a category for yet is shared parameters. These parameters enable learning with very few examples. The algorithm doesn't have to learn a full new category to make good predictions, it can simply notice a functional parametric pattern in a part of a scene and extrapolate immediately. I haven't cracked it however. It's grueling to debug probabilistic algorithms compared to clear cut binary ones!
- murbard2 11y ago
  Having some type of hierarchical structure is probably the right inductive bias to have (it may be one of the reasons why deep learning is so successful), but you also need a search direction. It's hopeless to randomly hop in a high dimensional space until you find matches.
  - BenoitEssiambre 11y ago
    The hierarchical structure means you don't hop randomly. You recognize simple and vague approximate models first then bias your search towards the more complex rules that are under you best top level approximations in the hierarchy. It's like a probabilistic binary search in the model taxonomy. One thing that makes me hopeful that the learning process is possible is that the non-terminal grammar rules are generic and can always become anything. You don't tend to get stuck in local maxima even if you only search a small part of the program space when any node can always be swapped for a node of any another category. My problem right now is that my MCMH doesn't mix well. Even though I tried to shorten the jumps between models of different complexity for a particular thing, once the vague model traces are well fitted to the training examples, there is too deep a score chasm to jump to the next level of complexity. My simulations seem to get stuck in simpler modelisations of things. I want simpler models to recursively open the door to learning and recognizing slightly more complex models just under them in the hierarchy but this doesn't seem to be happening. The hierarchy is not really even forming properly. I might be missing something stupid. I don't consider myself an expert in these things. I just dabble for fun. It's one of the reason I'm discussing it here. Maybe someone more knowledgeable will make me have an aha! moment. There is still a bunch of things I could try but like I said, it's long and grueling debugging work.

BenoitEssiambre 11y ago

I dunno, I still think it might be possible to make tractable program induction through MCMC. I've been trying (but failing) to do so by clustering similar generative rules of different complexity together then getting the algorithm to search the grammar space in a way that it recursively tries simpler models first then goes on to try generating (and learning) more complex rules that are known to have similar output to the best simpler ones. My intuition is that you have to cluster your generative rules into an taxonomy that your algorithm can navigate from the top. It's exponentially inefficient to try to recognize "dog" until you have recognized that the simpler "animal" is a good approximation. I also think that in the ultimate solution, the output leaf statements of the grammar will be parametrized like a normal programming language, so that for example, the algorithm can generate a color, a radius and then generate 100 circles referencing this single color and radius to represent repeated patterns without having to learn their size and color individually when they are clearly homogeneous or invariant across a bunch. The bayesian, occam's razor, solution to a bunch of similar things that you don't have a category for yet is shared parameters. These parameters enable learning with very few examples. The algorithm doesn't have to learn a full new category to make good predictions, it can simply notice a functional parametric pattern in a part of a scene and extrapolate immediately. I haven't cracked it however. It's grueling to debug probabilistic algorithms compared to clear cut binary ones!

murbard2 11y ago
Having some type of hierarchical structure is probably the right inductive bias to have (it may be one of the reasons why deep learning is so successful), but you also need a search direction. It's hopeless to randomly hop in a high dimensional space until you find matches.
- BenoitEssiambre 11y ago
  The hierarchical structure means you don't hop randomly. You recognize simple and vague approximate models first then bias your search towards the more complex rules that are under you best top level approximations in the hierarchy. It's like a probabilistic binary search in the model taxonomy. One thing that makes me hopeful that the learning process is possible is that the non-terminal grammar rules are generic and can always become anything. You don't tend to get stuck in local maxima even if you only search a small part of the program space when any node can always be swapped for a node of any another category. My problem right now is that my MCMH doesn't mix well. Even though I tried to shorten the jumps between models of different complexity for a particular thing, once the vague model traces are well fitted to the training examples, there is too deep a score chasm to jump to the next level of complexity. My simulations seem to get stuck in simpler modelisations of things. I want simpler models to recursively open the door to learning and recognizing slightly more complex models just under them in the hierarchy but this doesn't seem to be happening. The hierarchy is not really even forming properly. I might be missing something stupid. I don't consider myself an expert in these things. I just dabble for fun. It's one of the reason I'm discussing it here. Maybe someone more knowledgeable will make me have an aha! moment. There is still a bunch of things I could try but like I said, it's long and grueling debugging work.