43 ms·

Having some type of hierarchical structure is probably the right inductive bias to have (it may be one of the reasons why deep learning is so successful), but y

Having some type of hierarchical structure is probably the right inductive bias to have (it may be one of the reasons why deep learning is so successful), but you also need a search direction. It's hopeless to randomly hop in a high dimensional space until you find matches.

BenoitEssiambre 11y ago
The hierarchical structure means you don't hop randomly. You recognize simple and vague approximate models first then bias your search towards the more complex rules that are under you best top level approximations in the hierarchy. It's like a probabilistic binary search in the model taxonomy. One thing that makes me hopeful that the learning process is possible is that the non-terminal grammar rules are generic and can always become anything. You don't tend to get stuck in local maxima even if you only search a small part of the program space when any node can always be swapped for a node of any another category. My problem right now is that my MCMH doesn't mix well. Even though I tried to shorten the jumps between models of different complexity for a particular thing, once the vague model traces are well fitted to the training examples, there is too deep a score chasm to jump to the next level of complexity. My simulations seem to get stuck in simpler modelisations of things. I want simpler models to recursively open the door to learning and recognizing slightly more complex models just under them in the hierarchy but this doesn't seem to be happening. The hierarchy is not really even forming properly. I might be missing something stupid. I don't consider myself an expert in these things. I just dabble for fun. It's one of the reason I'm discussing it here. Maybe someone more knowledgeable will make me have an aha! moment. There is still a bunch of things I could try but like I said, it's long and grueling debugging work.

BenoitEssiambre 11y ago

The hierarchical structure means you don't hop randomly. You recognize simple and vague approximate models first then bias your search towards the more complex rules that are under you best top level approximations in the hierarchy. It's like a probabilistic binary search in the model taxonomy. One thing that makes me hopeful that the learning process is possible is that the non-terminal grammar rules are generic and can always become anything. You don't tend to get stuck in local maxima even if you only search a small part of the program space when any node can always be swapped for a node of any another category. My problem right now is that my MCMH doesn't mix well. Even though I tried to shorten the jumps between models of different complexity for a particular thing, once the vague model traces are well fitted to the training examples, there is too deep a score chasm to jump to the next level of complexity. My simulations seem to get stuck in simpler modelisations of things. I want simpler models to recursively open the door to learning and recognizing slightly more complex models just under them in the hierarchy but this doesn't seem to be happening. The hierarchy is not really even forming properly. I might be missing something stupid. I don't consider myself an expert in these things. I just dabble for fun. It's one of the reason I'm discussing it here. Maybe someone more knowledgeable will make me have an aha! moment. There is still a bunch of things I could try but like I said, it's long and grueling debugging work.