3 ms·

I've had this hypothesis for years that a good AI model would be something in between a neural net and probabilistic grammar. In my mind it would involve some

I've had this hypothesis for years that a good AI model would be something in between a neural net and probabilistic grammar. In my mind it would involve some kind of generative grammar that would generate nodes (select from a pool), then these nodes could be trained. I'm thinking about something like a grammar for Bayesian networks or, more broadly, a generative program induction scheme where you'd specify a programming language grammar, generate programs that fit the data and tune the parameters. I attempted and failed to implement some of these ideas (described here: https://www.quora.com/What-deep-learning-ideas-have-you-tried-that-didnt-work/answer/Benoit-Essiambre?ch=10&share=e04634aa&srid=lqC https://www.quora.com/What-deep-learning-ideas-have-you-trie...). The search space for generative programs is just so huge and irregular and I don't know how to keep things differentiable. In the paper, they mention gradient-based optimization so maybe to figured out part of it.

advael 5y ago
While a code-data distinction is often illusory in practice, figuring out where this separation can be useful runs throughout the information theory underlying learning systems and programming languages. My question about this proposed model is how it handles the tradeoff between generality and efficiency that most learning systems are up against better than "mere" function approximation (e.g. ANNs). Concretely, treating parts of the solution space of some problem as a grammar specification seems like a way of simplifying the representation domain, so that the differentiable bits of a given model have fewer ultimately meaningless degrees of freedom to search over, but at some point you have to choose a grammar, and how would this work in the proposed system?

3 ms·

I've had this hypothesis for years that a good AI model would be something in between a neural net and probabilistic grammar. In my mind it would involve some

by BenoitEssiambre 5y ago

advael 5y ago
While a code-data distinction is often illusory in practice, figuring out where this separation can be useful runs throughout the information theory underlying learning systems and programming languages. My question about this proposed model is how it handles the tradeoff between generality and efficiency that most learning systems are up against better than "mere" function approximation (e.g. ANNs). Concretely, treating parts of the solution space of some problem as a grammar specification seems like a way of simplifying the representation domain, so that the differentiable bits of a given model have fewer ultimately meaningless degrees of freedom to search over, but at some point you have to choose a grammar, and how would this work in the proposed system?