4 ms·

Better algorithms were invented because of bigger machines. Once computers got fast enough, researchers could experiment around with different algorithms on rea

Better algorithms were invented because of bigger machines. Once computers got fast enough, researchers could experiment around with different algorithms on realistic sized models and datasets. Without waiting 2 years for the experiment to finish training. Probabilistic programming isn't going to help general AI much. Things like dropout seem to work well enough, and for the most part AI is severely underfitting rather than overfitting. Our models are far to simple and small to really learn language and do complicated reasoning. Making them bayesian doesn't fix that.

eli_gottlieb 9y ago
>Probabilistic programming isn't going to help general AI much. Excuse me while I laugh.[1,2,3,4] >Things like dropout seem to work well enough, and for the most part AI is severely underfitting rather than overfitting. For the most part, neural networks can't reason at all. They just induce deterministic functions over high-dimensional Euclidean spaces. >Our models are far to simple and small to really learn language and do complicated reasoning. They're also not compositional (new concepts as functions of old concepts), productive (able to draw an unbounded number of inferences from each representation), or unbounded in size of representation (unboundedly many concepts). Neural networks don't even represent causal structure, let alone model how an intervention will affect outcomes! It is, however, really nice to hear an AI booster admit just how incredibly limited connectionist models actually are. >Making them bayesian doesn't fix that. No, changing to a causal, compositional representation that allows for productive and nonparametric (unboundedly large) learning does that. The Bayesian part just makes it extra nice by letting us "put information in" anywhere in the model (at any variable) by conditioning. [1] -- http://forestdb.org/models/learning-physics.html http://forestdb.org/models/learning-physics.html [2] -- http://forestdb.org/models/word-learning.html http://forestdb.org/models/word-learning.html [3] -- http://forestdb.org/models/arithmetic.html http://forestdb.org/models/arithmetic.html [4] -- http://forestdb.org/models/politeness.html http://forestdb.org/models/politeness.html

eli_gottlieb 9y ago

>Probabilistic programming isn't going to help general AI much. Excuse me while I laugh.[1,2,3,4] >Things like dropout seem to work well enough, and for the most part AI is severely underfitting rather than overfitting. For the most part, neural networks can't reason at all. They just induce deterministic functions over high-dimensional Euclidean spaces. >Our models are far to simple and small to really learn language and do complicated reasoning. They're also not compositional (new concepts as functions of old concepts), productive (able to draw an unbounded number of inferences from each representation), or unbounded in size of representation (unboundedly many concepts). Neural networks don't even represent causal structure, let alone model how an intervention will affect outcomes! It is, however, really nice to hear an AI booster admit just how incredibly limited connectionist models actually are. >Making them bayesian doesn't fix that. No, changing to a causal, compositional representation that allows for productive and nonparametric (unboundedly large) learning does that. The Bayesian part just makes it extra nice by letting us "put information in" anywhere in the model (at any variable) by conditioning. [1] -- http://forestdb.org/models/learning-physics.html http://forestdb.org/models/learning-physics.html [2] -- http://forestdb.org/models/word-learning.html http://forestdb.org/models/word-learning.html [3] -- http://forestdb.org/models/arithmetic.html http://forestdb.org/models/arithmetic.html [4] -- http://forestdb.org/models/politeness.html http://forestdb.org/models/politeness.html