3 ms·

> So how then can anyone expect that we're going to solve the hard problems the brain can, with ANNs? Because RNNs can approximate any self-referential circui

by wyager 10y ago

> So how then can anyone expect that we're going to solve the hard problems the brain can, with ANNs? Because RNNs can approximate any self-referential circuit with reasonable efficiency. Just like the brain does with neurons.

YeGoblynQueenne 10y ago
Sure. And all multi-layer networks with three or more layers can approximate any function with arbitrary accuracy... given sufficient many inputs. Where "sufficient many" translates as "for real problems, just too many".
- wyager 10y ago
  Note that I said "with reasonable efficiency", not "with some huge number of inputs". That's because we don't need to represent any function; we need to represent the class of functions that can be efficiently represented in a human brain as well, which is pretty much the same. Note that we can also implement any boolean component with only a few neurons in an NN, and using an RNN gives us working memory as well, so we can implement any sort of digital processor with reasonable efficiency in an RNN (where "reasonable efficiency" means "a linear multiple of the number of components in the original circuit").
  - YeGoblynQueenne 10y ago
    >> That's because we don't need to represent any function; we need to represent the class of functions that can be efficiently represented in a human brain as well, which is pretty much the same. The problem is that to learn a function from examples you need the right kind of examples, and for human cognitive faculties it's very hard to get that. For instance, take text- text is a staple in machine-learning models of language... but it is not language. It's a bunch of symbols that are only intelligible in the context of an already existing language faculty. In other words, text means nothing unless you already understand language, which is why although we can learn pretty good models of text, we haven't made much progress in learning models of language. Computers can generate or recognise language pretty damn well- but when it comes to understanding it... Well, we haven't even convincingly defined that task, let alone being able to train anything, RNN or whatever, to perform it. You can see similar issues with speech or image processing, where btw RNNs have performed much better than with language. So, just because RNNs can learn functions in principle it doesn't mean that we can really reproduce human behaviour in practice.

by wyager 10y ago

YeGoblynQueenne 10y ago

Sure. And all multi-layer networks with three or more layers can approximate any function with arbitrary accuracy... given sufficient many inputs. Where "sufficient many" translates as "for real problems, just too many".

wyager 10y ago
Note that I said "with reasonable efficiency", not "with some huge number of inputs". That's because we don't need to represent any function; we need to represent the class of functions that can be efficiently represented in a human brain as well, which is pretty much the same. Note that we can also implement any boolean component with only a few neurons in an NN, and using an RNN gives us working memory as well, so we can implement any sort of digital processor with reasonable efficiency in an RNN (where "reasonable efficiency" means "a linear multiple of the number of components in the original circuit").
- YeGoblynQueenne 10y ago
  >> That's because we don't need to represent any function; we need to represent the class of functions that can be efficiently represented in a human brain as well, which is pretty much the same. The problem is that to learn a function from examples you need the right kind of examples, and for human cognitive faculties it's very hard to get that. For instance, take text- text is a staple in machine-learning models of language... but it is not language. It's a bunch of symbols that are only intelligible in the context of an already existing language faculty. In other words, text means nothing unless you already understand language, which is why although we can learn pretty good models of text, we haven't made much progress in learning models of language. Computers can generate or recognise language pretty damn well- but when it comes to understanding it... Well, we haven't even convincingly defined that task, let alone being able to train anything, RNN or whatever, to perform it. You can see similar issues with speech or image processing, where btw RNNs have performed much better than with language. So, just because RNNs can learn functions in principle it doesn't mean that we can really reproduce human behaviour in practice.