4 ms·

I don't see why you're "amazed". Neural networks are _provably_ an universal approximator. As such, they can approximate _any_ function to an arbitrary degree o

by m0zg 6y ago

I don't see why you're "amazed". Neural networks are _provably_ an universal approximator. As such, they can approximate _any_ function to an arbitrary degree of precision. It just might require an impractically large/complex neural net to do so, and you might not be able to train it yet, but it's _provably_ doable in the limit, if you remove size/training constraints. I'm a practitioner in the field, and an AI skeptic because of it. I think perceptual tasks work reasonably well, NLP barely works for anything, and any sort of cognition doesn't work at all, as of August 2020. I will be the first to admit that we don't have real "AI" yet, and we won't have it in the foreseeable future. But to say that real intelligence is not possible with the current tool set is simply incorrect. It may be impractical. It may take 100+ years to figure out how to build nets this large and train them successfully before sun burns out, but it is _theoretically_ provably doable. In a way it's kind of like fusion: you have a proof that it can work right above your head during the day. We just don't know how to do it at our "human" scale with our knowledge and capabilities. Our knowledge and capabilities, however, are not a constant - they improve exponentially over time, although most people don't perceive this process as exponential.

deleted 6y ago
[deleted]
ckcheng 6y ago
For clarity, Universal approximation doesn't mean ANNs can approximate just absolutely any function... Depending on which proof you're looking at, ANNs can provably approximate any continuous function, or any Lebesgue integrable function, or any real vector-valued continuous function on a compact subset of R^n space [1]. That's certainly a very large class of functions, but it's not just any function. Also, those proofs only work if you allow ANNs with arbitrary number of layers or layer width. Perhaps the required ANN size to approximate the AI / AGI function is so huge that it's beyond the number of particles on Earth. At what point is it beyond impractical? What if the ANN size (number of layers or neurons) is beyond the number of particles in the galaxy? Maybe it's just the wrong approach. I haven't studied those proofs in detail, but as existence proofs, they just show that some ANN exists that can approximate a given function (of a certain class). There's no guarantee that there's a general algorithm that can learn the required ANN, is there? Maybe you just have to be really lucky to randomly initialize onto the correct approximating ANN... Also, I'm not aware of any proof that human or bio NNs implement a Lebesgue integrable function, or etc. It's entirely possible bio NNs uses some weird (quantum?) physical process, and implements a function outside the scope of any ANN universal approximation theorem. I'm just a little more cautious around those universal approximation theorems. I'm more or less in agreement with your comment otherwise! [1]: https://en.wikipedia.org/wiki/Universal_approximation_theorem https://en.wikipedia.org/wiki/Universal_approximation_theore...
- m0zg 6y ago
  > Perhaps the required ANN size to approximate the AI / AGI function is so huge that it's beyond the number of particles on Earth Perhaps there's a teapot orbiting Mars. It's unlikely tho. I think a better hypothesis would be that to model a brain (if we knew how) one would need the same order of magnitude of neurons and connections. This is completely unachievable now, and even if it were technically possible nobody has a faintest clue about how to train such a thing. We do not understand _many_ things about how the brain learns (no backprop!), how it retains and retrieves memories, etc, etc. The best studied system is vision, and computer vision does actually work pretty well by now, achieving superhuman results on some tasks. Thus far the most complex brain we've been able to accurately and completely model is not even a brain at all - it's just a handful of neurons of a worm. I took an online course in neuroscience a few years ago hoping to gain some insight for my ANN work. The situation with our understanding of even primitive brains of i.e. rodents is, well, primitive. We also don't really have math to accurately model more than a handful of real, biological spiking neurons - PDEs get way, way too complex for anybody to handle. But one could argue we don't have to: ANNs effectively operate in the frequency domain instead, where magnitude of activation approximates spike train frequency, and timing effects are ignored. Don't get me wrong, I'm pretty sure if we do achieve real AGI, it likely won't be with anything even remotely like the current tech and science stack. Energy efficiency and compute density is just not sufficient. But I'm also pretty confident that even with the current stack we will be able to, eventually, approximate intelligence closely enough to suit many practical uses that seem completely out of reach today, including even some limited degree of logical inference, and dare I say, cognition. If only people (including myself) weren't so chickenshit and actually seriously worked on (and funded) this stuff, that could even happen in my lifetime. As things stand right now, people are afraid to touch anything AGI related due mostly to fear of ridicule. In my comment above I am merely referring to the conjecture that _if_ we had ANNs that are large enough, and _if_ we knew how to structure and train them, there doesn't seem to be anything that would _fundamentally_ prevent them from supporting AGI. Those are very, very massive "ifs" indeed, without even a faintest glimmer of light at the end of the tunnel.
  - tsimionescu 6y ago
    I completely agree with your conclusions starting from these ifs! However, I would add that another interesting area of study that is essentially not being explored at all is exactly to try to understand more about human cognition at the computational level (whereas neuroscience is working at the biological level). Basically, it seems we have abandoned any work in AI that would try to model human learning or human understanding of language in a human-understandable way, and instead we are aiming for reproducing human behaviors for engineering purposes.
tsimionescu 6y ago
I'm not claiming that an NN can't approximate human or super human intelligence, I'm claiming that using human writing as the only training set is not going to get you there. The text simply doesn't contain enough information to draw a model of the world. Human language only makes sense if you understand the world, not the other way around. Essentially what we're doing is similar to a human trying to learn psychology by reading books in Mandarin on it, while knowing neither the writing system nor the language. Except that human languages share some universal grammar, and share more or less the same notions about the world.

by m0zg 6y ago

deleted 6y ago

[deleted]

ckcheng 6y ago

For clarity, Universal approximation doesn't mean ANNs can approximate just absolutely any function... Depending on which proof you're looking at, ANNs can provably approximate any continuous function, or any Lebesgue integrable function, or any real vector-valued continuous function on a compact subset of R^n space [1]. That's certainly a very large class of functions, but it's not just any function. Also, those proofs only work if you allow ANNs with arbitrary number of layers or layer width. Perhaps the required ANN size to approximate the AI / AGI function is so huge that it's beyond the number of particles on Earth. At what point is it beyond impractical? What if the ANN size (number of layers or neurons) is beyond the number of particles in the galaxy? Maybe it's just the wrong approach. I haven't studied those proofs in detail, but as existence proofs, they just show that some ANN exists that can approximate a given function (of a certain class). There's no guarantee that there's a general algorithm that can learn the required ANN, is there? Maybe you just have to be really lucky to randomly initialize onto the correct approximating ANN... Also, I'm not aware of any proof that human or bio NNs implement a Lebesgue integrable function, or etc. It's entirely possible bio NNs uses some weird (quantum?) physical process, and implements a function outside the scope of any ANN universal approximation theorem. I'm just a little more cautious around those universal approximation theorems. I'm more or less in agreement with your comment otherwise! [1]: https://en.wikipedia.org/wiki/Universal_approximation_theorem https://en.wikipedia.org/wiki/Universal_approximation_theore...

m0zg 6y ago
> Perhaps the required ANN size to approximate the AI / AGI function is so huge that it's beyond the number of particles on Earth Perhaps there's a teapot orbiting Mars. It's unlikely tho. I think a better hypothesis would be that to model a brain (if we knew how) one would need the same order of magnitude of neurons and connections. This is completely unachievable now, and even if it were technically possible nobody has a faintest clue about how to train such a thing. We do not understand _many_ things about how the brain learns (no backprop!), how it retains and retrieves memories, etc, etc. The best studied system is vision, and computer vision does actually work pretty well by now, achieving superhuman results on some tasks. Thus far the most complex brain we've been able to accurately and completely model is not even a brain at all - it's just a handful of neurons of a worm. I took an online course in neuroscience a few years ago hoping to gain some insight for my ANN work. The situation with our understanding of even primitive brains of i.e. rodents is, well, primitive. We also don't really have math to accurately model more than a handful of real, biological spiking neurons - PDEs get way, way too complex for anybody to handle. But one could argue we don't have to: ANNs effectively operate in the frequency domain instead, where magnitude of activation approximates spike train frequency, and timing effects are ignored. Don't get me wrong, I'm pretty sure if we do achieve real AGI, it likely won't be with anything even remotely like the current tech and science stack. Energy efficiency and compute density is just not sufficient. But I'm also pretty confident that even with the current stack we will be able to, eventually, approximate intelligence closely enough to suit many practical uses that seem completely out of reach today, including even some limited degree of logical inference, and dare I say, cognition. If only people (including myself) weren't so chickenshit and actually seriously worked on (and funded) this stuff, that could even happen in my lifetime. As things stand right now, people are afraid to touch anything AGI related due mostly to fear of ridicule. In my comment above I am merely referring to the conjecture that _if_ we had ANNs that are large enough, and _if_ we knew how to structure and train them, there doesn't seem to be anything that would _fundamentally_ prevent them from supporting AGI. Those are very, very massive "ifs" indeed, without even a faintest glimmer of light at the end of the tunnel.
- tsimionescu 6y ago
  I completely agree with your conclusions starting from these ifs! However, I would add that another interesting area of study that is essentially not being explored at all is exactly to try to understand more about human cognition at the computational level (whereas neuroscience is working at the biological level). Basically, it seems we have abandoned any work in AI that would try to model human learning or human understanding of language in a human-understandable way, and instead we are aiming for reproducing human behaviors for engineering purposes.

tsimionescu 6y ago

I'm not claiming that an NN can't approximate human or super human intelligence, I'm claiming that using human writing as the only training set is not going to get you there. The text simply doesn't contain enough information to draw a model of the world. Human language only makes sense if you understand the world, not the other way around. Essentially what we're doing is similar to a human trying to learn psychology by reading books in Mandarin on it, while knowing neither the writing system nor the language. Except that human languages share some universal grammar, and share more or less the same notions about the world.