3 ms·

I'm guessing almost all of it will be going into deep learning. On an interview with Sean Carroll, Gary Marcus said that >99% of AI funding gets awarded to DL-b

I'm guessing almost all of it will be going into deep learning. On an interview with Sean Carroll, Gary Marcus said that >99% of AI funding gets awarded to DL-based projects [0]. Has anyone has tried searching for new basic operations, below the level of neural networks? We've been using these methods for years, and I doubt the first major breakthrough in ML is the most optimal method possible. Consider the extreme case of searching over all mathematical operations to see if something really novel can be discovered. How feasible would this be? [0] https://www.preposterousuniverse.com/podcast/2022/02/14/184-gary-marcus-on-artificial-intelligence-and-common-sense/ https://www.preposterousuniverse.com/podcast/2022/02/14/184-...

visarga 5y ago
What if it just takes 100B weights to show interesting behavior? Maybe it's not the method that is bad, but the problems that are hard. This recent paper shows how at 10^11 weights the network starts behaving much better, as if going through a phase change. https://twitter.com/AnthropicAI/status/1494352855972540418 https://twitter.com/AnthropicAI/status/1494352855972540418
adastra22 5y ago
That's what the AGI conferences have been exploring for the past two decades: http://agi-conference.org http://agi-conference.org
liuliu 5y ago
The problem is about the structure, not the math operations. We know for a fact how Turning-Completeness is enough for artificial intelligent machines (as these machines are mechanical, not quantum). Deep-learning is about the structure to learn representations and reinforcement learning is about the curriculum of learning actions in a world space. None of this requires "breakthrough of math operations". It is IMHO that we have enough operations in our toolbox to build artificial intelligent machines, we don't exactly know the "structure" yet.
- candiodari 5y ago
  And yet there are basic problems with our method of representation. If you take error surfaces and project them into 3 dimensions ... they look smooth. And then you look at the polynomials (because vector multiplication yields polynomials) that we use to approximate those polynomials ... they look anything but smooth. They look like those scenes you got in Doom when you go out of bounds in a level. Very, very different from nature. It's a miracle it works at all. Making ML models is like making smooth and comfortable sofas by arranging spiked rocks. Now by arranging the right rock juuuuuust right you can get any shape, and indeed you can, but the shapes we're trying to make would be a hell of a lot easier to make using pillows. It's a miracle it ever succeeds at all, and the problem definitely is the basic building blocks. Polynomials are famous for 2 reasons: 1) they're "spikey". When making more accurate matches what you're doing is adding every "higher frequency components" that do their own spiking. They fix something at one place, and screw lots of stuff up everywhere else 2) at their ends they almost always go off into infinities. Technically zeroes are possible, but you just never see that. And Neural Networks have challenges that humans don't seem to have: 1) they have sudden, very "weird" ideas. All koalas are koala's, except if you modify pixel 381 down 10%, then it's an elephant. 2) they have ridiculous predictions outside of their training range. You might say "of course they do", but humans don't, and neither do animals ... what does a reasonable human/animal do when confronted with an impossible situation? They respond the same way they respond to the closest reasonable situation they know. A human seeing a tidal wave come for him runs away from the wave. That may be stupid, if they've got little chance to outrun it, but a neural network would just stand there, start shaking, and drop uncontrolled to the floor. These problems seem related to using polynomials (as opposed to, say, sum-of-10-gaussians) for prediction. Those error surfaces we started with ... they don't quite look like 10 gaussians either ... but they kind of look a lot closer to those gaussians than to polynomials. They look like smooth, slowly sloping curves.
jonas_kgomo 5y ago
Recently there was an article from Cornell about using arbitrary physical systems for training neural nets, i think a hybrid of a physical system would be more interesting in leading to breakthrough https://news.cornell.edu/stories/2022/01/physical-systems-perform-machine-learning-computations https://news.cornell.edu/stories/2022/01/physical-systems-pe...

visarga 5y ago

What if it just takes 100B weights to show interesting behavior? Maybe it's not the method that is bad, but the problems that are hard. This recent paper shows how at 10^11 weights the network starts behaving much better, as if going through a phase change. https://twitter.com/AnthropicAI/status/1494352855972540418 https://twitter.com/AnthropicAI/status/1494352855972540418

adastra22 5y ago

That's what the AGI conferences have been exploring for the past two decades: http://agi-conference.org http://agi-conference.org

liuliu 5y ago

The problem is about the structure, not the math operations. We know for a fact how Turning-Completeness is enough for artificial intelligent machines (as these machines are mechanical, not quantum). Deep-learning is about the structure to learn representations and reinforcement learning is about the curriculum of learning actions in a world space. None of this requires "breakthrough of math operations". It is IMHO that we have enough operations in our toolbox to build artificial intelligent machines, we don't exactly know the "structure" yet.

candiodari 5y ago
And yet there are basic problems with our method of representation. If you take error surfaces and project them into 3 dimensions ... they look smooth. And then you look at the polynomials (because vector multiplication yields polynomials) that we use to approximate those polynomials ... they look anything but smooth. They look like those scenes you got in Doom when you go out of bounds in a level. Very, very different from nature. It's a miracle it works at all. Making ML models is like making smooth and comfortable sofas by arranging spiked rocks. Now by arranging the right rock juuuuuust right you can get any shape, and indeed you can, but the shapes we're trying to make would be a hell of a lot easier to make using pillows. It's a miracle it ever succeeds at all, and the problem definitely is the basic building blocks. Polynomials are famous for 2 reasons: 1) they're "spikey". When making more accurate matches what you're doing is adding every "higher frequency components" that do their own spiking. They fix something at one place, and screw lots of stuff up everywhere else 2) at their ends they almost always go off into infinities. Technically zeroes are possible, but you just never see that. And Neural Networks have challenges that humans don't seem to have: 1) they have sudden, very "weird" ideas. All koalas are koala's, except if you modify pixel 381 down 10%, then it's an elephant. 2) they have ridiculous predictions outside of their training range. You might say "of course they do", but humans don't, and neither do animals ... what does a reasonable human/animal do when confronted with an impossible situation? They respond the same way they respond to the closest reasonable situation they know. A human seeing a tidal wave come for him runs away from the wave. That may be stupid, if they've got little chance to outrun it, but a neural network would just stand there, start shaking, and drop uncontrolled to the floor. These problems seem related to using polynomials (as opposed to, say, sum-of-10-gaussians) for prediction. Those error surfaces we started with ... they don't quite look like 10 gaussians either ... but they kind of look a lot closer to those gaussians than to polynomials. They look like smooth, slowly sloping curves.

jonas_kgomo 5y ago

Recently there was an article from Cornell about using arbitrary physical systems for training neural nets, i think a hybrid of a physical system would be more interesting in leading to breakthrough https://news.cornell.edu/stories/2022/01/physical-systems-perform-machine-learning-computations https://news.cornell.edu/stories/2022/01/physical-systems-pe...