4 ms·

Actually it’s not a trivial statement. He’s saying that we don’t have enough theory guiding us and instead we just spin up a tensorflow library without having a

Actually it’s not a trivial statement. He’s saying that we don’t have enough theory guiding us and instead we just spin up a tensorflow library without having any clue what’s going on underneath the hood. And I don’t mean understanding linear algebra in neural networks, I mean we don’t have a good theory of computation for Artificial neural networks like we do for Boolean electronic circuits. There is a little bit of work out there, but no one seems to be interested in the theory and why these work. The fact that we still represent networks as fully connected when many many of those weights w_ij are completely spurious tells me that we don’t spend enough time figuring out what’s going on and instead focus on the result. And this lack of attention will be why we’ll hit another wall. We should be reverse engineering intelligence and comparing that to AI so that we can build up the theoretical equivalent of aerodynamics.

return1 9y ago
it's not for the lack of trying however. in fact if it wasn't for people trying out random stuff they wouldn't have the leaps of 2010s in neural networks - that leap was not informed by some mathematical model. i don't see why trial-and-error cannot coexist with theory-building
- z3phyr 9y ago
  Trial and error is definitely a part of theory building. But with only fuzzing as a vector, we tend to hit a wall sooner or later.
graycat 9y ago
> We should be reverse engineering intelligence and comparing that to AI so that we can build up the theoretical equivalent of aerodynamics. Well, with a lot of assumptions, if believe they hold in practice, we have a lot of powerful theory for regression analysis, and we didn't get that theory by "reverse engineering intelligence". We got hypothesis tests, confidence intervals, prediction intervals, etc. So, more generally, we can proceed mathematically: State some assumptions and then use those to prove some theorems. We want the assumptions to be justified for our practical applications and we want the consequences of the theorems to be powerful for the applications. These steps have been followed often enough in math before, back to Euclid's plane geometry, the Pythagorean theorem, trigonometry, spherical triangles, the area of a circle, the volume of a sphere, the wave equation, ellipses, analytic geometry, calculus, differential equations, the stiffness of space frames, etc. For the current applications with no theory, maybe we need to stir up some more theorems and proofs.
- rdlecler1 9y ago
  I see this as the MIT school of though. This group was saying that we’d have strong AI back in the 60s. I believe that a brute force math approach that lacks a larger theory of how neural networks give us intelligence is going to be the slow path forward.
  - graycat 9y ago
    I was proposing how to make progress in understanding some of the data manipulations being done now by what appears to be most of ML/AI. That is, for the data manipulations, get those from the consequences of some theorems. For an example, apparently regression is important in current ML/AI: Well, going way back, 50+ years, we have a lot of solid math for regression. Then I mentioned that from Euclid through calculus to wave equations, there's more math from theorems and proofs we can and sometimes do apply. And we can stir up still more applicable math. My goal was just suggesting how to do better with real, valuable applications to important real world problems. We have such examples from US national security -- the A bomb, the H bomb, GPS, stealth, phased array radar, adaptive beam forming sonar, etc. By analogy, I was suggesting a better socket wrench or numerically controlled milling machine and not a self-driving car. I was not suggesting anything that we could regard as intelligent, not even as intelligent as a field mouse. IMHO, powerful math with valuable applications is now very doable. We can schedule equipment maintenance, airline crews, airline fleets, workers, trucks, etc. We can have computers do the data manipulations specified by the math and have the Internet move the data. But for also having systems that are as intelligent as a field mouse, kitten, puppy, octopus, song bird, even walk as well as a cockroach, that's harder. At one time I worked in some AI based on some MIT AI work; I wrote software, worked with GM Research, gave a paper at the Stanford AAAI IAAI conference, as sole author or co-author published a list of papers. Here I was not suggesting anything that has anything to do with any of the MIT AI work I've known about. For people at MIT who have done work more like I have in mind, I can think of D. Bertsekas and M. Athans. Athans was in deterministic optimal control, and the OP mentioned that that field did "back propagation" long ago. Athans is, was, a good applied mathematician. When I was at FedEx, I chatted with him in his office on how to find how best to climb, cruise, and descend airplanes. He told me a cute story about how an F-4 could get minimum time to climb, say, IIRC, to 100,000 feet: Climb up to just 5,000 feet or so, go into a dive, get supersonic where actually the drag was less, and then go nearly vertical, all supersonic, directly to 100,000 feet. For neural networks, IIRC there is some nice math that shows how general those can be for representing functions, and for some parts of stochastic optimal control Bertsekas proposed such a use for neural networks. There he, as usual, was being mathematical.
visarga 9y ago
We already hit a wall - neural nets are easily hacked. The field is called Adversarial Attacks and Defences of Neural Nets. There's no perfect fix yet and it is a major security flaw.
- rdlecler1 9y ago
  You’re example doesn’t prove they hit a wall, only that they’re fallible. Humans ‘get hacked’ every day. I wouldn’t exactly say human intelligence has hit a wall because of it, rather that human intelligence may have weaknesses and can be exploited. Frankly I wouldn’t expect neural networks to be any different.
- dingo_bat 9y ago
  > There's no perfect fix yet Reminds me of google's solution to their "racist" photos app problem: they just blacklisted certain search terms and images.
Retric 9y ago
We don't really have a theory for Boolean electronic circuits either. What separates bridges from computation is bridges are bound by the fixed laws of physics, computation is far more loosely bound by logic.
- rdlecler1 9y ago
  I’m pretty sure we have a rich body or work on information theory, Boolean logic, and circuit design. Also boolean computation is not ‘loosley bound’. It’s perfectly reproducible.
- mr_overalls 9y ago
  You really have no idea what you're talking about. Boolean logic - as a theoretical topic in itself and as it forms the basis for the circuitry of modern computing devices - is about as well-studied a topic as you can find.
  - Retric 9y ago
    What's the optimal layout for a hardware MPEG-4 decoder? We have heuristics not laws.
    - Retric 9y ago
      How about an even simpler question in the realm of Carnot efficiency. What's the theoretical minimum number of transistors to build a real time hardware MPEG-4 decoder @4k using Intel's 14nm process.

return1 9y ago

it's not for the lack of trying however. in fact if it wasn't for people trying out random stuff they wouldn't have the leaps of 2010s in neural networks - that leap was not informed by some mathematical model. i don't see why trial-and-error cannot coexist with theory-building

z3phyr 9y ago
Trial and error is definitely a part of theory building. But with only fuzzing as a vector, we tend to hit a wall sooner or later.

graycat 9y ago

> We should be reverse engineering intelligence and comparing that to AI so that we can build up the theoretical equivalent of aerodynamics. Well, with a lot of assumptions, if believe they hold in practice, we have a lot of powerful theory for regression analysis, and we didn't get that theory by "reverse engineering intelligence". We got hypothesis tests, confidence intervals, prediction intervals, etc. So, more generally, we can proceed mathematically: State some assumptions and then use those to prove some theorems. We want the assumptions to be justified for our practical applications and we want the consequences of the theorems to be powerful for the applications. These steps have been followed often enough in math before, back to Euclid's plane geometry, the Pythagorean theorem, trigonometry, spherical triangles, the area of a circle, the volume of a sphere, the wave equation, ellipses, analytic geometry, calculus, differential equations, the stiffness of space frames, etc. For the current applications with no theory, maybe we need to stir up some more theorems and proofs.

rdlecler1 9y ago
I see this as the MIT school of though. This group was saying that we’d have strong AI back in the 60s. I believe that a brute force math approach that lacks a larger theory of how neural networks give us intelligence is going to be the slow path forward.
- graycat 9y ago
  I was proposing how to make progress in understanding some of the data manipulations being done now by what appears to be most of ML/AI. That is, for the data manipulations, get those from the consequences of some theorems. For an example, apparently regression is important in current ML/AI: Well, going way back, 50+ years, we have a lot of solid math for regression. Then I mentioned that from Euclid through calculus to wave equations, there's more math from theorems and proofs we can and sometimes do apply. And we can stir up still more applicable math. My goal was just suggesting how to do better with real, valuable applications to important real world problems. We have such examples from US national security -- the A bomb, the H bomb, GPS, stealth, phased array radar, adaptive beam forming sonar, etc. By analogy, I was suggesting a better socket wrench or numerically controlled milling machine and not a self-driving car. I was not suggesting anything that we could regard as intelligent, not even as intelligent as a field mouse. IMHO, powerful math with valuable applications is now very doable. We can schedule equipment maintenance, airline crews, airline fleets, workers, trucks, etc. We can have computers do the data manipulations specified by the math and have the Internet move the data. But for also having systems that are as intelligent as a field mouse, kitten, puppy, octopus, song bird, even walk as well as a cockroach, that's harder. At one time I worked in some AI based on some MIT AI work; I wrote software, worked with GM Research, gave a paper at the Stanford AAAI IAAI conference, as sole author or co-author published a list of papers. Here I was not suggesting anything that has anything to do with any of the MIT AI work I've known about. For people at MIT who have done work more like I have in mind, I can think of D. Bertsekas and M. Athans. Athans was in deterministic optimal control, and the OP mentioned that that field did "back propagation" long ago. Athans is, was, a good applied mathematician. When I was at FedEx, I chatted with him in his office on how to find how best to climb, cruise, and descend airplanes. He told me a cute story about how an F-4 could get minimum time to climb, say, IIRC, to 100,000 feet: Climb up to just 5,000 feet or so, go into a dive, get supersonic where actually the drag was less, and then go nearly vertical, all supersonic, directly to 100,000 feet. For neural networks, IIRC there is some nice math that shows how general those can be for representing functions, and for some parts of stochastic optimal control Bertsekas proposed such a use for neural networks. There he, as usual, was being mathematical.

visarga 9y ago

We already hit a wall - neural nets are easily hacked. The field is called Adversarial Attacks and Defences of Neural Nets. There's no perfect fix yet and it is a major security flaw.

rdlecler1 9y ago
You’re example doesn’t prove they hit a wall, only that they’re fallible. Humans ‘get hacked’ every day. I wouldn’t exactly say human intelligence has hit a wall because of it, rather that human intelligence may have weaknesses and can be exploited. Frankly I wouldn’t expect neural networks to be any different.
dingo_bat 9y ago
> There's no perfect fix yet Reminds me of google's solution to their "racist" photos app problem: they just blacklisted certain search terms and images.

Retric 9y ago

We don't really have a theory for Boolean electronic circuits either. What separates bridges from computation is bridges are bound by the fixed laws of physics, computation is far more loosely bound by logic.

rdlecler1 9y ago
I’m pretty sure we have a rich body or work on information theory, Boolean logic, and circuit design. Also boolean computation is not ‘loosley bound’. It’s perfectly reproducible.
mr_overalls 9y ago
You really have no idea what you're talking about. Boolean logic - as a theoretical topic in itself and as it forms the basis for the circuitry of modern computing devices - is about as well-studied a topic as you can find.
- Retric 9y ago
  What's the optimal layout for a hardware MPEG-4 decoder? We have heuristics not laws.
  - Retric 9y ago
    How about an even simpler question in the realm of Carnot efficiency. What's the theoretical minimum number of transistors to build a real time hardware MPEG-4 decoder @4k using Intel's 14nm process.