7 ms·

that's a trivial statement, unless he can suggest a different way to do things. Should people have held off building any bridges, until the 20th century arrived

that's a trivial statement, unless he can suggest a different way to do things. Should people have held off building any bridges, until the 20th century arrived ? Building things and science have always evolved side-by-side in a feedback loop.

blacksmith_tb 8y ago
I'm not sure it was intended to be read "Can't win? Don't try." An interesting corollary to that analogy is that some of the things humans built thousands of years ago are still standing... so it's possible some of our systems could also pass the test of time (they can be optimized more easily than a bridge).
- return1 8y ago
  he does point elsewhere that machine learning solutions must account for "long tail" data , so it seems to me he 's looking for perfect solutions that will never exist.
rdlecler1 8y ago
Actually it’s not a trivial statement. He’s saying that we don’t have enough theory guiding us and instead we just spin up a tensorflow library without having any clue what’s going on underneath the hood. And I don’t mean understanding linear algebra in neural networks, I mean we don’t have a good theory of computation for Artificial neural networks like we do for Boolean electronic circuits. There is a little bit of work out there, but no one seems to be interested in the theory and why these work. The fact that we still represent networks as fully connected when many many of those weights w_ij are completely spurious tells me that we don’t spend enough time figuring out what’s going on and instead focus on the result. And this lack of attention will be why we’ll hit another wall. We should be reverse engineering intelligence and comparing that to AI so that we can build up the theoretical equivalent of aerodynamics.
- return1 8y ago
  it's not for the lack of trying however. in fact if it wasn't for people trying out random stuff they wouldn't have the leaps of 2010s in neural networks - that leap was not informed by some mathematical model. i don't see why trial-and-error cannot coexist with theory-building
  - z3phyr 8y ago
    Trial and error is definitely a part of theory building. But with only fuzzing as a vector, we tend to hit a wall sooner or later.
- graycat 8y ago
  > We should be reverse engineering intelligence and comparing that to AI so that we can build up the theoretical equivalent of aerodynamics. Well, with a lot of assumptions, if believe they hold in practice, we have a lot of powerful theory for regression analysis, and we didn't get that theory by "reverse engineering intelligence". We got hypothesis tests, confidence intervals, prediction intervals, etc. So, more generally, we can proceed mathematically: State some assumptions and then use those to prove some theorems. We want the assumptions to be justified for our practical applications and we want the consequences of the theorems to be powerful for the applications. These steps have been followed often enough in math before, back to Euclid's plane geometry, the Pythagorean theorem, trigonometry, spherical triangles, the area of a circle, the volume of a sphere, the wave equation, ellipses, analytic geometry, calculus, differential equations, the stiffness of space frames, etc. For the current applications with no theory, maybe we need to stir up some more theorems and proofs.
  - rdlecler1 8y ago
    I see this as the MIT school of though. This group was saying that we’d have strong AI back in the 60s. I believe that a brute force math approach that lacks a larger theory of how neural networks give us intelligence is going to be the slow path forward.
    - 8y ago

blacksmith_tb 8y ago

I'm not sure it was intended to be read "Can't win? Don't try." An interesting corollary to that analogy is that some of the things humans built thousands of years ago are still standing... so it's possible some of our systems could also pass the test of time (they can be optimized more easily than a bridge).

return1 8y ago
he does point elsewhere that machine learning solutions must account for "long tail" data , so it seems to me he 's looking for perfect solutions that will never exist.

rdlecler1 8y ago

Actually it’s not a trivial statement. He’s saying that we don’t have enough theory guiding us and instead we just spin up a tensorflow library without having any clue what’s going on underneath the hood. And I don’t mean understanding linear algebra in neural networks, I mean we don’t have a good theory of computation for Artificial neural networks like we do for Boolean electronic circuits. There is a little bit of work out there, but no one seems to be interested in the theory and why these work. The fact that we still represent networks as fully connected when many many of those weights w_ij are completely spurious tells me that we don’t spend enough time figuring out what’s going on and instead focus on the result. And this lack of attention will be why we’ll hit another wall. We should be reverse engineering intelligence and comparing that to AI so that we can build up the theoretical equivalent of aerodynamics.

return1 8y ago
it's not for the lack of trying however. in fact if it wasn't for people trying out random stuff they wouldn't have the leaps of 2010s in neural networks - that leap was not informed by some mathematical model. i don't see why trial-and-error cannot coexist with theory-building
- z3phyr 8y ago
  Trial and error is definitely a part of theory building. But with only fuzzing as a vector, we tend to hit a wall sooner or later.
graycat 8y ago
> We should be reverse engineering intelligence and comparing that to AI so that we can build up the theoretical equivalent of aerodynamics. Well, with a lot of assumptions, if believe they hold in practice, we have a lot of powerful theory for regression analysis, and we didn't get that theory by "reverse engineering intelligence". We got hypothesis tests, confidence intervals, prediction intervals, etc. So, more generally, we can proceed mathematically: State some assumptions and then use those to prove some theorems. We want the assumptions to be justified for our practical applications and we want the consequences of the theorems to be powerful for the applications. These steps have been followed often enough in math before, back to Euclid's plane geometry, the Pythagorean theorem, trigonometry, spherical triangles, the area of a circle, the volume of a sphere, the wave equation, ellipses, analytic geometry, calculus, differential equations, the stiffness of space frames, etc. For the current applications with no theory, maybe we need to stir up some more theorems and proofs.
- rdlecler1 8y ago
  I see this as the MIT school of though. This group was saying that we’d have strong AI back in the 60s. I believe that a brute force math approach that lacks a larger theory of how neural networks give us intelligence is going to be the slow path forward.
  - 8y ago