6 ms·

Last I checked bridges came before Newtonian Mechanics and it seems strange to argue this wasn't a good thing. Admittedly paper writing wasn't the main mechanis

by TimPC 9y ago

Last I checked bridges came before Newtonian Mechanics and it seems strange to argue this wasn't a good thing. Admittedly paper writing wasn't the main mechanism of transmitting knowledge but it's fairly common for human engineering to come before the full theoretical foundations as opposed to after.

joe_the_user 9y ago
It's not that bridges before Newton were bad, it's that Newton gave us the ability to design the strongest possible bridge of a given shape with the materials at hand - using not just calculus but calculus-of-variations, a subject nearly as old as Newtonian mechanics [1]. With this knowledge, what happens when one adds one or two columns to a bridge is now longer "news" the way it might have been before Newtonian mechanics. A stereotypical picture of an engineering approach without scientific knowledge would be a list of ways to do stuff combined with hints about how to vary the approach per-situation. It requires lots memorizing, trial-and-error and experts that often can't fully explain their reasoning. It's easy to believe bridge-building before Newton was like this though I'm not an expert. Present day AI sounds a lot like from what I've read (though I'm not an expert here either). Edit: And yes, one could argue that the progress Newton ushered in merely replaced one list of models with a higher, more general list of models - yes, but that is how progress gone so far. [1] https://en.wikipedia.org/wiki/Calculus_of_variations https://en.wikipedia.org/wiki/Calculus_of_variations
- openasocket 9y ago
  I completely agree with your assessment, but the problem is a bit worse in my opinion. We already have a pretty firm grasp of how different ML systems learn and converge towards a solution in the average case. It's not that we need to understand our neural networks better, it's that we need to understand our problem domain better. We can't determine how well some ML architecture will perform at an object recognition problem without some math describing object recognition. This makes things a lot more complicated, because it means we have to do a lot more work to understand every single application where we want to use ML. And, of course, if we had some really good mathematical framework for describing and reasoning about object recognition, we probably wouldn't need to turn to ML to solve it ;)
  - bitL 9y ago
    The whole point of Deep Learning is that we don't want to describe math behind object recognition; it was the failed "classical" approach where people spent decades figuring out complex features which worked horribly. Deep Learning is actually pretty simple, well understood and parallelizable, and it's basically a billion-dimensional non-linear optimization. As optimization is infested with NP-hard problems, it's as difficult as it gets. It's actually amazing what we can do with it in the real world right now (and we are still far away from seeing all its fruits). Of course, it would frustrate academics that can't base AGI on top of it, but did they really think this approach would do it anyway?
    - Retric 9y ago
      Deep learning does not seem to abstract very well. Train on a data set then test with images that are simply upside down and the preformance can be significant. Feature extraction also works much better when you toss a lot of data and processing power behind it. So, a lot of progress is simply more data and computing power vs better approaches. Consider how poorly deep leaning works when using a single 286.
      - soVeryTired 9y ago
        > Deep learning does not seem to abstract very well. Train on a data set then test with images that are simply upside down and the preformance can be significant. But that's true of people too. How quickly can you read upside-down? If you trained on a mixture of upside-down and right way up images, and tested on upside-down images, performance wouldn't take that much of a hit.
        Retric 9y ago
        > But that's true of people too. Sure, the problem is we are more willing to ignore failures that are similar to how we fail. IMO, when we compare AI approach X vs. Y we need to consider absolute performance not just performance similar to human performance. Deep learning for example gains a lot from texture detection in images. But, that also makes it really easy to fool.
- TimPC 9y ago
  Present day AI on the Deep Learning side is a lot like what you describe. We haven't really had the Newtonian foundations yet. The theoretical foundations are quite limited because they are hard to figure out. But the techniques with less established theory work far better on most applications in AI. Redirecting work into areas of AI that have more solid theoretical foundations but worse application performance is not the way forward. I'm all for figuring out hard theoretical foundations but I'm strongly opposed to redirecting research funding to techniques that result in worse applications. I'd also argue modelling the physics isn't always the right approach: vocal tract modelling for speech is an interesting approach that produces much worse speech than state of the art synthesis techniques. It will probably continue to do so for a long time. For vocal tract modelling to produce better synthesis you'd need the physical model to be less lossy in all it's parameterizations and modelling simplifications than any statistical fitting of data. And you'd still need some statistical model of the choices the human makes in producing speech and you'd want that statistical model to work better than the neural network that takes on a larger portion of the problem and replaces the physical model of sound production.
  - TimPC 9y ago
    I should point out in this case it's almost certainly a genuine call to research the foundations underlying the working techniques more as Duvenaud publishes research using mostly the techniques that work well on applications.
  - joe_the_user 9y ago
    You're right that the analogy doesn't imply that something analogous to physic is the answer. However, I would mention that there's a larger "overhead" than many realize to methods which work without the creator or the user understanding why. You have "racist" AI which don't undertstand that correlation may not be causation in questions like whether someone should be paroled or get a loan, you have the AIs subject to adversial attacks of various sorts, where not knowing why the AI works is also problematic, you have a situation where the target to match varies over time and so-forth. Which adds up to AI having more dimensions to it than simply "working well" and "working less well". Indeed, AI is effectively ad-hoc statistics with result derived heuristically. So in the process of "getting things right" exploring all sorts of things certainly sounds good, it seems like there's an "understanding gap" that needs to be closed and some broader model of what's happening would be useful but naturally there's no guarantee we can find one.
- Swizec 9y ago
  > It requires lots memorizing, trial-and-error and experts that often can't fully explain their reasoning. You just described all of software engineering.
  - sgt101 9y ago
    Yeah - but we should call it "programming" or if we want to indicate a wider activity "software development".
- mlevental 9y ago
  >using not just calculus but calculus-of-variations, a subject nearly as old as Newtonian mechanics [1] bridges aren't catenaries and Euler and Lagrange gave us the Euler-Lagrange equations, not Newton.
DougWebb 9y ago
It's been a long time since I got my BE in Mechanical Engineering, but I still remember being struck by the difference between well-understood engineering and rule-of-thumb engineering. Bridge building is mostly well-understood engineering. When you study Static Mechanics [0] you learn all sorts of Physics equations, including Newtonian Mechanics, that completely describe the forces and motions of a structure based on measurable physical properties of the materials used and details about the shapes of those materials. When you get into Fluid Dynamics, things are different. You start to encounter a bunch of things like Reynolds number [1], which is a dimensionless value related to turbulence that you just have to look up for the particular fluids and velocities you're working with. This number is pretty well defined, but there are a lot of others and their definitions and meanings aren't nearly as clear as F=ma. Back when I was in school, particle simulations for turbulent fluids was just beginning to be feasible, so to design something you plugged in dimensionless constants and didn't worry about the unpredictable fine-details. An example of this is the wind blowing through a bridge's structure, and water flowing around its base. The equations don't give you exact forces that the turbulent air and water will exert; they give you more of an average over time. A simulation, if you can do it, can show you things (like resonance) that the equations won't show you. Then there was Strength of Materials. Here, the big thing was the Factor of Safety [2]. This is solidly in the rule-of-thumb engineering camp. This is where the engineer says "I think two 16" steel beams would be sufficient... so lets use three 20" beams just to be sure." This is still the way a lot of engineering design is done, because the real world is never precisely known, and the factor of safety will save you when something unexpected happens. [0] https://en.wikipedia.org/wiki/Statics https://en.wikipedia.org/wiki/Statics [1] https://en.wikipedia.org/wiki/Reynolds_number https://en.wikipedia.org/wiki/Reynolds_number [2] https://en.wikipedia.org/wiki/Factor_of_safety https://en.wikipedia.org/wiki/Factor_of_safety
- accidentalrebel 9y ago
  These topics bring me back. The "rule of thumb" engineering that you speak of made me remember the different constants that were taught we should just accept as is because, well, it is considered constant. Nevermind where the guy in the book got it from, this is what works and this is what people in the industry has accepted to be standard.
justincormack 9y ago
Cathedrals regularly fell down, and domes, and no doubt bridges too.
- sgt101 9y ago
  Our friend wikipedia have a good list : https://en.wikipedia.org/wiki/List_of_structural_failures_and_collapses#Antiquity_.E2.80.93_Middle_Ages https://en.wikipedia.org/wiki/List_of_structural_failures_an... Yus - bridges.

by TimPC 9y ago

joe_the_user 9y ago

It's not that bridges before Newton were bad, it's that Newton gave us the ability to design the strongest possible bridge of a given shape with the materials at hand - using not just calculus but calculus-of-variations, a subject nearly as old as Newtonian mechanics [1]. With this knowledge, what happens when one adds one or two columns to a bridge is now longer "news" the way it might have been before Newtonian mechanics. A stereotypical picture of an engineering approach without scientific knowledge would be a list of ways to do stuff combined with hints about how to vary the approach per-situation. It requires lots memorizing, trial-and-error and experts that often can't fully explain their reasoning. It's easy to believe bridge-building before Newton was like this though I'm not an expert. Present day AI sounds a lot like from what I've read (though I'm not an expert here either). Edit: And yes, one could argue that the progress Newton ushered in merely replaced one list of models with a higher, more general list of models - yes, but that is how progress gone so far. [1] https://en.wikipedia.org/wiki/Calculus_of_variations https://en.wikipedia.org/wiki/Calculus_of_variations

openasocket 9y ago
I completely agree with your assessment, but the problem is a bit worse in my opinion. We already have a pretty firm grasp of how different ML systems learn and converge towards a solution in the average case. It's not that we need to understand our neural networks better, it's that we need to understand our problem domain better. We can't determine how well some ML architecture will perform at an object recognition problem without some math describing object recognition. This makes things a lot more complicated, because it means we have to do a lot more work to understand every single application where we want to use ML. And, of course, if we had some really good mathematical framework for describing and reasoning about object recognition, we probably wouldn't need to turn to ML to solve it ;)
- bitL 9y ago
  The whole point of Deep Learning is that we don't want to describe math behind object recognition; it was the failed "classical" approach where people spent decades figuring out complex features which worked horribly. Deep Learning is actually pretty simple, well understood and parallelizable, and it's basically a billion-dimensional non-linear optimization. As optimization is infested with NP-hard problems, it's as difficult as it gets. It's actually amazing what we can do with it in the real world right now (and we are still far away from seeing all its fruits). Of course, it would frustrate academics that can't base AGI on top of it, but did they really think this approach would do it anyway?
  - Retric 9y ago
    Deep learning does not seem to abstract very well. Train on a data set then test with images that are simply upside down and the preformance can be significant. Feature extraction also works much better when you toss a lot of data and processing power behind it. So, a lot of progress is simply more data and computing power vs better approaches. Consider how poorly deep leaning works when using a single 286.
    - soVeryTired 9y ago
      > Deep learning does not seem to abstract very well. Train on a data set then test with images that are simply upside down and the preformance can be significant. But that's true of people too. How quickly can you read upside-down? If you trained on a mixture of upside-down and right way up images, and tested on upside-down images, performance wouldn't take that much of a hit.
      - Retric 9y ago
        > But that's true of people too. Sure, the problem is we are more willing to ignore failures that are similar to how we fail. IMO, when we compare AI approach X vs. Y we need to consider absolute performance not just performance similar to human performance. Deep learning for example gains a lot from texture detection in images. But, that also makes it really easy to fool.
TimPC 9y ago
Present day AI on the Deep Learning side is a lot like what you describe. We haven't really had the Newtonian foundations yet. The theoretical foundations are quite limited because they are hard to figure out. But the techniques with less established theory work far better on most applications in AI. Redirecting work into areas of AI that have more solid theoretical foundations but worse application performance is not the way forward. I'm all for figuring out hard theoretical foundations but I'm strongly opposed to redirecting research funding to techniques that result in worse applications. I'd also argue modelling the physics isn't always the right approach: vocal tract modelling for speech is an interesting approach that produces much worse speech than state of the art synthesis techniques. It will probably continue to do so for a long time. For vocal tract modelling to produce better synthesis you'd need the physical model to be less lossy in all it's parameterizations and modelling simplifications than any statistical fitting of data. And you'd still need some statistical model of the choices the human makes in producing speech and you'd want that statistical model to work better than the neural network that takes on a larger portion of the problem and replaces the physical model of sound production.
- TimPC 9y ago
  I should point out in this case it's almost certainly a genuine call to research the foundations underlying the working techniques more as Duvenaud publishes research using mostly the techniques that work well on applications.
- joe_the_user 9y ago
  You're right that the analogy doesn't imply that something analogous to physic is the answer. However, I would mention that there's a larger "overhead" than many realize to methods which work without the creator or the user understanding why. You have "racist" AI which don't undertstand that correlation may not be causation in questions like whether someone should be paroled or get a loan, you have the AIs subject to adversial attacks of various sorts, where not knowing why the AI works is also problematic, you have a situation where the target to match varies over time and so-forth. Which adds up to AI having more dimensions to it than simply "working well" and "working less well". Indeed, AI is effectively ad-hoc statistics with result derived heuristically. So in the process of "getting things right" exploring all sorts of things certainly sounds good, it seems like there's an "understanding gap" that needs to be closed and some broader model of what's happening would be useful but naturally there's no guarantee we can find one.
Swizec 9y ago
> It requires lots memorizing, trial-and-error and experts that often can't fully explain their reasoning. You just described all of software engineering.
- sgt101 9y ago
  Yeah - but we should call it "programming" or if we want to indicate a wider activity "software development".
mlevental 9y ago
>using not just calculus but calculus-of-variations, a subject nearly as old as Newtonian mechanics [1] bridges aren't catenaries and Euler and Lagrange gave us the Euler-Lagrange equations, not Newton.

DougWebb 9y ago

It's been a long time since I got my BE in Mechanical Engineering, but I still remember being struck by the difference between well-understood engineering and rule-of-thumb engineering. Bridge building is mostly well-understood engineering. When you study Static Mechanics [0] you learn all sorts of Physics equations, including Newtonian Mechanics, that completely describe the forces and motions of a structure based on measurable physical properties of the materials used and details about the shapes of those materials. When you get into Fluid Dynamics, things are different. You start to encounter a bunch of things like Reynolds number [1], which is a dimensionless value related to turbulence that you just have to look up for the particular fluids and velocities you're working with. This number is pretty well defined, but there are a lot of others and their definitions and meanings aren't nearly as clear as F=ma. Back when I was in school, particle simulations for turbulent fluids was just beginning to be feasible, so to design something you plugged in dimensionless constants and didn't worry about the unpredictable fine-details. An example of this is the wind blowing through a bridge's structure, and water flowing around its base. The equations don't give you exact forces that the turbulent air and water will exert; they give you more of an average over time. A simulation, if you can do it, can show you things (like resonance) that the equations won't show you. Then there was Strength of Materials. Here, the big thing was the Factor of Safety [2]. This is solidly in the rule-of-thumb engineering camp. This is where the engineer says "I think two 16" steel beams would be sufficient... so lets use three 20" beams just to be sure." This is still the way a lot of engineering design is done, because the real world is never precisely known, and the factor of safety will save you when something unexpected happens. [0] https://en.wikipedia.org/wiki/Statics https://en.wikipedia.org/wiki/Statics [1] https://en.wikipedia.org/wiki/Reynolds_number https://en.wikipedia.org/wiki/Reynolds_number [2] https://en.wikipedia.org/wiki/Factor_of_safety https://en.wikipedia.org/wiki/Factor_of_safety

accidentalrebel 9y ago
These topics bring me back. The "rule of thumb" engineering that you speak of made me remember the different constants that were taught we should just accept as is because, well, it is considered constant. Nevermind where the guy in the book got it from, this is what works and this is what people in the industry has accepted to be standard.

justincormack 9y ago

Cathedrals regularly fell down, and domes, and no doubt bridges too.

sgt101 9y ago
Our friend wikipedia have a good list : https://en.wikipedia.org/wiki/List_of_structural_failures_and_collapses#Antiquity_.E2.80.93_Middle_Ages https://en.wikipedia.org/wiki/List_of_structural_failures_an... Yus - bridges.