7 ms·

The most relevant part of the article: David Duvenaud, an assistant professor in the same department as Hinton at the University of Toronto, says deep learning

The most relevant part of the article: David Duvenaud, an assistant professor in the same department as Hinton at the University of Toronto, says deep learning has been somewhat like engineering before physics. “Someone writes a paper and says, ‘I made this bridge and it stood up!’ Another guy has a paper: ‘I made this bridge and it fell down—but then I added pillars, and then it stayed up.’ Then pillars are a hot new thing. Someone comes up with arches, and it’s like, ‘Arches are great!’” With physics, he says, “you can actually understand what’s going to work and why.” Only recently, he says, have we begun to move into that phase of actual understanding with artificial intelligence. Hinton himself says, “Most conferences consist of making minor variations … as opposed to thinking hard and saying, ‘What is it about what we’re doing now that’s really deficient? What does it have difficulty with? Let’s focus on that.’”

bra-ket 9y ago
it's really simple 1) learn how the brain works 2) build a simulator most current AI research skips step 1
- danieltillett 9y ago
  In defence of AI researchers 1 is very, very hard and to the best of our knowledge there is not one way the brain works. The brain is a complex, cobbled together set of systems all using different ways of problem solving.
  - bra-ket 9y ago
    most AI researchers have never opened a textbook on cognitive psychology or neurobiology , or any of these 'soft' sciences. how do you plan to build artificial intelligence with no model of intelligence, without learning about important experiments in learning and memory , it's the complete ignorance that drives me crazy. AI is not for specialists.
    - danieltillett 9y ago
      Yes all of this is true. I do think studying how the brain works will provide very useful ideas of what might work in AI. At the very least it is a very interesting area to learn about.
    - openasocket 9y ago
      Most of those experts aren't looking to solve general AI problems, they're looking for solutions to specific problems like basic image recognition. And you don't need a full human brain to do that, and you don't need to conform to the way humans and other biological systems do it. You're not aiming for full human intelligence, so you don't need to care too much about how humans learn. That said, I find when trying to solve a problem with ML techniques, it's better to use someone who knows the problem domain really well than someone who only knows ML really well. Someone who really understands the problem they're trying to solve can encode that knowledge into their models when training the system. While I've seen people who really know ML but lack the specific domain knowledge labor for weeks, coming back to me with "discoveries" that are already well known.
- TimPC 9y ago
  We know the brain and associated sensor behaviours are too large for us to fully simulate in a reasonable way on anything resembling current hardware (We also can't fully model it but as we approached the size of hardware to do so we'd probably solve many of the problems of doing so). So which hacks and shortcuts do you want to apply to reduce the dimensionality to something runnable? Step 1) will take far too long so AI research looks for things it can do well in the category of 2) without being a full simulation. Deep Learning has been unreasonably effective here.
- sgt101 9y ago
  I think we should be more interested in how the mind works. Much of AI is a simulator of the mind.
- dragontamer 9y ago
  > 1) learn how the brain works 2) build a simulator I disagree that step #1 is important. Consider the "Air-foil", which led to flight. In one sense, its an approximation of the wings of birds and other animals. But ultimately, the discovery that the "Air-foil" shape turns sideways blowing wind into an upward force now called "lift" is completely different from how most people understand bird wings. Bird Wings flap, but Airplane Air Foils do not. -------- Another example: Neural Networks are one of the best mathematical simulations of the human brain (as we understand it, as well as a few simplifications to make Artificial Neural Networks possible to run on modern GPUs / CPUs). However, the big advances in "Game AI" the past few years are: 1. Monte Carlo Tree Search -- AlphaGo (although some of it is Neural Network training, the MCTS is the core of the algorithm) 2. Counterfactual Regret Minimization -- The Poker AI that out-bluffed humans There are other methodologies which have proven very successful, despite little to no biological roots. IIRC, Bayesian Inference is a widely deployed machine learning technique (for some definition of Machine Learning at least), but has almost nothing to do with how a human brain works. An interesting field of AI is "Genetic Algorithms", which have biological roots but not anything based on the biology of brains, to achieve machine learning. Overall, a "Genetic Algorithm" is really just a randomized search in a multidimensional problem, but the idea of it was inspired by Darwinian Evolution.
  - ridgeguy 9y ago
    Disclaimer: I have no expertise in AI. That said, I agree that learning how the brain works seems unimportant and unnecessary. Evolution doesn't know how a brain works, but it's given us Einstein, Michelangelo, and conversations on HN. It seems really important to learn how to build evolution into attempts at AI, given that evolution is the only known mechanism that leads to what we recognize as intelligence.
    - throwaway90001 9y ago
      > it's given us Einstein, Michelangelo, and conversations on HN two out of three ain't bad
    - dragontamer 9y ago
- psyc 9y ago
  In industry, yes. In academia, there's computational neuroscience: http://cocosci.mit.edu/people http://cocosci.mit.edu/people https://web.stanford.edu/group/mbc/research.html https://web.stanford.edu/group/mbc/research.html
- anakron 9y ago
  That's certainly one way to do it. However, we didn't succeed at building modern aircraft or earth moving machinery by building simulations of birds or muscles. There's enough that is unknown out there for a variety of approaches.
- white-flame 9y ago
  Medical research hasn't cracked step 1 either, at least not to a point of accurate simulation. Besides, if you could simulate a human brain, you will end up with something that needs to sleep, something with limited and unreliable memory, something that gets bored and distracted, something emotionally needy, etc. Then the extending of this chaotic, messy system is wildly unknown even if we could get a piece-for-piece replication to work. Such a thing would be of great benefit to medicine, but not really for AI to even start with until medicine is done reverse engineering it.
  - bra-ket 9y ago
    Piece-for-piece replication might not be the right level of abstraction. Blue Brain project is one unfortunate example, on the other hand the current neural nets are stuck with neural model from 1943.
- bra-ket 9y ago
  Prof. Hinton has an interesting talk about his new 'capsule' model based on psychology of shape perception: https://www.youtube.com/watch?v=rTawFwUvnLE https://www.youtube.com/watch?v=rTawFwUvnLE
PaulHoule 9y ago
Hinton was able to survive 30 years in the academic wilderness. Most academics can't. Thus they work on "safe" projects.
- disgruntledphd2 9y ago
  It can be done, even today. If you work outside the US and work on cheap things (i.e. no special equipment), especially if you can teach then you can hang around for a long time. I have met a lot of academics like this over the years, but I think your broader point might be that this is not possible today, which I agree with, and which is why I left academia (modulo personal situations).
TimPC 9y ago
Last I checked bridges came before Newtonian Mechanics and it seems strange to argue this wasn't a good thing. Admittedly paper writing wasn't the main mechanism of transmitting knowledge but it's fairly common for human engineering to come before the full theoretical foundations as opposed to after.
- joe_the_user 9y ago
  It's not that bridges before Newton were bad, it's that Newton gave us the ability to design the strongest possible bridge of a given shape with the materials at hand - using not just calculus but calculus-of-variations, a subject nearly as old as Newtonian mechanics [1]. With this knowledge, what happens when one adds one or two columns to a bridge is now longer "news" the way it might have been before Newtonian mechanics. A stereotypical picture of an engineering approach without scientific knowledge would be a list of ways to do stuff combined with hints about how to vary the approach per-situation. It requires lots memorizing, trial-and-error and experts that often can't fully explain their reasoning. It's easy to believe bridge-building before Newton was like this though I'm not an expert. Present day AI sounds a lot like from what I've read (though I'm not an expert here either). Edit: And yes, one could argue that the progress Newton ushered in merely replaced one list of models with a higher, more general list of models - yes, but that is how progress gone so far. [1] https://en.wikipedia.org/wiki/Calculus_of_variations https://en.wikipedia.org/wiki/Calculus_of_variations
  - openasocket 9y ago
    I completely agree with your assessment, but the problem is a bit worse in my opinion. We already have a pretty firm grasp of how different ML systems learn and converge towards a solution in the average case. It's not that we need to understand our neural networks better, it's that we need to understand our problem domain better. We can't determine how well some ML architecture will perform at an object recognition problem without some math describing object recognition. This makes things a lot more complicated, because it means we have to do a lot more work to understand every single application where we want to use ML. And, of course, if we had some really good mathematical framework for describing and reasoning about object recognition, we probably wouldn't need to turn to ML to solve it ;)
aub3bhat 9y ago
Humanity used fire for a long time before combustion was understood. Even today Anesthesia is not well understood at biological/physiological level that has not stopped its safe use and innovation through Clinical Trials. Maybe competitions and empiricism are the best approaches to building intelligent systems. Why get caught up in Physics/Math envy?
- AlexCoventry 9y ago
  Hinton isn't saying "Let's stop using fire," but "Let's understand the principles behind fire so we can use them in more sophisticated, informed and powerful ways."
  - aub3bhat 9y ago
    The ML community did take the Theory approach trying to prove bounds, SLT/SRM, PAC, etc. and that was an excercise in futility. While I don't deny that there is value to looking under the hood but for a long while the community abandoned any empirical results that didn't fit their paradigm. Between rigorously validating their methods and writing yet another 4 page long proof. A lot of researchers would prefer latter, effectively locking out empirical approaches from most dissemination venues and eventually funding.
- shimon 9y ago
  Because an improved theoretical understanding of how complex systems work can be incredibly valuable. Anesthesia is a good example - we get a lot of value from it, yes, but it would be way better if we could tailor dosages to individuals based on an understanding of how that individual will experience pain. There would be fewer severe complications, but also maybe you could wake up refreshed an hour after surgery instead of in a stupor. If this could work with computer-trained models, that would be incredible too. What could a great speech understanding system teach us about language? What tricks from a facial-expression classifier could help autistic kids understand their friends?
- ballenf 9y ago
  I took his point not to criticize those early stages, but simply to acknowledge them as such. Early fire users could not have built a rocket no matter how many experiments they performed until they understood combustion (and some other sciences). In AI, we're not building rockets yet, but we have some really awesome and really powerful bonfires or whatever. At least that's how I understood his point. (And in anesthesia, when we do understand those things, we may very well look on our use today as barbaric or dangerous.)
sgt101 9y ago
"engineering before physics" is exactly wrong. No one did Engineering before a sophisticated understanding of Physics was achieved. They built bridges and towers, Engineering enables statements to be made about the performance of machines and buildings; it will survive a wind like x, you can do n cycles, do not load the wings in this way.
- empath75 9y ago
  I guess the Romans didn’t have any engineers building siege engines and fortifications, then.
  - sgt101 9y ago
    Here is the test. Take the best Roman engineer. Translate a first year engineering paper on structures into Latin. Ask Roman to sit said paper. What will happen and why? The Roman chap will look very confused and will make statements (in Latin) about how stupid this stuff is and how it has nothing to do with proper engineering. The Roman will score 0. The why is that the understanding of structures and materials in the ancient world was artizanal, based on trade knowledge (often secret and hard to reproduce) and not systematic, based on the scientific method and inspectable or testable. Currently we accept that knives, cabinets and sheds may be built or made using artisanal knowledge, we do not accept that apartment blocks, aircraft or automobiles are built this way. Society insists that these are built using systematic knowledge because otherwise they sometimes fall down or crash. The systematic approach to aircraft is the best example - think how much civil air traffic there is now, and how rare air crashes are. The issues of subsonic flight have been systematically accounted for, right up to the point where we now see 1:2,000,000 crashes per flight. Mechanical, aeronautical and civil engineering proceed in this way. Issues are discovered with mechanisms or structures or materials, these are characterized with scientific investigation, the characterizations lead to constraints and parameters that are required to be accounted for in future designs and old designs are re-evaluated in the light of the new knowledge. Stating that you will build a new building in a certain way because domes are strong and concrete is strong would not cut the mustard in the modern world... The parthenon has stood for 2000 years, but how many similar structures collapsed after a few months?
markan 9y ago
Great quote from Hinton. The biggest deficiency in AI is that we still don't have artificial systems which simulate human thought with any fidelity. Sooner or later that's bound to become a focus of attention.
- nesyt 9y ago
  Why bother simulating human thought? It's not the only road to Rome.
  - Demiurge 9y ago
    It's probably more of a map than a road.
make3 9y ago
except that this has really only been going on for five years, which is nothing in the scale of human history or even of human rational thought. Some record number of people/scientists are working on getting the physics level understanding to happen, with crazy record breaking year after year quantity of people publishing and attending scientific conferences that fill up in like two days now. it is happening, and will happen even more in depth as time goes on
indescions_2017 9y ago
OTOH, we are merely at circa Year Five into deep reinforcement learning research. It started as a cluster of 16M CPUs having taught itself to recognize a cat 95% of the time after training on 1B google images. And we are now at One-Shot Imitation Learning, "a general system that can turn any demonstrations into robust policies that can accomplish an overwhelming variety of tasks". One Shot Imitation Learning https://arxiv.org/abs/1703.07326 https://arxiv.org/abs/1703.07326
- graycat 9y ago
  It's easy to recognize a cat 95% of the time. I can write a program in 30 seconds that will recognize a cat 95% of the time. No, wait, this just in! My program will recognize a cat 100% of the time! The program has just one line: Print "It's a cat!"
  - graycat 9y ago
    Tutorial: So, with that program, whenever the picture is a cat, the program DOES recognize it. So the program DOES recognize a cat 100% of the time. The OP only claimed 95% of the time. Uh, we need TWO (2), that's TWO numbers: conditional probability of recognizing a cat when there is one (detection rate) conditional probability of claiming there is a cat when there isn't one. The second is the false alarm rate or the conditional probability of a false alarm or the conditional probability of Type I error or the significance level of the test or the p-value, the most heavily used quantity in all of statistics. One minus the detection rate is the conditional probability of Type II error. Typically we can adjust the false alarm rate, and, if we are willing to accept a higher false alarm rate, then we can get a higher detection rate. With my little program, the false alarm rate is also 100%. So, as a detector, my little program is worthless. But the program does have a 100% detection rate, and that's 5% better than the OP claimed. If focus ONLY on detection rate, that is, recognizing a cat when there is one, then it's easy to get a 100% detection rate with just a trivial test -- just say everything is a cat as I did. What's tricky is to have the detection rate high and the false alarm rate low. The best way to do that is in the classic Neyman-Pearson lemma. A good proof is possible using the Hahn decomposition from the Radon-Nikodym theorem in measure theory with the famous proof by von Neumann in W. Rudin, Real and Complex Analysis. My little program was correct and not a joke. Again, to evaluate a detector, need TWO, that's two, or 1 + 1 = 2 numbers. What about a detector that is overall 95% correct? That's easy, too: Just show my detector cats 95% of the time. If we are to be good at computer science, data science, ML/AI, and dip our toes into a huge ocean of beautifully done applied math, then we need to understand Type I and Type II errors. Sorry 'bout that. Are we learning yet?
justonepost 9y ago
The fundamental problem with AI is the high dimensionality of the solution space. We simply can't understand why the brains we are building can think better than us. We can build smarter brains only by trial and error - at least until error outsmarts us, reproduces and takes over. Kind of like having kids.
deleted 9y ago
[deleted]
olalonde 9y ago
Hinton's quote is taken a bit out of context though. I just watched his interview on Andrew Ng's "Neural Networks and Deep Learning" class on Coursera and he seemed convinced that the next "breakthrough" will come from (a variant on) neural networks.
- meheleventyone 9y ago
  Right but to extend the bridge analogy what's interesting isn't the materials (neural nets) as such but the structure and why that structure works.
zby 9y ago
But maybe there are no universal laws that govern AI like physics governs bridges? AI is something that finds universal laws in stuff - there is no meta level over this - all the meta is AI itself.
zwischenzug 9y ago
I feel like Hofstadter was one of those people thinking really deeply about AI. Anyone who doesn't know what I'm talking about should read 'Goedel, Escher, Bach', or 'Fluid Analogies'. I haven't read them in a long while, but I'm sure they're going to be relevant for decades, because they deal with the fundamental challenge of what it means to think. Backpropagation may be part of the puzzle, but the brain (and intelligence) is so much more than that.
- markan 9y ago
  I second this recommendation! Here's some more reading for anyone interested in Hofstadter: http://www.popularmechanics.com/science/a3278/why-watson-and-siri-are-not-real-ai-16477207/ http://www.popularmechanics.com/science/a3278/why-watson-and... http://www.basicai.org/blog/hofstadter-2017-09-25.html http://www.basicai.org/blog/hofstadter-2017-09-25.html

bra-ket 9y ago

it's really simple 1) learn how the brain works 2) build a simulator most current AI research skips step 1

danieltillett 9y ago
In defence of AI researchers 1 is very, very hard and to the best of our knowledge there is not one way the brain works. The brain is a complex, cobbled together set of systems all using different ways of problem solving.
- bra-ket 9y ago
  most AI researchers have never opened a textbook on cognitive psychology or neurobiology , or any of these 'soft' sciences. how do you plan to build artificial intelligence with no model of intelligence, without learning about important experiments in learning and memory , it's the complete ignorance that drives me crazy. AI is not for specialists.
  - danieltillett 9y ago
    Yes all of this is true. I do think studying how the brain works will provide very useful ideas of what might work in AI. At the very least it is a very interesting area to learn about.
  - openasocket 9y ago
    Most of those experts aren't looking to solve general AI problems, they're looking for solutions to specific problems like basic image recognition. And you don't need a full human brain to do that, and you don't need to conform to the way humans and other biological systems do it. You're not aiming for full human intelligence, so you don't need to care too much about how humans learn. That said, I find when trying to solve a problem with ML techniques, it's better to use someone who knows the problem domain really well than someone who only knows ML really well. Someone who really understands the problem they're trying to solve can encode that knowledge into their models when training the system. While I've seen people who really know ML but lack the specific domain knowledge labor for weeks, coming back to me with "discoveries" that are already well known.
TimPC 9y ago
We know the brain and associated sensor behaviours are too large for us to fully simulate in a reasonable way on anything resembling current hardware (We also can't fully model it but as we approached the size of hardware to do so we'd probably solve many of the problems of doing so). So which hacks and shortcuts do you want to apply to reduce the dimensionality to something runnable? Step 1) will take far too long so AI research looks for things it can do well in the category of 2) without being a full simulation. Deep Learning has been unreasonably effective here.
sgt101 9y ago
I think we should be more interested in how the mind works. Much of AI is a simulator of the mind.
dragontamer 9y ago
> 1) learn how the brain works 2) build a simulator I disagree that step #1 is important. Consider the "Air-foil", which led to flight. In one sense, its an approximation of the wings of birds and other animals. But ultimately, the discovery that the "Air-foil" shape turns sideways blowing wind into an upward force now called "lift" is completely different from how most people understand bird wings. Bird Wings flap, but Airplane Air Foils do not. -------- Another example: Neural Networks are one of the best mathematical simulations of the human brain (as we understand it, as well as a few simplifications to make Artificial Neural Networks possible to run on modern GPUs / CPUs). However, the big advances in "Game AI" the past few years are: 1. Monte Carlo Tree Search -- AlphaGo (although some of it is Neural Network training, the MCTS is the core of the algorithm) 2. Counterfactual Regret Minimization -- The Poker AI that out-bluffed humans There are other methodologies which have proven very successful, despite little to no biological roots. IIRC, Bayesian Inference is a widely deployed machine learning technique (for some definition of Machine Learning at least), but has almost nothing to do with how a human brain works. An interesting field of AI is "Genetic Algorithms", which have biological roots but not anything based on the biology of brains, to achieve machine learning. Overall, a "Genetic Algorithm" is really just a randomized search in a multidimensional problem, but the idea of it was inspired by Darwinian Evolution.
- ridgeguy 9y ago
  Disclaimer: I have no expertise in AI. That said, I agree that learning how the brain works seems unimportant and unnecessary. Evolution doesn't know how a brain works, but it's given us Einstein, Michelangelo, and conversations on HN. It seems really important to learn how to build evolution into attempts at AI, given that evolution is the only known mechanism that leads to what we recognize as intelligence.
  - throwaway90001 9y ago
    > it's given us Einstein, Michelangelo, and conversations on HN two out of three ain't bad
  - dragontamer 9y ago
psyc 9y ago
In industry, yes. In academia, there's computational neuroscience: http://cocosci.mit.edu/people http://cocosci.mit.edu/people https://web.stanford.edu/group/mbc/research.html https://web.stanford.edu/group/mbc/research.html
anakron 9y ago
That's certainly one way to do it. However, we didn't succeed at building modern aircraft or earth moving machinery by building simulations of birds or muscles. There's enough that is unknown out there for a variety of approaches.
white-flame 9y ago
Medical research hasn't cracked step 1 either, at least not to a point of accurate simulation. Besides, if you could simulate a human brain, you will end up with something that needs to sleep, something with limited and unreliable memory, something that gets bored and distracted, something emotionally needy, etc. Then the extending of this chaotic, messy system is wildly unknown even if we could get a piece-for-piece replication to work. Such a thing would be of great benefit to medicine, but not really for AI to even start with until medicine is done reverse engineering it.
- bra-ket 9y ago
  Piece-for-piece replication might not be the right level of abstraction. Blue Brain project is one unfortunate example, on the other hand the current neural nets are stuck with neural model from 1943.
bra-ket 9y ago
Prof. Hinton has an interesting talk about his new 'capsule' model based on psychology of shape perception: https://www.youtube.com/watch?v=rTawFwUvnLE https://www.youtube.com/watch?v=rTawFwUvnLE

PaulHoule 9y ago

Hinton was able to survive 30 years in the academic wilderness. Most academics can't. Thus they work on "safe" projects.

disgruntledphd2 9y ago
It can be done, even today. If you work outside the US and work on cheap things (i.e. no special equipment), especially if you can teach then you can hang around for a long time. I have met a lot of academics like this over the years, but I think your broader point might be that this is not possible today, which I agree with, and which is why I left academia (modulo personal situations).

TimPC 9y ago

Last I checked bridges came before Newtonian Mechanics and it seems strange to argue this wasn't a good thing. Admittedly paper writing wasn't the main mechanism of transmitting knowledge but it's fairly common for human engineering to come before the full theoretical foundations as opposed to after.

joe_the_user 9y ago
It's not that bridges before Newton were bad, it's that Newton gave us the ability to design the strongest possible bridge of a given shape with the materials at hand - using not just calculus but calculus-of-variations, a subject nearly as old as Newtonian mechanics [1]. With this knowledge, what happens when one adds one or two columns to a bridge is now longer "news" the way it might have been before Newtonian mechanics. A stereotypical picture of an engineering approach without scientific knowledge would be a list of ways to do stuff combined with hints about how to vary the approach per-situation. It requires lots memorizing, trial-and-error and experts that often can't fully explain their reasoning. It's easy to believe bridge-building before Newton was like this though I'm not an expert. Present day AI sounds a lot like from what I've read (though I'm not an expert here either). Edit: And yes, one could argue that the progress Newton ushered in merely replaced one list of models with a higher, more general list of models - yes, but that is how progress gone so far. [1] https://en.wikipedia.org/wiki/Calculus_of_variations https://en.wikipedia.org/wiki/Calculus_of_variations
- openasocket 9y ago
  I completely agree with your assessment, but the problem is a bit worse in my opinion. We already have a pretty firm grasp of how different ML systems learn and converge towards a solution in the average case. It's not that we need to understand our neural networks better, it's that we need to understand our problem domain better. We can't determine how well some ML architecture will perform at an object recognition problem without some math describing object recognition. This makes things a lot more complicated, because it means we have to do a lot more work to understand every single application where we want to use ML. And, of course, if we had some really good mathematical framework for describing and reasoning about object recognition, we probably wouldn't need to turn to ML to solve it ;)

aub3bhat 9y ago

Humanity used fire for a long time before combustion was understood. Even today Anesthesia is not well understood at biological/physiological level that has not stopped its safe use and innovation through Clinical Trials. Maybe competitions and empiricism are the best approaches to building intelligent systems. Why get caught up in Physics/Math envy?

AlexCoventry 9y ago
Hinton isn't saying "Let's stop using fire," but "Let's understand the principles behind fire so we can use them in more sophisticated, informed and powerful ways."
- aub3bhat 9y ago
  The ML community did take the Theory approach trying to prove bounds, SLT/SRM, PAC, etc. and that was an excercise in futility. While I don't deny that there is value to looking under the hood but for a long while the community abandoned any empirical results that didn't fit their paradigm. Between rigorously validating their methods and writing yet another 4 page long proof. A lot of researchers would prefer latter, effectively locking out empirical approaches from most dissemination venues and eventually funding.
shimon 9y ago
Because an improved theoretical understanding of how complex systems work can be incredibly valuable. Anesthesia is a good example - we get a lot of value from it, yes, but it would be way better if we could tailor dosages to individuals based on an understanding of how that individual will experience pain. There would be fewer severe complications, but also maybe you could wake up refreshed an hour after surgery instead of in a stupor. If this could work with computer-trained models, that would be incredible too. What could a great speech understanding system teach us about language? What tricks from a facial-expression classifier could help autistic kids understand their friends?
ballenf 9y ago
I took his point not to criticize those early stages, but simply to acknowledge them as such. Early fire users could not have built a rocket no matter how many experiments they performed until they understood combustion (and some other sciences). In AI, we're not building rockets yet, but we have some really awesome and really powerful bonfires or whatever. At least that's how I understood his point. (And in anesthesia, when we do understand those things, we may very well look on our use today as barbaric or dangerous.)

sgt101 9y ago

"engineering before physics" is exactly wrong. No one did Engineering before a sophisticated understanding of Physics was achieved. They built bridges and towers, Engineering enables statements to be made about the performance of machines and buildings; it will survive a wind like x, you can do n cycles, do not load the wings in this way.

empath75 9y ago
I guess the Romans didn’t have any engineers building siege engines and fortifications, then.
- sgt101 9y ago
  Here is the test. Take the best Roman engineer. Translate a first year engineering paper on structures into Latin. Ask Roman to sit said paper. What will happen and why? The Roman chap will look very confused and will make statements (in Latin) about how stupid this stuff is and how it has nothing to do with proper engineering. The Roman will score 0. The why is that the understanding of structures and materials in the ancient world was artizanal, based on trade knowledge (often secret and hard to reproduce) and not systematic, based on the scientific method and inspectable or testable. Currently we accept that knives, cabinets and sheds may be built or made using artisanal knowledge, we do not accept that apartment blocks, aircraft or automobiles are built this way. Society insists that these are built using systematic knowledge because otherwise they sometimes fall down or crash. The systematic approach to aircraft is the best example - think how much civil air traffic there is now, and how rare air crashes are. The issues of subsonic flight have been systematically accounted for, right up to the point where we now see 1:2,000,000 crashes per flight. Mechanical, aeronautical and civil engineering proceed in this way. Issues are discovered with mechanisms or structures or materials, these are characterized with scientific investigation, the characterizations lead to constraints and parameters that are required to be accounted for in future designs and old designs are re-evaluated in the light of the new knowledge. Stating that you will build a new building in a certain way because domes are strong and concrete is strong would not cut the mustard in the modern world... The parthenon has stood for 2000 years, but how many similar structures collapsed after a few months?

markan 9y ago

Great quote from Hinton. The biggest deficiency in AI is that we still don't have artificial systems which simulate human thought with any fidelity. Sooner or later that's bound to become a focus of attention.

nesyt 9y ago
Why bother simulating human thought? It's not the only road to Rome.
- Demiurge 9y ago
  It's probably more of a map than a road.

make3 9y ago

except that this has really only been going on for five years, which is nothing in the scale of human history or even of human rational thought. Some record number of people/scientists are working on getting the physics level understanding to happen, with crazy record breaking year after year quantity of people publishing and attending scientific conferences that fill up in like two days now. it is happening, and will happen even more in depth as time goes on

indescions_2017 9y ago

OTOH, we are merely at circa Year Five into deep reinforcement learning research. It started as a cluster of 16M CPUs having taught itself to recognize a cat 95% of the time after training on 1B google images. And we are now at One-Shot Imitation Learning, "a general system that can turn any demonstrations into robust policies that can accomplish an overwhelming variety of tasks". One Shot Imitation Learning https://arxiv.org/abs/1703.07326 https://arxiv.org/abs/1703.07326

graycat 9y ago
It's easy to recognize a cat 95% of the time. I can write a program in 30 seconds that will recognize a cat 95% of the time. No, wait, this just in! My program will recognize a cat 100% of the time! The program has just one line: Print "It's a cat!"
- graycat 9y ago
  Tutorial: So, with that program, whenever the picture is a cat, the program DOES recognize it. So the program DOES recognize a cat 100% of the time. The OP only claimed 95% of the time. Uh, we need TWO (2), that's TWO numbers: conditional probability of recognizing a cat when there is one (detection rate) conditional probability of claiming there is a cat when there isn't one. The second is the false alarm rate or the conditional probability of a false alarm or the conditional probability of Type I error or the significance level of the test or the p-value, the most heavily used quantity in all of statistics. One minus the detection rate is the conditional probability of Type II error. Typically we can adjust the false alarm rate, and, if we are willing to accept a higher false alarm rate, then we can get a higher detection rate. With my little program, the false alarm rate is also 100%. So, as a detector, my little program is worthless. But the program does have a 100% detection rate, and that's 5% better than the OP claimed. If focus ONLY on detection rate, that is, recognizing a cat when there is one, then it's easy to get a 100% detection rate with just a trivial test -- just say everything is a cat as I did. What's tricky is to have the detection rate high and the false alarm rate low. The best way to do that is in the classic Neyman-Pearson lemma. A good proof is possible using the Hahn decomposition from the Radon-Nikodym theorem in measure theory with the famous proof by von Neumann in W. Rudin, Real and Complex Analysis. My little program was correct and not a joke. Again, to evaluate a detector, need TWO, that's two, or 1 + 1 = 2 numbers. What about a detector that is overall 95% correct? That's easy, too: Just show my detector cats 95% of the time. If we are to be good at computer science, data science, ML/AI, and dip our toes into a huge ocean of beautifully done applied math, then we need to understand Type I and Type II errors. Sorry 'bout that. Are we learning yet?

justonepost 9y ago

The fundamental problem with AI is the high dimensionality of the solution space. We simply can't understand why the brains we are building can think better than us. We can build smarter brains only by trial and error - at least until error outsmarts us, reproduces and takes over. Kind of like having kids.

deleted 9y ago

[deleted]

olalonde 9y ago

Hinton's quote is taken a bit out of context though. I just watched his interview on Andrew Ng's "Neural Networks and Deep Learning" class on Coursera and he seemed convinced that the next "breakthrough" will come from (a variant on) neural networks.

meheleventyone 9y ago
Right but to extend the bridge analogy what's interesting isn't the materials (neural nets) as such but the structure and why that structure works.

zby 9y ago

But maybe there are no universal laws that govern AI like physics governs bridges? AI is something that finds universal laws in stuff - there is no meta level over this - all the meta is AI itself.

zwischenzug 9y ago

I feel like Hofstadter was one of those people thinking really deeply about AI. Anyone who doesn't know what I'm talking about should read 'Goedel, Escher, Bach', or 'Fluid Analogies'. I haven't read them in a long while, but I'm sure they're going to be relevant for decades, because they deal with the fundamental challenge of what it means to think. Backpropagation may be part of the puzzle, but the brain (and intelligence) is so much more than that.

markan 9y ago
I second this recommendation! Here's some more reading for anyone interested in Hofstadter: http://www.popularmechanics.com/science/a3278/why-watson-and-siri-are-not-real-ai-16477207/ http://www.popularmechanics.com/science/a3278/why-watson-and... http://www.basicai.org/blog/hofstadter-2017-09-25.html http://www.basicai.org/blog/hofstadter-2017-09-25.html