6 ms·

DeepMind to Work Directly With Google’s Search Team

66 points by jamesjyu 13y ago

nl 13y ago
It's difficult not to get caught up in the excitement around Deep Learning. The unsupervised nature of it at its best really does seem to approach human levels of learning (if not intelligence). Also, I thought this was hilarious: In a 2011 interview that predated DeepMind, co-founder Shane Legg said he gave only a 50 percent chance that human level-machine intelligence would exist by 2028. ONLY?!? By 2028? That is a pretty radical prediction. Indeed, Legg says there is a 10% chance of human level intelligence by 2018, and 90% by 2050[1]. Legg has multiple papers published in the field of measuring machine intelligence, so I guess he has a pretty good view of the field. If that 2028 prediction is even close to correct then I think setting up an ethics board now probably is the correct thing to do. (the notion that DeepMind asked that Google create an internal ethics board as a condition of the acquisition, as reported by The Information, had some AI researchers griping) Edit: Legg's posting on his blog in 2011 deserves to be read: I’ve decided to once again leave my prediction for when human level AGI will arrive unchanged. That is, I give it a log-normal distribution with a mean of 2028 and a mode of 2025, under the assumption that nothing crazy happens like a nuclear war. I’d also like to add to this prediction that I expect to see an impressive proto-AGI within the next 8 years. By this I mean a system with basic vision, basic sound processing, basic movement control, and basic language abilities, with all of these things being essentially learnt rather than preprogrammed. It will also be able to solve a range of simple problems, including novel ones. [1] http://lesswrong.com/r/discussion/lw/691/shane_legg_on_risks_from_ai/ http://lesswrong.com/r/discussion/lw/691/shane_legg_on_risks... [2] http://www.vetta.org/2011/12/goodbye-2011-hello-2012/ http://www.vetta.org/2011/12/goodbye-2011-hello-2012/
- PhearTheCeal 13y ago
  Perhaps his predictions are correct. Or perhaps his predictions are similar to those made by a professional software developer when asked "How long will it take?" and they respond "Oh, about five minutes." Then it ends up taking days.
  - nl 13y ago
    Indeed. Or perhaps they are like a doomsday cultist's predictions of the end of the earth. Nevertheless, I think predictions like this from people actively working in the field are something that shouldn't be dismissed out of hand. I know 10 years ago I didn't think I'd see self-driving cars in my lifetime.
- sp332 13y ago
  Our ethics are far behind our tech and have been for a long time. Maybe we should make ethics courses mandatory for science degrees, because most tech people just subscribe to a worldview they pick in high school or college, and never do any critical thinking about their own projects.
  - nl 13y ago
    I think most science degrees have some units relating to ethics. I know my computer science degree had something called something like "computing in society" that kind of tackled it. But the questions regarding - say - what to do with predictions from a super-intelligent sentient AI weren't really addressed. I think one problem is that anyone attempting to address those questions in a serious fashion risks being branded a crackpot.
    - endtime 13y ago
      The MIRI folks (intelligence.org) are trying to address exactly that, though I'm sure some consider them crackpots.
      - nl 13y ago
        Yes, they are the only group I'm aware of that is addressing this. I'd note that they publicise their ethics work a lot less then they used to (eg, nothing on the http://intelligence.org/research/ http://intelligence.org/research/ page). http://intelligence.org/files/MachineEthicsSuperintelligence.pdf http://intelligence.org/files/MachineEthicsSuperintelligence... is good though.
      - eli_gottlieb 13y ago
        Their former incarnation as the Singularity Institute were crackpots. Luke Muehlhauser has done a hell of a lot to make them a more serious organization. The Future of Humanity Institute at Oxford University also work on this stuff, with plenty of mainstream credibility.
        13y ago
- kamaal 13y ago
  You are neglecting what singularists call 'The law of accelerating returns'(LOAR). What that means is the growth in areas like these is not linear but exponential. The more you grow, the more that growth helps in further growth. And that proceeds in a never ending fashion.
  - jmmcd 13y ago
    All of this is correct except for "never ending". There are no never-ending exponentials.
- camillomiller 13y ago
  Considering that Google's ethical statement still amounts to the "don't be evil" non-sense, yeah, I guess an ethics board is needed even today, without any actual AI implication. We all think of incredibly futuristic sentient robots, but what we should really be worried about are the implications this acquisition will have on Google Search, a tool that is already used by millions of people. A tool that millions of people already superimpose to "the Internet" itself. A tool that already has a strong and unmitigated influence on people choices. That may not be completely clear in the U.S. Market, but take a stroll in peripheral markets and, to quip the famous commercial, you'll see why 2014 already looks a lot like "1984".
- eli_gottlieb 13y ago
  Notably, he said that he expected impressive proto-AGI that would be able to play a few video games within eight years. He then started accomplishing it within three, if the "Playing Atari via Deep Learning" paper is to be believed.
zvanness 13y ago
Here's what I said when the story first came out: "I'm not so sure their technology is as futuristic as everyone thinks it is. If I had to take an educated guess, I would say it's some powerful AI that makes their knowledge graph smarter. Currently Google's Knowledge Graph uses more structured data sets and depends on a mechanism like this: http://www.zachvanness.com/nanobird_relevancy_engine.pdf http://www.zachvanness.com/nanobird_relevancy_engine.pdf But the real challenge is to make the knowledge graph update in real time and take meaning from something as unstructured as a blog post or an email. And to do something like that requires some really unique AI." This is most likely what they are going to be implementing: http://www.zachvanness.com/nanobird_relevancy_engine_vision.pdf http://www.zachvanness.com/nanobird_relevancy_engine_vision....
- hnriot 13y ago
  why repeat an empty comment? It's clear that you don't really have any knowledge of, or insight into state of the art machine learning, so I don't see the need to paste in what you've already said. It's very hard for those not familiar with the field to see when something is novel, much of ML looks "obvious" to a typical developer. As for Google's KG, updating the model in real time isn't really necessary, knowledge doesn't typically advance emails or blog posts at a time. They already have that in their search, knowledge is something that is mutually agreed upon and takes some amount of consensus. Using the structured sources (like freebase) is much better than trying to do dependency parsing on some blog post and try to make sense of it. Powerset tried that and it didn't work then, and won't work for at least a few years. ML advances not in leaps and bounds but rather by little pieces of the puzzle falling into place. The biggest thing to happen to the ML world isn't algorithms or fast computers, but the abundance of data available. Like pagerank, the more data it has, the better it works, the same is true of ML systems. Google has more data than god, and it is through this they will push the frontiers of AI - along with IBM and the others working in this field.
  - nl 13y ago
    Using the structured sources (like freebase) is much better than trying to do dependency parsing on some blog post and try to make sense of it. Powerset tried that and it didn't work then, and won't work for at least a few years. Evidence extraction from unstructured data is a pretty active area of research. Clearly structured data is "better", but the issue with many structured sources is they don't accurately represent knowledge. For example, answering "What is the capital of Israel" and/or "what is the capital of Palestine" really depends on both who is asking and who is answering. Unstructured data is generally good at showing controversy like that.
sjg007 13y ago
What I am really curious about is the difference between Deep Learning using RBMs, PCA, Factor Analysis, Bayesian graphical models. All of them seem to treat a numerical problem in different ways. I'm particularly interested in the identifiability of the equations (and introduce some constraints to do so).. or else you have multiple (infinite) solutions. As far as I can tell, the deep learning is basically a multi layer RBM with greedy optimization at each level. Maybe that is good enough for text summarization or robotics, but in a mathematical sense depending on your starting vectors you can arrive at different local minima. In physical dynamical systems we are typically interested in the global solution from which we do further analyses to prove stability etc... Bayesian networks have loopy belief propagation, and of course PCA/SVD require strict linear independence.
- dave_sullivan 13y ago
  There's a bit of confusion around this, but "deep learning" is nearly interchangeable with "neural networks". So PCA, for instance, isn't deep learning (which definitely doesn't make it bad, it's just definitionally not deep learning ie not a neural network). Now, as far as the difference between PCA and eg an autoencoder (which is similar to an RBM, but different--both RBMs and autoencoders are "deep neural networks")--well, PCA just learns a linear mapping. You can stack PCAs on top of each other I guess, but it is functionally equivalent to a single layer. So it makes certain assumptions about the behavior of the data--specifically, that it's linear. A neural network layer calculation looks like this: nonlin(Wx + b) A deep neural network forward pass looks something like this: W3 .* nonlin(W2 .* nonlin(W1 .* INPUT+b1)+ b2) + b3) Nonlin can mean a variety of things. Usually tanh, sigmoid, or max(0,x). If you remove those nonlinearities, it basically becomes like PCA, like a model that learns a linear mapping. When you have those nonlinearities, you get a non-linear mapping which, theoretically, lets you address a wider variety of problems. But yeah, if you want to learn about deep learning, just learn about neural networks. It's the same thing. Also, re: your point on local minima--on the upper end of experiments that have been done, you've got over a billion parameters. 10 million parameters would be on the relatively small side. I think it's reasonable for there to be a variety of possible solutions to any given problem, and the whole idea is that we're using models that are very flexible due to their high capacity. The problem historically has been using the capacity effectively. However, regularization methods like dropout help a lot, and I've found it's easier to think of neural networks as "feature learners". Each one of those neurons is theoretically a feature its learning. And even still, if you look at the features that are learned by convolutional nets (which are easier to visualize due to their being used on vision problems), they tend to arrive at similar features even though it starts from fairly different initializations.
deleted 13y ago
[deleted]
gulbrandr 13y ago
Google will soon be able to predict your searches before you. It would require a mix of this kind of technology and all the data it already has, and will have, on its users. This acquisition is just another step in this direction.

66 points by jamesjyu 13y ago

nl 13y ago

It's difficult not to get caught up in the excitement around Deep Learning. The unsupervised nature of it at its best really does seem to approach human levels of learning (if not intelligence). Also, I thought this was hilarious: In a 2011 interview that predated DeepMind, co-founder Shane Legg said he gave only a 50 percent chance that human level-machine intelligence would exist by 2028. ONLY?!? By 2028? That is a pretty radical prediction. Indeed, Legg says there is a 10% chance of human level intelligence by 2018, and 90% by 2050[1]. Legg has multiple papers published in the field of measuring machine intelligence, so I guess he has a pretty good view of the field. If that 2028 prediction is even close to correct then I think setting up an ethics board now probably is the correct thing to do. (the notion that DeepMind asked that Google create an internal ethics board as a condition of the acquisition, as reported by The Information, had some AI researchers griping) Edit: Legg's posting on his blog in 2011 deserves to be read: I’ve decided to once again leave my prediction for when human level AGI will arrive unchanged. That is, I give it a log-normal distribution with a mean of 2028 and a mode of 2025, under the assumption that nothing crazy happens like a nuclear war. I’d also like to add to this prediction that I expect to see an impressive proto-AGI within the next 8 years. By this I mean a system with basic vision, basic sound processing, basic movement control, and basic language abilities, with all of these things being essentially learnt rather than preprogrammed. It will also be able to solve a range of simple problems, including novel ones. [1] http://lesswrong.com/r/discussion/lw/691/shane_legg_on_risks_from_ai/ http://lesswrong.com/r/discussion/lw/691/shane_legg_on_risks... [2] http://www.vetta.org/2011/12/goodbye-2011-hello-2012/ http://www.vetta.org/2011/12/goodbye-2011-hello-2012/

PhearTheCeal 13y ago
Perhaps his predictions are correct. Or perhaps his predictions are similar to those made by a professional software developer when asked "How long will it take?" and they respond "Oh, about five minutes." Then it ends up taking days.
- nl 13y ago
  Indeed. Or perhaps they are like a doomsday cultist's predictions of the end of the earth. Nevertheless, I think predictions like this from people actively working in the field are something that shouldn't be dismissed out of hand. I know 10 years ago I didn't think I'd see self-driving cars in my lifetime.
sp332 13y ago
Our ethics are far behind our tech and have been for a long time. Maybe we should make ethics courses mandatory for science degrees, because most tech people just subscribe to a worldview they pick in high school or college, and never do any critical thinking about their own projects.
- nl 13y ago
  I think most science degrees have some units relating to ethics. I know my computer science degree had something called something like "computing in society" that kind of tackled it. But the questions regarding - say - what to do with predictions from a super-intelligent sentient AI weren't really addressed. I think one problem is that anyone attempting to address those questions in a serious fashion risks being branded a crackpot.
  - endtime 13y ago
    The MIRI folks (intelligence.org) are trying to address exactly that, though I'm sure some consider them crackpots.
    - nl 13y ago
      Yes, they are the only group I'm aware of that is addressing this. I'd note that they publicise their ethics work a lot less then they used to (eg, nothing on the http://intelligence.org/research/ http://intelligence.org/research/ page). http://intelligence.org/files/MachineEthicsSuperintelligence.pdf http://intelligence.org/files/MachineEthicsSuperintelligence... is good though.
    - eli_gottlieb 13y ago
      Their former incarnation as the Singularity Institute were crackpots. Luke Muehlhauser has done a hell of a lot to make them a more serious organization. The Future of Humanity Institute at Oxford University also work on this stuff, with plenty of mainstream credibility.
      - 13y ago
kamaal 13y ago
You are neglecting what singularists call 'The law of accelerating returns'(LOAR). What that means is the growth in areas like these is not linear but exponential. The more you grow, the more that growth helps in further growth. And that proceeds in a never ending fashion.
- jmmcd 13y ago
  All of this is correct except for "never ending". There are no never-ending exponentials.
camillomiller 13y ago
Considering that Google's ethical statement still amounts to the "don't be evil" non-sense, yeah, I guess an ethics board is needed even today, without any actual AI implication. We all think of incredibly futuristic sentient robots, but what we should really be worried about are the implications this acquisition will have on Google Search, a tool that is already used by millions of people. A tool that millions of people already superimpose to "the Internet" itself. A tool that already has a strong and unmitigated influence on people choices. That may not be completely clear in the U.S. Market, but take a stroll in peripheral markets and, to quip the famous commercial, you'll see why 2014 already looks a lot like "1984".
eli_gottlieb 13y ago
Notably, he said that he expected impressive proto-AGI that would be able to play a few video games within eight years. He then started accomplishing it within three, if the "Playing Atari via Deep Learning" paper is to be believed.

zvanness 13y ago

Here's what I said when the story first came out: "I'm not so sure their technology is as futuristic as everyone thinks it is. If I had to take an educated guess, I would say it's some powerful AI that makes their knowledge graph smarter. Currently Google's Knowledge Graph uses more structured data sets and depends on a mechanism like this: http://www.zachvanness.com/nanobird_relevancy_engine.pdf http://www.zachvanness.com/nanobird_relevancy_engine.pdf But the real challenge is to make the knowledge graph update in real time and take meaning from something as unstructured as a blog post or an email. And to do something like that requires some really unique AI." This is most likely what they are going to be implementing: http://www.zachvanness.com/nanobird_relevancy_engine_vision.pdf http://www.zachvanness.com/nanobird_relevancy_engine_vision....

hnriot 13y ago
why repeat an empty comment? It's clear that you don't really have any knowledge of, or insight into state of the art machine learning, so I don't see the need to paste in what you've already said. It's very hard for those not familiar with the field to see when something is novel, much of ML looks "obvious" to a typical developer. As for Google's KG, updating the model in real time isn't really necessary, knowledge doesn't typically advance emails or blog posts at a time. They already have that in their search, knowledge is something that is mutually agreed upon and takes some amount of consensus. Using the structured sources (like freebase) is much better than trying to do dependency parsing on some blog post and try to make sense of it. Powerset tried that and it didn't work then, and won't work for at least a few years. ML advances not in leaps and bounds but rather by little pieces of the puzzle falling into place. The biggest thing to happen to the ML world isn't algorithms or fast computers, but the abundance of data available. Like pagerank, the more data it has, the better it works, the same is true of ML systems. Google has more data than god, and it is through this they will push the frontiers of AI - along with IBM and the others working in this field.
- nl 13y ago
  Using the structured sources (like freebase) is much better than trying to do dependency parsing on some blog post and try to make sense of it. Powerset tried that and it didn't work then, and won't work for at least a few years. Evidence extraction from unstructured data is a pretty active area of research. Clearly structured data is "better", but the issue with many structured sources is they don't accurately represent knowledge. For example, answering "What is the capital of Israel" and/or "what is the capital of Palestine" really depends on both who is asking and who is answering. Unstructured data is generally good at showing controversy like that.

sjg007 13y ago

What I am really curious about is the difference between Deep Learning using RBMs, PCA, Factor Analysis, Bayesian graphical models. All of them seem to treat a numerical problem in different ways. I'm particularly interested in the identifiability of the equations (and introduce some constraints to do so).. or else you have multiple (infinite) solutions. As far as I can tell, the deep learning is basically a multi layer RBM with greedy optimization at each level. Maybe that is good enough for text summarization or robotics, but in a mathematical sense depending on your starting vectors you can arrive at different local minima. In physical dynamical systems we are typically interested in the global solution from which we do further analyses to prove stability etc... Bayesian networks have loopy belief propagation, and of course PCA/SVD require strict linear independence.

dave_sullivan 13y ago
There's a bit of confusion around this, but "deep learning" is nearly interchangeable with "neural networks". So PCA, for instance, isn't deep learning (which definitely doesn't make it bad, it's just definitionally not deep learning ie not a neural network). Now, as far as the difference between PCA and eg an autoencoder (which is similar to an RBM, but different--both RBMs and autoencoders are "deep neural networks")--well, PCA just learns a linear mapping. You can stack PCAs on top of each other I guess, but it is functionally equivalent to a single layer. So it makes certain assumptions about the behavior of the data--specifically, that it's linear. A neural network layer calculation looks like this: nonlin(Wx + b) A deep neural network forward pass looks something like this: W3 .* nonlin(W2 .* nonlin(W1 .* INPUT+b1)+ b2) + b3) Nonlin can mean a variety of things. Usually tanh, sigmoid, or max(0,x). If you remove those nonlinearities, it basically becomes like PCA, like a model that learns a linear mapping. When you have those nonlinearities, you get a non-linear mapping which, theoretically, lets you address a wider variety of problems. But yeah, if you want to learn about deep learning, just learn about neural networks. It's the same thing. Also, re: your point on local minima--on the upper end of experiments that have been done, you've got over a billion parameters. 10 million parameters would be on the relatively small side. I think it's reasonable for there to be a variety of possible solutions to any given problem, and the whole idea is that we're using models that are very flexible due to their high capacity. The problem historically has been using the capacity effectively. However, regularization methods like dropout help a lot, and I've found it's easier to think of neural networks as "feature learners". Each one of those neurons is theoretically a feature its learning. And even still, if you look at the features that are learned by convolutional nets (which are easier to visualize due to their being used on vision problems), they tend to arrive at similar features even though it starts from fairly different initializations.

deleted 13y ago

[deleted]

gulbrandr 13y ago

Google will soon be able to predict your searches before you. It would require a mix of this kind of technology and all the data it already has, and will have, on its users. This acquisition is just another step in this direction.