11 ms·

Microsoft Challenges Google’s Artificial Brain With ‘Project Adam’

98 points by bra-ket 12y ago

rbanffy 12y ago
I like to see someone taking a contrarian approach and exploring an unexplored implementation space.
Locke1689 12y ago
This is because, when each machine updates the master server, the update tends to be additive. Hmm, reminds me of GFS (append-only file system). It seems like I managed to really short my machine learning education during my degrees. Anyone have any recommendations for good introductory resources for practical machine learning design and implementation?
- fenomas 12y ago
  I did Stanford's online ML course (taught by Andrew Ng, quoted in the article), and heartily recommend it. I found it to be a nice balance of theory and practice - it's also not a bad way to learn Octave (the GNU equivalent of matlab) if one wants to do that. http://online.stanford.edu/course/machine-learning http://online.stanford.edu/course/machine-learning (I don't know if there's a way to just access the materials and videos though, rather than signing up for the course.)
- dekhn 12y ago
  You're conflating the append operation with (in their example) addition, which is completely different. Append makes a list longer. Addition combines two elements in a reduction. Addition is commutative. And the other issue here is that learning is tolerant of a small amount of noise. You don't want your filesystem of record to be.
  - Locke1689 12y ago
    I'm not actually. They both share the commonality of non-destructive updates, which seems to be a recently pervasive technique of performing unsynchronized parallel work. I do grant that the techniques are not identical, though. Certainly GFS has at least minor locking so that it can grab the current high water mark.
pradn 12y ago
The Hogwild approach mentioned is extremely simple. When you do incremental updates, don't hold locks. You'll rarely conflict, because the number of data points to update is usually much higher than the number of threads doing the updates. And even if you do conflict, the effects don't affect accuracy in any noticeable way. (I found this empirically.) It looks like Microsoft is applying this technique not to stochastic gradient descent, but to building neural networks, in this case.
- liuliu 12y ago
  Actually, I don't quite understand their method. At the end of the day, it is not a gigantic shared memory, as in Hogwild! paper conveniently describes. You need to synchronize slave's model with master model, in model parallelism case, you need to send responses to another machine for further processes. This sounds extremely complicated actually to implement in modern HPC environment (assuming MPI as the main implementation technique). Sorry if this sounds a bit defensive than it is meant to be. I just think that there are much more to the implementation than the simple Hogwild! approach (probably some clever pipelining to cover the communication / computation cost as the Hogwild! approach relaxed consistency of the model). It seems that this year's ImageNet classification competition result will be more interesting than what I expected.
chriskanan 12y ago
This is a duplicate of this earlier post: https://news.ycombinator.com/item?id=8031546 https://news.ycombinator.com/item?id=8031546
thefreeman 12y ago
I'm only about... 50% serious when I say this. But all of these huge tech companies getting into serious AI and learning really makes me worry that Singularity is coming way sooner then I would have hoped. Like in the next 10 years.
- rimantas 12y ago
  Our brains are so far away from understanding themselves that those 10 years won't bring much, let alone the singularity.
- msutherl 12y ago
  I honestly don't see any evidence to suggest this.
- shock-value 12y ago
  Most of the recent advances don't really constitute what you are probably thinking of as "serious" AI. What's happening now is mostly just scientists putting newly available massively parallel processing units (GPUs etc.) to work at sort of "brute forcing" the problem of AI. You could technically say it's a step above brute-forcing in that, after randomly assigning neural weights, gradient descent and other tricks are used to more quickly settle into good solutions. But the science is simply nowhere near the level of the human brain, and the current state of the art is still only "proficient" at relatively simple pattern matching, rather than coming up with decisions that actually influence the state of the world that it is perceiving. The deep architectural insights that will likely be needed before computers can approach the level of human intelligence still haven't been made yet, it seems to me based on what I know of the current research.
  - nl 12y ago
    Yeah, this is almost completely wrong - or at least outdated thinking. While it's entirely likely that better AI algorithms will become available, the progress being made by "brute forcing" problems using lots of data and lots of computing power is much more than was expected 10 years ago. For example, Facebook's facial recognition software now correctly recognizes faces 97.25% of the time. Humans do 97.53%[1]. I'd encourage you to read the Norvig et. al paper: "The Unreasonable Effectiveness of Data" [1] http://www.technologyreview.com/news/525586/facebook-creates-software-that-matches-faces-almost-as-well-as-you-do/ http://www.technologyreview.com/news/525586/facebook-creates... [2] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35179.pdf https://static.googleusercontent.com/media/research.google.c...
dghf 12y ago
"Project Adam"? Second Impact, anyone?
- astrobe_ 12y ago
  Also: Buffy the vampire slayer (season 4). It's a bit strange people make biblic references for such projects when the "don't you play God" theme (itself a variation on the myth of the golem) is, volens nolens, part of popular culture. Or maybe the team deliberately did it, as a kind of memo?
  - TeMPOraL 12y ago
    I sometimes wonder how people can reconcile the "don't play God" meme with the concept of humans being "created in the image of God".
  - sp332 12y ago
    It was probably deliberate. It reminds me of Dresden Codak's "Caveman Science Fiction" http://dresdencodak.com/2009/09/22/caveman-science-fiction/ http://dresdencodak.com/2009/09/22/caveman-science-fiction/
nwatson 12y ago
I was surprised the article didn't mention IBM's Watson project (the Jeopardy game-show winning "AI" bot) and I assumed IBM would be farther along in actual commercialization in "outside" domains. Google and others appear to focus on using AI techniques for their core business (though Google's "core" is very diffuse). This article (January 2014) implies IBM's having trouble with applying Watson for profit: http://www.zdnet.com/ibm-forms-watson-business-group-will-commercialization-follow-7000024929/ http://www.zdnet.com/ibm-forms-watson-business-group-will-co...
- m_ke 12y ago
  That's because IBM Watson is a gimmick that doesn't do much besides search.
  - mkautzm 12y ago
    I thought the end goal with Watson was to develop something that can effectively diagnose medical issues or some such?
    - m_ke 12y ago
      That's how they're selling it but at the end of the day it's just a search engine with some NLP up front to parse the query. The reason why I said that it was a gimmick is because they still partly rely on a rule based system and hand engineered features.
  - cscurmudgeon 12y ago
    Deep learning is totally not a gimmick and does everything under the sun including winning Jeopardy! contests against humans and bringing the Holy Singularity in a few weeks.
icantthinkofone 12y ago
Despite the hoopla, isn't this project in Microsoft's lab while Google's is in current use online?

98 points by bra-ket 12y ago

I like to see someone taking a contrarian approach and exploring an unexplored implementation space.

This is because, when each machine updates the master server, the update tends to be additive. Hmm, reminds me of GFS (append-only file system). It seems like I managed to really short my machine learning education during my degrees. Anyone have any recommendations for good introductory resources for practical machine learning design and implementation?

fenomas 12y ago
I did Stanford's online ML course (taught by Andrew Ng, quoted in the article), and heartily recommend it. I found it to be a nice balance of theory and practice - it's also not a bad way to learn Octave (the GNU equivalent of matlab) if one wants to do that. http://online.stanford.edu/course/machine-learning http://online.stanford.edu/course/machine-learning (I don't know if there's a way to just access the materials and videos though, rather than signing up for the course.)
dekhn 12y ago
You're conflating the append operation with (in their example) addition, which is completely different. Append makes a list longer. Addition combines two elements in a reduction. Addition is commutative. And the other issue here is that learning is tolerant of a small amount of noise. You don't want your filesystem of record to be.
- Locke1689 12y ago
  I'm not actually. They both share the commonality of non-destructive updates, which seems to be a recently pervasive technique of performing unsynchronized parallel work. I do grant that the techniques are not identical, though. Certainly GFS has at least minor locking so that it can grab the current high water mark.

pradn 12y ago

The Hogwild approach mentioned is extremely simple. When you do incremental updates, don't hold locks. You'll rarely conflict, because the number of data points to update is usually much higher than the number of threads doing the updates. And even if you do conflict, the effects don't affect accuracy in any noticeable way. (I found this empirically.) It looks like Microsoft is applying this technique not to stochastic gradient descent, but to building neural networks, in this case.

liuliu 12y ago
Actually, I don't quite understand their method. At the end of the day, it is not a gigantic shared memory, as in Hogwild! paper conveniently describes. You need to synchronize slave's model with master model, in model parallelism case, you need to send responses to another machine for further processes. This sounds extremely complicated actually to implement in modern HPC environment (assuming MPI as the main implementation technique). Sorry if this sounds a bit defensive than it is meant to be. I just think that there are much more to the implementation than the simple Hogwild! approach (probably some clever pipelining to cover the communication / computation cost as the Hogwild! approach relaxed consistency of the model). It seems that this year's ImageNet classification competition result will be more interesting than what I expected.

chriskanan 12y ago

This is a duplicate of this earlier post: https://news.ycombinator.com/item?id=8031546 https://news.ycombinator.com/item?id=8031546

thefreeman 12y ago

I'm only about... 50% serious when I say this. But all of these huge tech companies getting into serious AI and learning really makes me worry that Singularity is coming way sooner then I would have hoped. Like in the next 10 years.

rimantas 12y ago
Our brains are so far away from understanding themselves that those 10 years won't bring much, let alone the singularity.
msutherl 12y ago
I honestly don't see any evidence to suggest this.
shock-value 12y ago
Most of the recent advances don't really constitute what you are probably thinking of as "serious" AI. What's happening now is mostly just scientists putting newly available massively parallel processing units (GPUs etc.) to work at sort of "brute forcing" the problem of AI. You could technically say it's a step above brute-forcing in that, after randomly assigning neural weights, gradient descent and other tricks are used to more quickly settle into good solutions. But the science is simply nowhere near the level of the human brain, and the current state of the art is still only "proficient" at relatively simple pattern matching, rather than coming up with decisions that actually influence the state of the world that it is perceiving. The deep architectural insights that will likely be needed before computers can approach the level of human intelligence still haven't been made yet, it seems to me based on what I know of the current research.
- nl 12y ago
  Yeah, this is almost completely wrong - or at least outdated thinking. While it's entirely likely that better AI algorithms will become available, the progress being made by "brute forcing" problems using lots of data and lots of computing power is much more than was expected 10 years ago. For example, Facebook's facial recognition software now correctly recognizes faces 97.25% of the time. Humans do 97.53%[1]. I'd encourage you to read the Norvig et. al paper: "The Unreasonable Effectiveness of Data" [1] http://www.technologyreview.com/news/525586/facebook-creates-software-that-matches-faces-almost-as-well-as-you-do/ http://www.technologyreview.com/news/525586/facebook-creates... [2] https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/35179.pdf https://static.googleusercontent.com/media/research.google.c...

dghf 12y ago

"Project Adam"? Second Impact, anyone?

astrobe_ 12y ago
Also: Buffy the vampire slayer (season 4). It's a bit strange people make biblic references for such projects when the "don't you play God" theme (itself a variation on the myth of the golem) is, volens nolens, part of popular culture. Or maybe the team deliberately did it, as a kind of memo?
- TeMPOraL 12y ago
  I sometimes wonder how people can reconcile the "don't play God" meme with the concept of humans being "created in the image of God".
- sp332 12y ago
  It was probably deliberate. It reminds me of Dresden Codak's "Caveman Science Fiction" http://dresdencodak.com/2009/09/22/caveman-science-fiction/ http://dresdencodak.com/2009/09/22/caveman-science-fiction/

nwatson 12y ago

I was surprised the article didn't mention IBM's Watson project (the Jeopardy game-show winning "AI" bot) and I assumed IBM would be farther along in actual commercialization in "outside" domains. Google and others appear to focus on using AI techniques for their core business (though Google's "core" is very diffuse). This article (January 2014) implies IBM's having trouble with applying Watson for profit: http://www.zdnet.com/ibm-forms-watson-business-group-will-commercialization-follow-7000024929/ http://www.zdnet.com/ibm-forms-watson-business-group-will-co...

m_ke 12y ago
That's because IBM Watson is a gimmick that doesn't do much besides search.
- mkautzm 12y ago
  I thought the end goal with Watson was to develop something that can effectively diagnose medical issues or some such?
  - m_ke 12y ago
    That's how they're selling it but at the end of the day it's just a search engine with some NLP up front to parse the query. The reason why I said that it was a gimmick is because they still partly rely on a rule based system and hand engineered features.
- cscurmudgeon 12y ago
  Deep learning is totally not a gimmick and does everything under the sun including winning Jeopardy! contests against humans and bringing the Holy Singularity in a few weeks.

icantthinkofone 12y ago

Despite the hoopla, isn't this project in Microsoft's lab while Google's is in current use online?