11 ms·

Inside Google Brain

136 points by albertzeyer 12y ago

thisjepisje 12y ago
I knew this was a Wired article without even looking at the URL.
- arel 12y ago
  I really don't know who buys Wired... despite it being a quality publication on the whole it seems far too dumbed down for real geeks and too niche for the layman. I imagine it ends up on the reception table of lots of startups and desgin agencies who want to some glossy 'we do technology' badge.
- soapdog 12y ago
  me too... argh, I hate those headlines. Its a proof of how much tech and digital culture journalism has fallen. I remember on my pre-broadband days here in Brazil subscribing to Doctor Dobbs Journal to the tune of 25 USD per issue and being happy. Each issue filled with little gems that would advance my knowledge a lot... these days its all those glossy covers with photoshop covers and over the top headlines. :-(
  - icebraining 12y ago
    I don't think tech journalism as really fallen - Wired has always been fluffy, and Dr. Dobbs is still around [1]. I think the change happened more on the shelves, as laymen with an interest in technology replaced geeks as the most profitable group. [1] http://www.drdobbs.com/ http://www.drdobbs.com/
sidcool 12y ago
It seems they missed mentioning Ray Kurzweil, the AI master who's a Director of Engineering at Google.
- eitally 12y ago
  To clarify, Ray is a director, not the director. Google has many engineering directors.
  - sidcool 12y ago
    Corrected it. But I do think he deserves a mention when talking about AI and Google in particular.
- defo_nonconvex 12y ago
  Kurzweil seems more like a master of generating linkbait-like titles. "The singularity is near: When humans transcend biology" "The age of spiritual machines: When computers exceed human intelligence" I'd argue that Stephen Boyd is more of a master of AI than Kurzweil.
- snowwrestler 12y ago
  There's pretty good evidence that the entity we call Ray Kurzweil is just an early prototype of Google Brain built by Jeff Dean.
cliveowen 12y ago
"They’ve also found that the models tend to become more accurate the more data they consume. That may be the next big goal for Google: building AI models that are based on billions of data points, not just millions. " I'm not versed in machine learning, but it looks to me that any model whose output quality is dependent on the quantity of data it ingests is deeply flawed. There's no doubt a bigger number of samples will make the predictions more accurate, but isn't the challenge to develop a system that is as accurate as possible regardless of the number of data points its fed, like the human brain?
- mendicantB 12y ago
  Every model's output quality is dependent on the quantity of data it ingests. Statistics developed as a science because of the need to overcome the weakness of large samples being expensive. Machine learning has taken off as a direct result of the field's ability to take advantage of and get serious performance gains from the massive amounts of data being generated and leveraged recently. Here is the best summation I can reference, and I can tell you from personal experience it is very true: "The accuracy & nature of answers you get on large data sets can be completely different from what you see on small samples. Big data provides a competitive advantage. For the web data sets you describe, it turns out that having 10x the amount of data allows you to automatically discover patterns that would be impossible with smaller samples (think Signal to Noise). The deeper into demographic slices you want to dive, the more data you will need to get the same accuracy." http://www.quora.com/Big-Data/Why-the-current-obsession-with-big-data http://www.quora.com/Big-Data/Why-the-current-obsession-with...
  - cliveowen 12y ago
    I completely agree with this and, as I said, it's obvious that more data produces better predictions, even with simple models. My point is that it looks backwards to me putting effort into finding more and better data (creating a corpus for a given subject is a challenge in itself) instead of trying to come up with a model that infers more and produces better predictions with less data. Once you have such a model then you can surely collect and feed it a lot of data to improve the output, but until then, why even bother?
astazangasta 12y ago
>About a year later, Google had reduced Android’s voice recognition error rate by an astounding 25 percent. lol. This is the grand payoff?
- deong 12y ago
  Considering that for the preceding 25 years or so, progress in the state of the art had been annual reductions of far less than 1%, it's a pretty big deal.
  - astazangasta 12y ago
    From an academic perspective, maybe. In terms of practical use of machine methods, not much. Machine learning is largely hype. I pity the army of PhDs they must have building training sets.
    - sp332 12y ago
      Have you tried Google's voice recognition? It's been fantastic for me.
- sp332 12y ago
  It was already as good as a bunch of PhD's could make it after working on it for years. Removing 25% of the remaining errors after that is a grand payoff!
- DanBC 12y ago
  There's a bunch of stuff that gets optimised. You start with big gains, and those get smaller and smaller until you're spending a lot of effort to get 0.5% increase in performance. See, for example, compressing the ENWIK8 in the Hutter Prize. http://prize.hutter1.net/ http://prize.hutter1.net/ If you can compress this approx 100 MB file to less than approx 16 MB (including the decompressor) you win cash. That website shows small decreases in filesize over a few years. Alexander Rhatushnyak 23.May 2009 6.27 | 1614€ Marcus Hutter Alexander Rhatushnyak 14.May 2007 6.07 | 1732€ Marcus Hutter Alexander Rhatushnyak 25.Sep.2006 5.86 | 3416€ Marcus Hutter Matt Mahoney 24.Mar.2006 5.46 | pre-prize - The compression ratio goes from 5.46 to 5.86 to 6.07 to 6.27.
daviddumenil 12y ago
The original paper being discussed: http://arxiv.org/abs/1312.6082 http://arxiv.org/abs/1312.6082
- rtkwe 12y ago
  Strange they didn't link to google's own publication page: http://research.google.com/pubs/pub42241.html http://research.google.com/pubs/pub42241.html
michaelochurch 12y ago
Google is not really a search company. It's a machine-learning company. It wants to be seen that way, but until it abolishes closed allocation (and maybe it has, but I haven't heard anything to indicate that it has) it will just be an ads company. It's still a pretty good place to work, by industry standards, but the percentage of people who'll get to work on machine learning is very low. Google definitely wants to have the image of being the machine learning company because that's a great way to attract talent (even if that talent is mostly wasted under closed allocation). And if you land in the right place, there is interesting work. The reality most people face, though, is that most people (especially outside of Mt. View) aren't going to get real projects and won't be anywhere near the machine learning work. Google does have a lot of talent and probably would be the undisputed #1 tech company if it implemented open allocation, though.
- programminggeek 12y ago
  Google is an Advertising company and it delivers ads via these platforms like search, mobile apps, etc. It clearly is a tech company, but to pretend it's anything other than an ad delivery machine is sort of ignoring the elephant in the room. Everything goes back to ads. Even Google doesn't bite the hand that feeds it.
  - dsymonds 12y ago
    That's a common trope, but it doesn't make sense. We don't call the New York Times an advertising company simply because the majority of its revenue comes from ads. Why apply the same standard to Google?
woodchuck64 12y ago
Seems inevitable that the company with the world's largest distributed compute engine (estimated at 40 petaflops as of 2012, must be 80 petaflops by now) would eventually get into machine-learning.
api 12y ago
"Google isn't really a search company-- it's a machine learning company." No, it's an advertising company. That's who pays the bills. At the end of the day all this cool tech is to better understand and model human beings in order to better push ads. Sometimes it depresses me that so many of the world's most brilliant minds are working on that, but a generation or two ago they'd all be building doomsday bombs. I guess that's some progress.
- dm2 12y ago
  If there was anything else they could do that was profitable then they would. Do you have any suggestions? The only reason Apple has as much money as it does is because it greatly overcharges customers, doesn't participate in research that benefits society, and takes advantage of it's customers psychological need to have the latest model (even if the improvements are minimal).
  - api 12y ago
    I wasn't even really dissing Google, just pointing out the reality of the world we live in. The fact is that very very few people really care about visionary things. There has to be a way to make it pay, and unfortunately surveillance-based marketing is the only big meal ticket in town for services like Google.
    - dm2 12y ago
      It is true about the ads being their primary source of revenue, I've thought long and hard about the concept but eventually accepted the ads as being necessary to fund the other projects they work on. Their research isn't solely focused on gathering data for ads. If these technologies can be applied to ads then it helps justify large costs because it'll increase revenue, but that is not the sole driving motivator for their moonshots and research. Hopefully the percentage of people who are passionate about futuristic concepts and visionary ideas will rise in the near future. The faster we get to a post-scarcity society the greater chance we, as an intelligent species, will have to survive long-term. Just my 2 cents. I love them as a company and have a huge amount of respect for what they've done and for sticking to their founding principals of transparency and "do no evil" (for the most part, within reason for a company of their size). I know it's pretty subjective and irreverent to bring Apple into any Google debate, the amount of people who put them on a pedestal drives me crazy. Anything I can do to redirect money from going to Apple is a positive thing in my opinion.
roneesh 12y ago
These Wired click-bait headlines are terrible.
- dang 12y ago
  They certainly are. Can anyone suggest a better one here? We want accurate and neutral.
  - dcre 12y ago
    Inside Google Brain?
    - dang 12y ago
      Good suggestion. We changed it. Thanks!
  - michael_nielsen 12y ago
    Could submissions from Wired (& other sites which practice link-baity titles) be automatically put in a moderation queue? They wouldn't appear in the "New" queue until the title had been reviewed by a moderator, and fixed if appropriate. That sort of fix seems to be necessary a large fraction of the time. Regularly having such titles on the front page is a small but real hit to HN's quality.
    - dang 12y ago
      There actually is a status like that, sort of. But Wired isn't currently in that bucket. We may have to put it in there; their titles have gotten noticeably (to me) more sensational lately.
  - ttctciyf 12y ago
    The section title: “Google is not really a search company. It’s a machine-learning company.” might be better, but ... meh, title fussiness. Anyway, it's an ad company!
bcRIPster 12y ago
Anyone else notice the sample street number photos looked a lot like some of the CAPTCHA images going around not too long ago? Hmm...
- VikingCoder 12y ago
  That's old, published and confirmed news: http://techcrunch.com/2012/03/29/google-now-using-recaptcha-to-decode-street-view-addresses/ http://techcrunch.com/2012/03/29/google-now-using-recaptcha-...
gunnario 12y ago
"This form of internal code-sharing has already helped another cutting-edge Google technology called MapReduce catch fire." Map-Reduce is not a Google technology.
- paperwork 12y ago
  I believe it is. Obviously map and reduce operations have existed for a long time, but MapReduce was Google's work, before it was popularize by hadoop and friends. Soon after MapReduce became popular, there was a paper published (by non-googlers) which 'described' the algorithm and poked some fun at the terminology: http://lambda-the-ultimate.org/node/1669 http://lambda-the-ultimate.org/node/1669
  - dekhn 12y ago
    Most people miss this important point: the most important part of MapReduce is the shuffle step, which is a global, partitioned disk-to-disk sort. See the FlumeJava for a bit more discussion on why this is relevant. Everything else about MapReduce is just framework to make programmer's lives easier. BTW, not having strong typing in classic MR is a major pain point. Flume goes a long way to addressing this in a practical way. BTW, this point elucidates the importance of this classic Google Interview question: http://www.glassdoor.com/Interview/Sort-a-million-32-bit-integers-using-only-2MB-of-RAM-QTN_120936.htm http://www.glassdoor.com/Interview/Sort-a-million-32-bit-int...

Inside Google Brain

136 points by albertzeyer 12y ago

thisjepisje 12y ago

I knew this was a Wired article without even looking at the URL.

arel 12y ago
I really don't know who buys Wired... despite it being a quality publication on the whole it seems far too dumbed down for real geeks and too niche for the layman. I imagine it ends up on the reception table of lots of startups and desgin agencies who want to some glossy 'we do technology' badge.
soapdog 12y ago
me too... argh, I hate those headlines. Its a proof of how much tech and digital culture journalism has fallen. I remember on my pre-broadband days here in Brazil subscribing to Doctor Dobbs Journal to the tune of 25 USD per issue and being happy. Each issue filled with little gems that would advance my knowledge a lot... these days its all those glossy covers with photoshop covers and over the top headlines. :-(
- icebraining 12y ago
  I don't think tech journalism as really fallen - Wired has always been fluffy, and Dr. Dobbs is still around [1]. I think the change happened more on the shelves, as laymen with an interest in technology replaced geeks as the most profitable group. [1] http://www.drdobbs.com/ http://www.drdobbs.com/

sidcool 12y ago

It seems they missed mentioning Ray Kurzweil, the AI master who's a Director of Engineering at Google.

eitally 12y ago
To clarify, Ray is a director, not the director. Google has many engineering directors.
- sidcool 12y ago
  Corrected it. But I do think he deserves a mention when talking about AI and Google in particular.
defo_nonconvex 12y ago
Kurzweil seems more like a master of generating linkbait-like titles. "The singularity is near: When humans transcend biology" "The age of spiritual machines: When computers exceed human intelligence" I'd argue that Stephen Boyd is more of a master of AI than Kurzweil.
snowwrestler 12y ago
There's pretty good evidence that the entity we call Ray Kurzweil is just an early prototype of Google Brain built by Jeff Dean.

cliveowen 12y ago

"They’ve also found that the models tend to become more accurate the more data they consume. That may be the next big goal for Google: building AI models that are based on billions of data points, not just millions. " I'm not versed in machine learning, but it looks to me that any model whose output quality is dependent on the quantity of data it ingests is deeply flawed. There's no doubt a bigger number of samples will make the predictions more accurate, but isn't the challenge to develop a system that is as accurate as possible regardless of the number of data points its fed, like the human brain?

mendicantB 12y ago
Every model's output quality is dependent on the quantity of data it ingests. Statistics developed as a science because of the need to overcome the weakness of large samples being expensive. Machine learning has taken off as a direct result of the field's ability to take advantage of and get serious performance gains from the massive amounts of data being generated and leveraged recently. Here is the best summation I can reference, and I can tell you from personal experience it is very true: "The accuracy & nature of answers you get on large data sets can be completely different from what you see on small samples. Big data provides a competitive advantage. For the web data sets you describe, it turns out that having 10x the amount of data allows you to automatically discover patterns that would be impossible with smaller samples (think Signal to Noise). The deeper into demographic slices you want to dive, the more data you will need to get the same accuracy." http://www.quora.com/Big-Data/Why-the-current-obsession-with-big-data http://www.quora.com/Big-Data/Why-the-current-obsession-with...
- cliveowen 12y ago
  I completely agree with this and, as I said, it's obvious that more data produces better predictions, even with simple models. My point is that it looks backwards to me putting effort into finding more and better data (creating a corpus for a given subject is a challenge in itself) instead of trying to come up with a model that infers more and produces better predictions with less data. Once you have such a model then you can surely collect and feed it a lot of data to improve the output, but until then, why even bother?

astazangasta 12y ago

>About a year later, Google had reduced Android’s voice recognition error rate by an astounding 25 percent. lol. This is the grand payoff?

deong 12y ago
Considering that for the preceding 25 years or so, progress in the state of the art had been annual reductions of far less than 1%, it's a pretty big deal.
- astazangasta 12y ago
  From an academic perspective, maybe. In terms of practical use of machine methods, not much. Machine learning is largely hype. I pity the army of PhDs they must have building training sets.
  - sp332 12y ago
    Have you tried Google's voice recognition? It's been fantastic for me.
sp332 12y ago
It was already as good as a bunch of PhD's could make it after working on it for years. Removing 25% of the remaining errors after that is a grand payoff!
DanBC 12y ago
There's a bunch of stuff that gets optimised. You start with big gains, and those get smaller and smaller until you're spending a lot of effort to get 0.5% increase in performance. See, for example, compressing the ENWIK8 in the Hutter Prize. http://prize.hutter1.net/ http://prize.hutter1.net/ If you can compress this approx 100 MB file to less than approx 16 MB (including the decompressor) you win cash. That website shows small decreases in filesize over a few years. Alexander Rhatushnyak 23.May 2009 6.27 | 1614€ Marcus Hutter Alexander Rhatushnyak 14.May 2007 6.07 | 1732€ Marcus Hutter Alexander Rhatushnyak 25.Sep.2006 5.86 | 3416€ Marcus Hutter Matt Mahoney 24.Mar.2006 5.46 | pre-prize - The compression ratio goes from 5.46 to 5.86 to 6.07 to 6.27.

daviddumenil 12y ago

The original paper being discussed: http://arxiv.org/abs/1312.6082 http://arxiv.org/abs/1312.6082

rtkwe 12y ago
Strange they didn't link to google's own publication page: http://research.google.com/pubs/pub42241.html http://research.google.com/pubs/pub42241.html

michaelochurch 12y ago

Google is not really a search company. It's a machine-learning company. It wants to be seen that way, but until it abolishes closed allocation (and maybe it has, but I haven't heard anything to indicate that it has) it will just be an ads company. It's still a pretty good place to work, by industry standards, but the percentage of people who'll get to work on machine learning is very low. Google definitely wants to have the image of being the machine learning company because that's a great way to attract talent (even if that talent is mostly wasted under closed allocation). And if you land in the right place, there is interesting work. The reality most people face, though, is that most people (especially outside of Mt. View) aren't going to get real projects and won't be anywhere near the machine learning work. Google does have a lot of talent and probably would be the undisputed #1 tech company if it implemented open allocation, though.

programminggeek 12y ago
Google is an Advertising company and it delivers ads via these platforms like search, mobile apps, etc. It clearly is a tech company, but to pretend it's anything other than an ad delivery machine is sort of ignoring the elephant in the room. Everything goes back to ads. Even Google doesn't bite the hand that feeds it.
- dsymonds 12y ago
  That's a common trope, but it doesn't make sense. We don't call the New York Times an advertising company simply because the majority of its revenue comes from ads. Why apply the same standard to Google?

woodchuck64 12y ago

Seems inevitable that the company with the world's largest distributed compute engine (estimated at 40 petaflops as of 2012, must be 80 petaflops by now) would eventually get into machine-learning.

api 12y ago

"Google isn't really a search company-- it's a machine learning company." No, it's an advertising company. That's who pays the bills. At the end of the day all this cool tech is to better understand and model human beings in order to better push ads. Sometimes it depresses me that so many of the world's most brilliant minds are working on that, but a generation or two ago they'd all be building doomsday bombs. I guess that's some progress.

dm2 12y ago
If there was anything else they could do that was profitable then they would. Do you have any suggestions? The only reason Apple has as much money as it does is because it greatly overcharges customers, doesn't participate in research that benefits society, and takes advantage of it's customers psychological need to have the latest model (even if the improvements are minimal).
- api 12y ago
  I wasn't even really dissing Google, just pointing out the reality of the world we live in. The fact is that very very few people really care about visionary things. There has to be a way to make it pay, and unfortunately surveillance-based marketing is the only big meal ticket in town for services like Google.
  - dm2 12y ago
    It is true about the ads being their primary source of revenue, I've thought long and hard about the concept but eventually accepted the ads as being necessary to fund the other projects they work on. Their research isn't solely focused on gathering data for ads. If these technologies can be applied to ads then it helps justify large costs because it'll increase revenue, but that is not the sole driving motivator for their moonshots and research. Hopefully the percentage of people who are passionate about futuristic concepts and visionary ideas will rise in the near future. The faster we get to a post-scarcity society the greater chance we, as an intelligent species, will have to survive long-term. Just my 2 cents. I love them as a company and have a huge amount of respect for what they've done and for sticking to their founding principals of transparency and "do no evil" (for the most part, within reason for a company of their size). I know it's pretty subjective and irreverent to bring Apple into any Google debate, the amount of people who put them on a pedestal drives me crazy. Anything I can do to redirect money from going to Apple is a positive thing in my opinion.

roneesh 12y ago

These Wired click-bait headlines are terrible.

dang 12y ago
They certainly are. Can anyone suggest a better one here? We want accurate and neutral.
- dcre 12y ago
  Inside Google Brain?
  - dang 12y ago
    Good suggestion. We changed it. Thanks!
- michael_nielsen 12y ago
  Could submissions from Wired (& other sites which practice link-baity titles) be automatically put in a moderation queue? They wouldn't appear in the "New" queue until the title had been reviewed by a moderator, and fixed if appropriate. That sort of fix seems to be necessary a large fraction of the time. Regularly having such titles on the front page is a small but real hit to HN's quality.
  - dang 12y ago
    There actually is a status like that, sort of. But Wired isn't currently in that bucket. We may have to put it in there; their titles have gotten noticeably (to me) more sensational lately.
- ttctciyf 12y ago
  The section title: “Google is not really a search company. It’s a machine-learning company.” might be better, but ... meh, title fussiness. Anyway, it's an ad company!

bcRIPster 12y ago

Anyone else notice the sample street number photos looked a lot like some of the CAPTCHA images going around not too long ago? Hmm...

VikingCoder 12y ago
That's old, published and confirmed news: http://techcrunch.com/2012/03/29/google-now-using-recaptcha-to-decode-street-view-addresses/ http://techcrunch.com/2012/03/29/google-now-using-recaptcha-...

gunnario 12y ago

"This form of internal code-sharing has already helped another cutting-edge Google technology called MapReduce catch fire." Map-Reduce is not a Google technology.

paperwork 12y ago
I believe it is. Obviously map and reduce operations have existed for a long time, but MapReduce was Google's work, before it was popularize by hadoop and friends. Soon after MapReduce became popular, there was a paper published (by non-googlers) which 'described' the algorithm and poked some fun at the terminology: http://lambda-the-ultimate.org/node/1669 http://lambda-the-ultimate.org/node/1669
- dekhn 12y ago
  Most people miss this important point: the most important part of MapReduce is the shuffle step, which is a global, partitioned disk-to-disk sort. See the FlumeJava for a bit more discussion on why this is relevant. Everything else about MapReduce is just framework to make programmer's lives easier. BTW, not having strong typing in classic MR is a major pain point. Flume goes a long way to addressing this in a practical way. BTW, this point elucidates the importance of this classic Google Interview question: http://www.glassdoor.com/Interview/Sort-a-million-32-bit-integers-using-only-2MB-of-RAM-QTN_120936.htm http://www.glassdoor.com/Interview/Sort-a-million-32-bit-int...