4 ms·

I concur. ML isn't programming per se; it is experimental problem-solving with a particular dataset and algorithm. Your result may/not work well, may/not genera

I concur. ML isn't programming per se; it is experimental problem-solving with a particular dataset and algorithm. Your result may/not work well, may/not generalise, and will almost undoubtedly not contribute anything new to any discipline, even to ML. When all ML work is done we'll have great pattern recognizers but nothing remotely akin to thought. And we won't understand how they work or the best way to build the next one. It isn't AI, although it is a part of AI, just as the visual system is part of AI. I was reading Domingos' "The Master Algorithm" several days ago and a mathematician inquired about the book. He knew a group of ML developers. His opinion was that "ML doesn't look very interesting: all you do is play with the parameters, turn the knobs, and/or change the model until something works. There's no real progress there; nothing substantial." Rather than sending a batallion of bright developers into the ML swamp where they will largely be frustrated, learn little and contribute less, I'd be tempted to guide them into other fields.

visarga 10y ago
> And we won't understand how they work Is this a critique of the human mind or a praise of AI? > When all ML work is done we'll have great pattern recognizers but nothing remotely akin to thought Maybe our brains too are nothing but pattern recognizers. Maybe they are nothing but chemical reactions, or energy fields. But being reductionist about AI won't help us understand it either.
zby 10y ago
What we need is models that are more retrospectable - so that you can find the rules that it learned. Most of the time they will be too complex for any human understanding - but from time to time we'll find something interesting, something that we can build other things upon. I have never used neural nets etc - but with simplified bayes spam filters this was possible and quite useful. I used to check which words were pushing a text into one or other category and which did not (when they should).
ioeu 10y ago
What fields would you guide them to instead? I may be one of the developers you speak of (with academic aspirations), presently considering my path forward. I'm sceptical if going down the ML swamp is the best way forward.
tdullien 10y ago
I am a mathematician by trade, and was doing development along with other stuff (reverse engineering and security work, first in my own company, then at Google). So ... 1) I think working knowledge of ML is extremely useful to many developers, and generally under-taught in universities. See the old Joel article which mentions "Google uses Bayesian filtering like MS uses the IF statement" http://www.joelonsoftware.com/items/2005/10/17.html http://www.joelonsoftware.com/items/2005/10/17.html). A well-rounded developer should know the basics (Logistic regression, SVMs, know some things about CDNNs etc.), it will make him much more adept at problem-solving. I suspect Google's internal push to get people up to speed is not to turn them all into ML researchers, but rather to make sure that everybody "knows the basics well enough". So I think it is useful to teach developers about the things ML has to offer. 2) Mathematically, it seems that in ML the "engineering" side has run far ahead of the theory side. The sudden breakthrough in the mid-2000s is IMO still not fully understood - and parts of it may have been very accidental. Initially, it was thought that pre-training was the big breakthrough, but it is quite unclear what the big breakthrough was. It could be that simply the increase of data / compute sizes and the switch to minibatch-SGD explains why modern DNNs generalize well (interesting paper on the topic: https://arxiv.org/abs/1509.01240 https://arxiv.org/abs/1509.01240). There is a lot of good mathematics to be written, but I am not sure whether the folks at Google will write it - given the incentive structures (performance reviews, impact statements) it is unlikely that somebody gets promoted for "cleaning up the theory". 3) From a development perspective: There are a ton of interesting engineering problems underneath the progress in ML. If you look at Jeff Dean, he is a superstar engineer, not necessarily a mathematician, and a lot of the progress the Google Brain team made were engineering advances to scale / distribute etc. - so by training the engineers in ML, you also get to have better infrastructure over time. So I don't think they are sending "developers into ML swamps"; I think they are trying to reach the point where "Google uses DNNs like MS uses IF". Cheers, Thomas
- randcraw 10y ago
  I don't think your points are invalid, but I think you overvalue the data that's available and relevant to most programming tasks. And without novel data, ML can offer little novel value. Google, Facebook, M$ Research, and perhaps Yahoo are extreme outliers. They have zottabytes of broad unstructured text data, so they mine it. Everybody else has megabytes of narrow structured data, most of it commercial transations of their products. That stuff has already been effectively mined by traditional basic OLAP methods. Most/all of the value has been extracted. Mainstream software apps have yet to show the value of using ML. Such apps have access to very limited data of very narrow relevance. The utility of ML in such domains isn't new; it's classic optimization. Or it's bayesian anticipation. But it's not a game changer. Frankly, the use of ML in most mainstream apps is more likely to add distraction and annoyance as the computer mispredicts your intent -- like Microsoft Bob did. Maybe "life in the cloud" will create new opportunities for smarter software. But I definitely don't want free apps making their own decisions when to notify me. I guarantee that will get old immediately. So how will this work? Frankly, I can't guess. Like Apple's iAds, programming ML into the mainstream or cloud sounds like an idea that will serve the software / cloud vendor far better than the user.

visarga 10y ago

> And we won't understand how they work Is this a critique of the human mind or a praise of AI? > When all ML work is done we'll have great pattern recognizers but nothing remotely akin to thought Maybe our brains too are nothing but pattern recognizers. Maybe they are nothing but chemical reactions, or energy fields. But being reductionist about AI won't help us understand it either.

zby 10y ago

What we need is models that are more retrospectable - so that you can find the rules that it learned. Most of the time they will be too complex for any human understanding - but from time to time we'll find something interesting, something that we can build other things upon. I have never used neural nets etc - but with simplified bayes spam filters this was possible and quite useful. I used to check which words were pushing a text into one or other category and which did not (when they should).

ioeu 10y ago

What fields would you guide them to instead? I may be one of the developers you speak of (with academic aspirations), presently considering my path forward. I'm sceptical if going down the ML swamp is the best way forward.

tdullien 10y ago

I am a mathematician by trade, and was doing development along with other stuff (reverse engineering and security work, first in my own company, then at Google). So ... 1) I think working knowledge of ML is extremely useful to many developers, and generally under-taught in universities. See the old Joel article which mentions "Google uses Bayesian filtering like MS uses the IF statement" http://www.joelonsoftware.com/items/2005/10/17.html http://www.joelonsoftware.com/items/2005/10/17.html). A well-rounded developer should know the basics (Logistic regression, SVMs, know some things about CDNNs etc.), it will make him much more adept at problem-solving. I suspect Google's internal push to get people up to speed is not to turn them all into ML researchers, but rather to make sure that everybody "knows the basics well enough". So I think it is useful to teach developers about the things ML has to offer. 2) Mathematically, it seems that in ML the "engineering" side has run far ahead of the theory side. The sudden breakthrough in the mid-2000s is IMO still not fully understood - and parts of it may have been very accidental. Initially, it was thought that pre-training was the big breakthrough, but it is quite unclear what the big breakthrough was. It could be that simply the increase of data / compute sizes and the switch to minibatch-SGD explains why modern DNNs generalize well (interesting paper on the topic: https://arxiv.org/abs/1509.01240 https://arxiv.org/abs/1509.01240). There is a lot of good mathematics to be written, but I am not sure whether the folks at Google will write it - given the incentive structures (performance reviews, impact statements) it is unlikely that somebody gets promoted for "cleaning up the theory". 3) From a development perspective: There are a ton of interesting engineering problems underneath the progress in ML. If you look at Jeff Dean, he is a superstar engineer, not necessarily a mathematician, and a lot of the progress the Google Brain team made were engineering advances to scale / distribute etc. - so by training the engineers in ML, you also get to have better infrastructure over time. So I don't think they are sending "developers into ML swamps"; I think they are trying to reach the point where "Google uses DNNs like MS uses IF". Cheers, Thomas

randcraw 10y ago
I don't think your points are invalid, but I think you overvalue the data that's available and relevant to most programming tasks. And without novel data, ML can offer little novel value. Google, Facebook, M$ Research, and perhaps Yahoo are extreme outliers. They have zottabytes of broad unstructured text data, so they mine it. Everybody else has megabytes of narrow structured data, most of it commercial transations of their products. That stuff has already been effectively mined by traditional basic OLAP methods. Most/all of the value has been extracted. Mainstream software apps have yet to show the value of using ML. Such apps have access to very limited data of very narrow relevance. The utility of ML in such domains isn't new; it's classic optimization. Or it's bayesian anticipation. But it's not a game changer. Frankly, the use of ML in most mainstream apps is more likely to add distraction and annoyance as the computer mispredicts your intent -- like Microsoft Bob did. Maybe "life in the cloud" will create new opportunities for smarter software. But I definitely don't want free apps making their own decisions when to notify me. I guarantee that will get old immediately. So how will this work? Frankly, I can't guess. Like Apple's iAds, programming ML into the mainstream or cloud sounds like an idea that will serve the software / cloud vendor far better than the user.