4 ms·

You have hit upon the dark secret of the industry. ML is a solution looking for a problem. It’s nowhere near as bad as blockchain mind, but ML and Kubernetes bo

by gaius 8y ago

You have hit upon the dark secret of the industry. ML is a solution looking for a problem. It’s nowhere near as bad as blockchain mind, but ML and Kubernetes both have far more people eager to do them than there is actual work... Having said that there must be a beancounter in your organisation whose job is making forecasts. That person wouldn’t hesitate to lay you off if the company hits a rough patch. I think you get my meaning.

curun1r 8y ago
In so many ways, I disagree. There's an entire classes of algorithms that we've almost been conditioned as developers to avoid that are amenable to ML techniques. Whether it's ranking/filtering sets of data, where the best we've previously done is taking a SWAG at the characteristics of the data should be valued most highly or parsing unstructured data, learning to solve these classes of problems with ML expands our repertoire of what we can do as developers. That doesn't mean that there aren't people applying ML techniques to problems that can easily be solved in other ways, but that happens with every hyped technology. And for developers learning these techniques, there's going to be a certain amount of applying them more widely than is justified until they learn them well enough to improve their judgment. That's okay and all part of the learning (no pun intended) process. As far as the original poster is concerned, since he works at a shipping company, a problem as simple as "find the address on a webpage" could be a neat proof of concept to show his bosses. Imagine if they could offer customers the ability to "ship" to a URL. That kind of thing might be doable with regexes and such if you constrain the problem to only addresses in the US/Canada, but worldwide address formats are so diverse that ML really helps.
- gaius 8y ago
  True story, 10 years ago I was doing linear regression on infra data to predict capacity issues. My system knew when it had made a prediction and when capacity increased so based on the lag it could tell you the exact date to place an order so more disks could be added to the array before a threshold was breached. And whatever else we were monitoring too, batch job completion times vs number of CPUs, the whole works. PCA to figure out what was relevant. Clever stuff you might think. In reality no one cared. We went on placing a disk order whenever the first dumb nagios alert paged someone. We bought more CPUs when the moaning from the analysts reached the CEO and he in turn moaned to our manager. It was fun to do and I learnt alot but realistically the impact on the business was minimal...
screye 8y ago
> ML is a solution looking for a problem I don't think it is true. ML is legit. The dark secret is that a lot of it isn't new. It is only now, that the data, compute and hype have caught up to how useful good-old applied statistics is. Now a days, a lot of older areas have been repacked into a fancier term of ML and AI. This goes hand-in-hand with some legitimately awesome cutting edge research with wide spread uses (Smart phone cameras, Self driving, Language translators, Massive social media knowledge graphs). Places such as MIT and CMU would not have started undergrad programs in AI, unless they though it was here to stay.
- sidlls 8y ago
  ML in the hands of an individual with relevant educational credentials (e.g. statistics, science) is legit.
- humanrebar 8y ago
  ML is great if you already have scale, data quality, and problem sets that match ML approaches. But often the bulk of the work is getting or cleaning data to make ML work in the first place. And even after that, the results need to be better or cheaper than having an employee personally look at the same data sometimes (i.e. a low tech expert system). If you have a thousand emails and you want an emotion score assigned to the contents, it's often cheaper to use temp workers or something like Mechanical Turk. It's also worth noting that many interesting automation tasks don't have room for 99% confidence intervals. It's fine if Google flips it's search result rankings rarely. It's not OK to only miscalculate 1% of account balances. And predicting which balances are bad might be useful, but the bulk of the work is the boring enterprise work of getting the concerns in front of a human or something. That's why ML seems like a solution in search of a problem. Unless your problem is getting people to click on ads.

by gaius 8y ago

curun1r 8y ago

In so many ways, I disagree. There's an entire classes of algorithms that we've almost been conditioned as developers to avoid that are amenable to ML techniques. Whether it's ranking/filtering sets of data, where the best we've previously done is taking a SWAG at the characteristics of the data should be valued most highly or parsing unstructured data, learning to solve these classes of problems with ML expands our repertoire of what we can do as developers. That doesn't mean that there aren't people applying ML techniques to problems that can easily be solved in other ways, but that happens with every hyped technology. And for developers learning these techniques, there's going to be a certain amount of applying them more widely than is justified until they learn them well enough to improve their judgment. That's okay and all part of the learning (no pun intended) process. As far as the original poster is concerned, since he works at a shipping company, a problem as simple as "find the address on a webpage" could be a neat proof of concept to show his bosses. Imagine if they could offer customers the ability to "ship" to a URL. That kind of thing might be doable with regexes and such if you constrain the problem to only addresses in the US/Canada, but worldwide address formats are so diverse that ML really helps.

gaius 8y ago
True story, 10 years ago I was doing linear regression on infra data to predict capacity issues. My system knew when it had made a prediction and when capacity increased so based on the lag it could tell you the exact date to place an order so more disks could be added to the array before a threshold was breached. And whatever else we were monitoring too, batch job completion times vs number of CPUs, the whole works. PCA to figure out what was relevant. Clever stuff you might think. In reality no one cared. We went on placing a disk order whenever the first dumb nagios alert paged someone. We bought more CPUs when the moaning from the analysts reached the CEO and he in turn moaned to our manager. It was fun to do and I learnt alot but realistically the impact on the business was minimal...

screye 8y ago

> ML is a solution looking for a problem I don't think it is true. ML is legit. The dark secret is that a lot of it isn't new. It is only now, that the data, compute and hype have caught up to how useful good-old applied statistics is. Now a days, a lot of older areas have been repacked into a fancier term of ML and AI. This goes hand-in-hand with some legitimately awesome cutting edge research with wide spread uses (Smart phone cameras, Self driving, Language translators, Massive social media knowledge graphs). Places such as MIT and CMU would not have started undergrad programs in AI, unless they though it was here to stay.

sidlls 8y ago
ML in the hands of an individual with relevant educational credentials (e.g. statistics, science) is legit.
humanrebar 8y ago
ML is great if you already have scale, data quality, and problem sets that match ML approaches. But often the bulk of the work is getting or cleaning data to make ML work in the first place. And even after that, the results need to be better or cheaper than having an employee personally look at the same data sometimes (i.e. a low tech expert system). If you have a thousand emails and you want an emotion score assigned to the contents, it's often cheaper to use temp workers or something like Mechanical Turk. It's also worth noting that many interesting automation tasks don't have room for 99% confidence intervals. It's fine if Google flips it's search result rankings rarely. It's not OK to only miscalculate 1% of account balances. And predicting which balances are bad might be useful, but the bulk of the work is the boring enterprise work of getting the concerns in front of a human or something. That's why ML seems like a solution in search of a problem. Unless your problem is getting people to click on ads.