5 ms·

To offhand dismiss ML is also a cardinal sin. Control/treatment groups can show unambiguously when ML outperforms expert hand-crafted rules, pure random decisi

To offhand dismiss ML is also a cardinal sin. Control/treatment groups can show unambiguously when ML outperforms expert hand-crafted rules, pure random decisions or a simple model. The point is to measure, and not go 100% all in with one approach, but try many things and measure. I've done some process optimization with black-box methods, simple models, and SQL using domain expertise. In business you typically have budget and time constraints, so you go for the simplest and quickest solution first, show unambiguously that it works better, and then ask for more time and budget to build something more fancy. I ask myself "if this was my business, and my money, would I spend it doing this", if the answer is no, then you probably shouldn't.

mjburgess 5y ago
The thing is, it's almost always clear when ML will outperform and when it wont. It isn't magic. ML systems are just compressed aggregations of their input datasets. The question is then, (1) do we have datasets that are highly representative of the solutions to our problems? and (2) are our current systems sensitive to the relevant variations in these datasets? If (1) is NO, then ML is impossible. If (2) is YES, then it's unlikely to provide a big ROI.
- antupis 5y ago
  I would add (3) can we leverage existing models.
- HelloNurse 5y ago
  When ML replaces human decisions or very strict old software, there's also a more fundamental problem: are we enabling new mistakes that weren't possible before? How catastrophic? For example, processing images according to some trained model instead of fixed rules and formulas introduces the risk of mismatched models (e.g. landscape photographs treated as line art from anime). Cases like self-driving cars not seeing obstacles are more obvious and more tragic.
- kumarvvr 5y ago
  >ML systems are just compressed aggregations of their input datasets I like to think of them as forgiving sieves of patterns in data. Overfitting a sieve will exclude a large number of almost positive cases, loose fitting will include a large number of mostly negative cases. And there is always a danger of falling into a local minima and not being able to come out of it.

mjburgess 5y ago

The thing is, it's almost always clear when ML will outperform and when it wont. It isn't magic. ML systems are just compressed aggregations of their input datasets. The question is then, (1) do we have datasets that are highly representative of the solutions to our problems? and (2) are our current systems sensitive to the relevant variations in these datasets? If (1) is NO, then ML is impossible. If (2) is YES, then it's unlikely to provide a big ROI.

antupis 5y ago
I would add (3) can we leverage existing models.
HelloNurse 5y ago
When ML replaces human decisions or very strict old software, there's also a more fundamental problem: are we enabling new mistakes that weren't possible before? How catastrophic? For example, processing images according to some trained model instead of fixed rules and formulas introduces the risk of mismatched models (e.g. landscape photographs treated as line art from anime). Cases like self-driving cars not seeing obstacles are more obvious and more tragic.
kumarvvr 5y ago
>ML systems are just compressed aggregations of their input datasets I like to think of them as forgiving sieves of patterns in data. Overfitting a sieve will exclude a large number of almost positive cases, loose fitting will include a large number of mostly negative cases. And there is always a danger of falling into a local minima and not being able to come out of it.