4 ms·

> 3. Avoid Machine Learning Agreed this usually makes sense. My only qualm is that I've seen many systems developed by teams without any ML experience develop

> 3. Avoid Machine Learning Agreed this usually makes sense. My only qualm is that I've seen many systems developed by teams without any ML experience develop tools that (1) throw away or don't maintain data integrity and (2) cannot rank or handle uncertainty in the underlying data. These systems often bake in heuristics that cannot be validated and it's very difficult to change them later. Users feel like they are losing something if you migrate from a tool that automatically chooses the best option for them to one where they need to choose among 5 alternatives, for example, even if that "best option" was actually just noise. I think the spirit of "avoid machine learning" is great, but I do think having some forethought about how ML might integrate into the system later on is pretty important.

srcreigh 6y ago
What are some resources to learn more about this? This is not "How to do ML". It's a very much more interesting question "How to build products that can be extended with ML later." I wouldn't even know how to Google for this! Very interesting. Thank you so much.
- twelfthnight 6y ago
  I'm not aware of any resources for that specific question. The closest resource I can think of would be "Designing Data Intensive Applications" which is great for designing systems with data integrity that will lend themselves to ML later on. My next recommendation would actually be to work on a Kaggle project (esp the toy starter project https://www.kaggle.com/c/titanic https://www.kaggle.com/c/titanic). Looking at Kaggle notebooks, sklearn documentation, etc is really valuable for understanding how to pose problems in a way solvable with ML.
- tzsdata 6y ago
  The latest thinking around 'data science as a product' seems to take this approach. See Building Machine Learning Powered Applications for one resource https://www.oreilly.com/library/view/building-machine-learning/9781492045106/ https://www.oreilly.com/library/view/building-machine-learni...
  - twelfthnight 6y ago
    Great reference. Thanks.
lumost 6y ago
I think the problems you mention start off a bit of a rabbit hole trap which most ML teams fall into. Effectively, many problems do not have formal "correct" solutions which can easily be applied. One can spend an unbounded amount of time maintaining data-integrity or improving uncertainty handling. As you add more people familiar with this depth to the team, the "best/correct approach" becomes harder to achieve. In the end, a naive estimate of uncertainty is likely about as valuable to the customer as a formal estimate in most problem domains. "Avoid ML" can easily be code for avoid ambiguous research oriented tasks while building an industrial project, or alternately "don't trust that a magic algorithm will solve your customer problem".

srcreigh 6y ago

What are some resources to learn more about this? This is not "How to do ML". It's a very much more interesting question "How to build products that can be extended with ML later." I wouldn't even know how to Google for this! Very interesting. Thank you so much.

twelfthnight 6y ago
I'm not aware of any resources for that specific question. The closest resource I can think of would be "Designing Data Intensive Applications" which is great for designing systems with data integrity that will lend themselves to ML later on. My next recommendation would actually be to work on a Kaggle project (esp the toy starter project https://www.kaggle.com/c/titanic https://www.kaggle.com/c/titanic). Looking at Kaggle notebooks, sklearn documentation, etc is really valuable for understanding how to pose problems in a way solvable with ML.
tzsdata 6y ago
The latest thinking around 'data science as a product' seems to take this approach. See Building Machine Learning Powered Applications for one resource https://www.oreilly.com/library/view/building-machine-learning/9781492045106/ https://www.oreilly.com/library/view/building-machine-learni...
- twelfthnight 6y ago
  Great reference. Thanks.

lumost 6y ago

I think the problems you mention start off a bit of a rabbit hole trap which most ML teams fall into. Effectively, many problems do not have formal "correct" solutions which can easily be applied. One can spend an unbounded amount of time maintaining data-integrity or improving uncertainty handling. As you add more people familiar with this depth to the team, the "best/correct approach" becomes harder to achieve. In the end, a naive estimate of uncertainty is likely about as valuable to the customer as a formal estimate in most problem domains. "Avoid ML" can easily be code for avoid ambiguous research oriented tasks while building an industrial project, or alternately "don't trust that a magic algorithm will solve your customer problem".