4 ms·

From the RAND report "First, industry stakeholders often misunderstand — or miscommunicate — what problem needs to be solved using AI." From personal experienc

From the RAND report "First, industry stakeholders often misunderstand — or miscommunicate — what problem needs to be solved using AI." From personal experience this seems like it holds for most data-products, and doubly so for basically any statistical model. As a data scientist, it seems like my domain partners' vision for my contribution very often goes something like: 0. It would be great if we were omniscient 1. Here's some data we have related to a problem we'd like to be omniscient about 2. Please fit a model to it 3. ???? 4. Profit Data scientists and ML engineers need to be aggressive at early planning stages to actually determine what impact the requested model/data product will have. They need to be ready for the model to be wrong, and need to deeply internalize the concept of error bars, and how errors relate to their use-case. But so often 'the business stuff' gets left to the domain people due to organizational politics and people not wanting to get fired. I think the most successful AI orgs will be the ones that can most effectively close the gap between people who can build/manage models, and the people who understand the problem space. Treating AI/ML tools as simple plug and play solutions I think will lead to lots of expensive failures.

victor9000 2y ago
Except there is no winning move as an IC in pushing back against a half baked product definition that lacks business rigor. I pushed back in my org against features whose unit economics didn't add up, and I was labeled not a team player, leading to negative professional development. One year later, the entire ML org was laid off because investors lost confidence in our ability to produce a sustainable business model. There is no fix as an IC for unsophisticated product and business leadership.
- tech_ken 2y ago
  Right, I’m saying that if you’re looking at a roadmap and it’s vague then you need to give feedback and walk. Businesses are failing because of what you’re describing, hopefully the survivors have figured out more effective management strategies.
  - jononor 2y ago
    Why walk as an IC? Without seriou equity, it makes sense to cash in the checks and line up the next gig?
    - tech_ken 2y ago
      Idk man I'm just responding here, fit it into your own situation as appropriate
iknownthing 2y ago
Completely agree. The vast majority of the failed projects fail at the planning stage. To put it simply, if the cost of a misprediction is high then it is usually not going to work because all models make mispredictions. This is something that can be identified in the planning stage i.e. it's usually completely avoidable. Also, of the 20% of projects that succeed, I wonder how many of them actually needed ML.
- steveBK123 2y ago
  > The vast majority of the failed projects fail at the planning stage. This is my experience in 20 years of SWE as well. The biggest project failure debacles were obvious from the get-go to all the senior ICs on the team. Generally managers were pushing them for "reasons", and in some cases even wink-wink about the fact they too didn't believe in the project but.. "reasons".
- pseudosavant 2y ago
  This is my experience too. I find that most "regular" people think 95% is basically 100%, and not "it fails 1 out of 20 times consistently". Even 99% isn't good enough in many cases and should not be considering infallible.
- Yizahi 2y ago
  Also in a case when misprediction is cheap, but makes the whole system unrealiable in such mode of work. E.g. my company tries (too late) to ride the hype and created a halfbaked tool for internal use to classify test results. Since the accuracy of neural networks is never 100% it simply doesn't matter as it doesn't save any time, all test results needs to be verified manually anyway. But several people are busy full time working on it, reporting some results, some amazing performance metrics and so on. And are very visibly upset when we push back or plainly say that the tool is worthless. :)

victor9000 2y ago

Except there is no winning move as an IC in pushing back against a half baked product definition that lacks business rigor. I pushed back in my org against features whose unit economics didn't add up, and I was labeled not a team player, leading to negative professional development. One year later, the entire ML org was laid off because investors lost confidence in our ability to produce a sustainable business model. There is no fix as an IC for unsophisticated product and business leadership.

tech_ken 2y ago
Right, I’m saying that if you’re looking at a roadmap and it’s vague then you need to give feedback and walk. Businesses are failing because of what you’re describing, hopefully the survivors have figured out more effective management strategies.
- jononor 2y ago
  Why walk as an IC? Without seriou equity, it makes sense to cash in the checks and line up the next gig?
  - tech_ken 2y ago
    Idk man I'm just responding here, fit it into your own situation as appropriate

iknownthing 2y ago

Completely agree. The vast majority of the failed projects fail at the planning stage. To put it simply, if the cost of a misprediction is high then it is usually not going to work because all models make mispredictions. This is something that can be identified in the planning stage i.e. it's usually completely avoidable. Also, of the 20% of projects that succeed, I wonder how many of them actually needed ML.

steveBK123 2y ago
> The vast majority of the failed projects fail at the planning stage. This is my experience in 20 years of SWE as well. The biggest project failure debacles were obvious from the get-go to all the senior ICs on the team. Generally managers were pushing them for "reasons", and in some cases even wink-wink about the fact they too didn't believe in the project but.. "reasons".
pseudosavant 2y ago
This is my experience too. I find that most "regular" people think 95% is basically 100%, and not "it fails 1 out of 20 times consistently". Even 99% isn't good enough in many cases and should not be considering infallible.
Yizahi 2y ago
Also in a case when misprediction is cheap, but makes the whole system unrealiable in such mode of work. E.g. my company tries (too late) to ride the hype and created a halfbaked tool for internal use to classify test results. Since the accuracy of neural networks is never 100% it simply doesn't matter as it doesn't save any time, all test results needs to be verified manually anyway. But several people are busy full time working on it, reporting some results, some amazing performance metrics and so on. And are very visibly upset when we push back or plainly say that the tool is worthless. :)