4 ms·

For anyone that knows anything about research and development, a 20% hit rate is actually quite high.

tech_ken 2y ago
From the report 80% is "twice the rate of failure for information technology projects that do not involve AI" (https://www.rand.org/pubs/research_reports/RRA2680-1.html https://www.rand.org/pubs/research_reports/RRA2680-1.html) so seems that a 20% hit-rate isn't actually that great. Possibly it's a quirk of how they're normalizing the success/failure count?
- dwallin 2y ago
  Most of those non-AI projects are likely using well-established practices and technologies. From that perspective a 20% success rate seems pretty good to me.
  - tech_ken 2y ago
    Sure that makes sense, overall "success of a technology product" seems like a very fuzzy thing to try and measure so I imagine one could spin the numbers basically however you want
nerdjon 2y ago
> research and development. I think that is the question, how much legitimate R&D is really going on here vs trying to shove an LLM into some random hardware and ship it? Or shove an LLM in some app trying to solve a problem that it isn't capable of. No doubt that creating these models are hard, having the data is hard. But how much of the AI startups is actually that vs just shoving the OpenAI API in something.
janalsncm 2y ago
I think we’d need to dig into the 80% that failed and what kind of “AI project” they were. Is this really R&D? Or were you trying to insert AI into something that didn’t need it, and failed because no one is using your expensive and annoying “chat with us” popup that your VP insisted would keep your company competitive?
Yizahi 2y ago
It's not a hit rate I think, it's a not yet failed rate. A lot of these were started just recently. And a lot is deep in red but is supported by the VC IV line. And by supported I mean totally. If a Word competitor needs to cut costs it theoretically can scale down and survive. If a neural network startup needs to cut costs it has to shut down operations, due to incredible price to train models and run inference.

From the report 80% is "twice the rate of failure for information technology projects that do not involve AI" (https://www.rand.org/pubs/research_reports/RRA2680-1.html https://www.rand.org/pubs/research_reports/RRA2680-1.html) so seems that a 20% hit-rate isn't actually that great. Possibly it's a quirk of how they're normalizing the success/failure count?

dwallin 2y ago
Most of those non-AI projects are likely using well-established practices and technologies. From that perspective a 20% success rate seems pretty good to me.
- tech_ken 2y ago
  Sure that makes sense, overall "success of a technology product" seems like a very fuzzy thing to try and measure so I imagine one could spin the numbers basically however you want

nerdjon 2y ago

> research and development. I think that is the question, how much legitimate R&D is really going on here vs trying to shove an LLM into some random hardware and ship it? Or shove an LLM in some app trying to solve a problem that it isn't capable of. No doubt that creating these models are hard, having the data is hard. But how much of the AI startups is actually that vs just shoving the OpenAI API in something.

janalsncm 2y ago

I think we’d need to dig into the 80% that failed and what kind of “AI project” they were. Is this really R&D? Or were you trying to insert AI into something that didn’t need it, and failed because no one is using your expensive and annoying “chat with us” popup that your VP insisted would keep your company competitive?

Yizahi 2y ago

It's not a hit rate I think, it's a not yet failed rate. A lot of these were started just recently. And a lot is deep in red but is supported by the VC IV line. And by supported I mean totally. If a Word competitor needs to cut costs it theoretically can scale down and survive. If a neural network startup needs to cut costs it has to shut down operations, due to incredible price to train models and run inference.