33 ms·

95% of generative AI pilots at companies are failing – MIT report

230 points by amirkabbara 1y ago

https://mlq.ai/media/quarterly_decks/v0.1_State_of_AI_in_Business_2025_Report.pdf https://mlq.ai/media/quarterly_decks/v0.1_State_of_AI_in_Bus...

amirkabbara 1y ago
Why so bad?
- longtimelistnr 1y ago
  Because for the typical office - documents are strewn about on random network drives and are not formatted similarly. This combined with the inability to nail down 100% accuracy on even just internal doc search is just too much to overcome for non-tech industry offices. My office is mind blown if i use Gemini to extract data from a PDF and convert it to an .xlsx or .csv As a technically minded person but not a comp sci guy, refining document search is like staring into a void and every option uses different (confusing) terminology. This makes it extra difficult for me to both do my regular job AND learn the multiple names/ways to do the exact same thing between platforms. The only solution that has any reliability for me so far are Gemini instances where i upload only the files i wish to search and just keep it locked to a few questions per instance before it starts to hallucinate. My attempt at RAG search implementation was a disaster that left me more confused than anything.
  - noddingham 1y ago
    Because you mentioned the use case specifically, I wanted to point you to the fact that Excel has been able to convert images to tables for a while now. Literally screenshot a table from your PDF and it will convert to table. Not trying to diminish any additional capabilities you're getting from Gemini, but this screenshot to table feature has been huge for my finance team. https://support.microsoft.com/en-us/office/insert-data-from-picture-3c1bb58d-2c59-4bc0-b04a-a671a6868fd7 https://support.microsoft.com/en-us/office/insert-data-from-...
  - amirkabbara 1y ago
    try https://www.papr.ai https://www.papr.ai for RAG. built it to solve this problem
- appease7727 1y ago
  Turns out that garbage text has very little intrinsic value
- nathan_compton 1y ago
  I think one reason for this is that LLMs are sort of maximally if accidentally designed to fuck up our brains. Despite all the advancements in the last five years I see them as still, fundamentally, text transformation machines which have only very limited sort of intelligence. Yet because nothing in history has been able to generate language except humans, most of us are not prepared to make rational judgements about their capabilities and those of us that may be also often fail to do so. The fact that we live in an era where tech people have been so investor pilled that overstating the capabilities of technology is basically second nature does not help.
- troupo 1y ago
  It's in the name: generative AIs. There are very few use cases at companies where you need to generate something. You want to work with the company's often very private disparate data (with access controls etc.) You wouldn't even have enough data to train a custom LLM, much less use a generic one.
- ARandumGuy 1y ago
  Any consumer facing AI project has to contend with the fact that GenAI is predominantly associated with "slop." If you're not actively using an AI tool, most of your experience with GenAI is seeing social media or Youtube flooded with low quality AI content, or having to deal with useless AI customer support. This gives the impression that AI is just cheap garbage, and something that should be actively avoided.
- morkalork 1y ago
  In my experience is that LLMs get you 80%of the way to a solution almost immediately but that last 20% when it comes to missing knowledge, data, or accuracy is a complete tar pit and will wreck adoption. Especially since many vendors are selling products that are wrappers and provide generic, non-customised solutions. I hear the same from others doing trials with various AI tools as well.
JohnClark1337 1y ago
[dead]
trenchpilgrim 1y ago
What's the failure rates if technology pilots in general for comparison? For example, I heard that SAP has an 80-90% deployment failure rate back in the day, but don't have a citable source for it.
- kqr 1y ago
  Depends on industry I would think. In my previous industry it was something like 25 %, in my current industry it is closer to 80 %.
- RaftPeople 1y ago
  > I heard that SAP has an 80-90% deployment failure rate Something to keep in mind is that ERP "failure" is frequently defined as went over budget or over time, even if it ultimately completed and provided the desired functionality. It's a much smaller percentage of projects that are either cancelled or went live and significantly did not function as the business needed.
- aprilthird2021 1y ago
  That is not remotely true tbh. The company would have failed long ago if it were
  - mike_hearn 1y ago
    Not if every manufacturing company in the world decided to use your software anyway. ERP rollouts can "fail" for lots of reasons that aren't to do with the software. They are usually business failures. Mostly, companies end up spending so much on trying to endlessly customize it to their idiosyncratic workflows that they exceed their project budgets and abandon the effort. In really bad cases like Birmingham they go live before actually finishing setup, and then lose control of their books and have to resort to hiring people to do the admin manually. There's a saying about SAP: at some point gaining competitive advantage in manufacturing/retail became all about who could make SAP deployment a success. This is no different to many other IT projects, most of them fail too. I think people who have never worked in an enterprise context don't realize that; it's not like working in the tech sector. In the tech industry if a project fails, it's probably because it was too ambitious and the tech itself just didn't work well. Or it was a startup whose tech worked, but they couldn't find PMF. But in normal, mature, profitable non-tech businesses a staggering number of business automation projects just fail for social or business reasons. AI deployments inside companies are going to be like that. The tech works. The business side problems are where the failures are going to happen. Reasons will include: • Not really knowing what they want the AI to do. • No way to measure improved productivity, so no way to decide if the API spend is worth it. • Concluding the only way to get a return is entirely replace people with AI and then having to re-hire them because the AI can't handle the last 5% of the work. • Non-tech executives doing deals to use models or tech stacks that aren't the right kind or good enough. etc
etothet 1y ago
https://archive.is/bdi7b https://archive.is/bdi7b
zahlman 1y ago
Am I the only one who looked at this shortened headline and wondered why anyone is allowing AIs to fly airplanes?
- madcaptenor 1y ago
  No. I also thought that even a 95% success rate wouldn't be good enough for airplanes.
  - mr_toad 1y ago
    I just assumed it was developed by Boeing.
    - rigrassm 1y ago
      Thank you for starting my week with a good laugh!
  - Culonavirus 1y ago
    It's very much enough for drones tho... all you need is a tiny Jensen's chip, moped engine, some boom boom play-doh and you're ready to rock. No remote control needed.
    - zero_shift 1y ago
      Drones are expensive. Solid six figures expensive. And they are used around or on things that are even more expensive. You wouldn't want ChatGPT piloting them.
      - Culonavirus 1y ago
        Under $50k for a Geran-2 level drone.
        stockresearcher 1y ago
brettgriffin 1y ago
> Despite the rush to integrate powerful new models, about 5% of AI pilot programs achieve rapid revenue acceleration; the vast majority stall, delivering little to no measurable impact on P&L. This summer, I built two very sophisticated pieces of software. A financial ledger to power accrual accounting operations and a code generation framework that scaffolds a database from a defined data model to the frontend components and everything in between. I used ChatGPT substantially. I'm not sure how long it would have taken without generative AI, but in reality, I would have just given up out of frustration or exhaustion. From the outside, it would appear to any domain expert that at least three other people worked on these giving the pace at which they got completed. The completion of those two were seminal moments for me. I can't imagine how anyone, in any field of information systems, is not multiples more effective than they were five years ago. That directly affects a P&L and I can't think of anything in my career that is even remotely close to having that magnitude. I don't know what encapsulates an AI pilot in these orgs, and I'm sure they are massively more complex than anything I've done. But to hear 95% of these efforts don't have a demonstrable effect is just wild.
- nemomarx 1y ago
  I think they mean integrating AI into the business system directly and not using it to code things. I can see that having a more neutral impact
  - brettgriffin 1y ago
    > Generic tools like ChatGPT excel for individuals because of their flexibility, but they stall in enterprise use since they don’t learn from or adapt to workflows, Challapally explained. Maybe I misunderstood this, but I took this to mean that people inside enterprises are struggling using tools like ChatGPT. They do point out that perhaps the tools are being deployed in the wrong areas: > The data also reveals a misalignment in resource allocation. More than half of generative AI budgets are devoted to sales and marketing tools, yet MIT found the biggest ROI in back-office automation—eliminating business process outsourcing, cutting external agency costs, and streamlining operations. But I've seen some amazing automation does in sales and marketing that directly affected sales efficiency and reduced sales and marketing expenses.
bilsbie 1y ago
I can’t help feeling that we’re rapidly heading towards the “trough of disillusionment”. (How should I invest if I have this thesis)
- Davidzheng 1y ago
  short nvidia?
deleted 1y ago
[deleted]
K0nserv 1y ago
I'm arriving at the conclusion that deployments of LLMs is most suitable in areas where the cost of false positives and, crucially, false negatives are low. If you cannot tolerate false negatives I don't see how you get around the inaccuracy of LLMs. As long as you can spot false positives and their rate is sufficiently low they are merely an annoyance. I think this is a good consideration before starting a project leveraging LLMs
- infecto 1y ago
  Has inaccuracies been an issue for any of the systems you have developed using LLMs? I hear your complaint quite a bit but it does not align with my experience. Definitely one shotting a chatbot around an esoteric problem introduces possible inaccuracies. If I get an LLM to interrogate a pdf or other document that error rate drops significantly and is mostly on the part of the structuring process and not the LLM. Genuinely curious what others have experienced but specifically those that are using LLMs for business workflows. It is not to say any system is perfect but for purpose driven data pipelines LLMs can be pretty great.
  - K0nserv 1y ago
    Yes I've seen issues with both, but in part what's tricky about false negatives is also that you don't necessarily realise they are there. In the systems I've worked on we've made it simple for operators to verify the work the LLM has done, but this only guards against false positives, which are less problematic. I've had pretty good success using LLMs for coding and in some ways they are perfect for that. False positives are usually obvious and false negatives don't matter because as long as the LLM finds a solution, it's not a huge deal if there was a better way to do it. Even when the LLM cannot solve the problem at all, it usually produces some useful artifacts for the human to build on.
    - birn559 1y ago
      > as long as the LLM finds a solution, it's not a huge deal if there was a better way to do it It might not matter short term, but midterm such debt becomes a huge burden.
michaelfm1211 1y ago
> The data also reveals a misalignment in resource allocation. More than half of generative AI budgets are devoted to sales and marketing tools, yet MIT found the biggest ROI in back-office automation—eliminating business process outsourcing, cutting external agency costs, and streamlining operations. Makes sense. The people in charge of setting AI initiatives and policies are office people and managers who could be easily replaced by AI, but the people in charge not going to let themselves be replaced. Salesmen and engineers are the hardest to replace, yet they aren't in charge so they get replaced the fastest.
- zoeysmithe 1y ago
  I think this is being overly complimenting to AI. I think the most obvious reason is that for almost all business use cases its not very helpful. All these initiatives have the same problem. Staff asking 'how can this actually help me,' because they can't get it to help them other than polishing emails, polishing code, and writing summaries which is not what most people's jobs are. Then you have to proofread all of this because AI makes a lot of mistakes and poor assumptions, on top of hallucinations. I dont think Joe and Jane worker are purposely not using to protect their jobs, everyone wants ease at work, its just these LLM-based AI's dont offer much outside of some use cases. AI is vastly over-hyped and now we're in the part of the hype cycle where people are more comfortable saying to power, "This thing you love and think will raise your stock price is actually pretty terrible for almost all the things you said it would help with." AI has its place, but its not some kind of universal mind that will change everything and be applicable in significant and fundamentally changing ways outside of some narrow use cases. I'm on week 3 of making a video game (something I've never done before) with Claude/Chat and once I got past the 'tutorial level' design, these tools really struggle. I think even where an LLM would naturally be successful (structured logical languages), its still very underwhelming. I think we're just seeing people push back on hype and feeling empowered to say "This weird text autogenerator isn't helping me."
  - HankStallone 1y ago
    Part of it is that the bosses often don't know what they want, so they leave the details up to marketing or whoever, so replacing marketing or whoever with AI would mean figuring out what they want. The boss can tell marketing, "Make a brochure for new product ABC," and marketing can run with that and present him with a mock-up, he can make a couple revisions, they shine it up based on those, and then they're done. To replace them completely with AI, he would have to provide a lot more guidance and it would take more iterations to get a correct result that he likes. It wouldn't be completely unlike the current process, but it would demand more of him, which wouldn't make him happy. Last week I was talking to my boss about a project I've been working on for him, and he asked whether AI could help me with it to save time. I pointed out that a lot of the holdup in the project has been his not knowing exactly what he wants (because he's not sure what the software we're working with can do until I do it and show it to him), and an AI can't tell him what he wants any more than I can. Sometimes you just have to do the work, and technology can't help you.
agloe_dreams 1y ago
Nobody actually wants half the useless tools companies are coming up with because most of the solutions are not really novel. They are just wrapping an LLM. It's kinda like what I realized with the meta Ray-Bans: I can have these things on my face, they can tell me the answer to virtually any question in 10 seconds or less. But I, as a human, rarely have questions to ask. When you walk in to your local grocery store - you generally know what you want and where to find it. A ton of companies are just gluing LLM text boxes into apps and then scratching their heads when people don't use them. Why? Because the customer wasn't the user - it was their boss and shareholders. It was all done to make someone else think 'woah, they are following the trend!'. The core issue with generative AI is that it all works best when focused in a narrow sense. There is like one or two really clever uses I've seen - disappointingly, one of them was Jira. The internal jargon dictionary tool was legitimately impressive. Will it make any more money? Probably not.
- thewebguyd 1y ago
  > There is like one or two really clever uses I've seen - disappointingly, one of them was Jira. The internal jargon dictionary tool was legitimately impressive. Will it make any more money? Probably not. Sounds like Microsoft 365 Copilot at my org. Sucks at nearly everything, but it actually makes a fantastic search engine for emails, teams convos, sharepoint docs, etc. Much better that Microsoft's own global search stuff. Outside of coding, that's the only other real world use case I've found for LLMs - "get me all the emails, chats, and documents related to this upcoming meeting" and it's pretty good at that. Though I'm not sure we should be killing the earth for better search, there are probably other, better ways to do it.
  - ljf 1y ago
    Agreed - 95% of the questions I ask Copilot, I could answer myself by searching emails, Teams messages and files - BUT Copilot does a far far better job than me, and quicker. I went from barely using it, to using it daily. I wouldn't say it is a massive speed boost for me, but I'd miss it if it was taken away. Then the other 5% is the 'extra; it does for me, and gets me details I wouldn't have even known where to find. But it is just fancy search for me so far - but fancy search I see as valuable.
candiddevmike 1y ago
Actual report (State of AI in Business 2025): https://news.ycombinator.com/item?id=44941374 https://news.ycombinator.com/item?id=44941374
- deleted 1y ago
  [deleted]
airstrike 1y ago
Same source as https://news.ycombinator.com/item?id=44940944 https://news.ycombinator.com/item?id=44940944
onlyrealcuzzo 1y ago
These seems like a glass-is-half-empty view. 5% are succeeding. People are trying AI for just about everything right now. 5% is pretty damn good, when AI clearly has a lot of room to get better. The good models are quite expensive and slow. The fast & cheap models aren't that great - unless very specifically fine-tuned. Will it get better enough so that that growth rate in success pilots grows from 5% - 25% in 5 years or 20? Who knows, but it almost certainly will grow. It's hard to tell how much better the top foundation models will get over the next 5-10 years, but one thing that's certain is that the cost will go down substantially for the same quality over that time frame. Not to mention all the new use cases people will keep trying over that timeline. If in 10-years time, AI is succeeding in 2x as many use cases - that might not justify current valuations, but it will be a much better future - and necessary if we're planning on having ~25% of the population being retired / not working by then. Without AI replacing a lot of jobs, we're gonna have a tough time retiring all the people we promised retirements to.
- zero_shift 1y ago
  > 5% is pretty damn good, when AI clearly has a lot of room to get better. That depends if the AI successes depended much on the leading edge of LLM developments, or if actually most of the value was just "low hanging fruit". If the latter, that would imply the utility curve is levelling out, because new developments are not proving instrumental enough. I'm thinking of an S curve: slow improvements through the 2010s, then a burst of activity as the tech became good enough to do something "real", followed by more gradual wins in efficiency and accuracy.
  - onlyrealcuzzo 1y ago
    I agree it's an S-curve, but it's anyone's guess where on the S we are. And regardless, I still see this as very positive for society - and don't care as much about whether or not this is an AI bubble or not.
kubb 1y ago
How much money can you pull out as a failed startup founder? About a mil? Maybe two? Seems realistic… People have to invent whatever seems reasonable while squinting given how much accumulation of capital there is. The guys with money are easy to fool. Just lie to them about your „product”, get the cash, get out of the rat race, smooth sailing. Of course easier said than done. I can’t lie this convincingly, I don’t have the con man skillset or connections. So I’m stuck in a 9 to 5. Zzz…
- antisthenes 1y ago
  > Of course easier said than done. I can’t lie this convincingly, I don’t have the con man skillset or connections. Isn't the idea that you're not a shitty human being enough in and of itself?
  - kubb 1y ago
    I am. I'm working for a despicable company for money. And an incompetent one at that. I can't grab a bag and leave.
hendo3000 1y ago
There was an article on HN about the valuations of AI being out of touch with the question; what problem is being solved? We use generative imagery/video at my job and it's adding value. I see value being added for coders. There's real innovation happening, but I find it's mostly companies cutting corners making customer service even shittier than it already was.
- ModernMech 1y ago
  > real innovation happening, but I find it's mostly companies cutting corners There's a meme that I think fits: https://i.redd.it/20rpdamxef0f1.jpeg https://i.redd.it/20rpdamxef0f1.jpeg I think for a long time, cutting corners so that the number can go up next quarter has worked surprisingly well. Genuinely, I don't think a lot of corporations view offering a better product as a viable means of competing in the 2025 marketplace. For them, AI is not the next industrial revolution, it's the next overseas outsourcing; AI isn't a way to bring new value to customers, it's a way to bring roughly the same value (read worse) but at a much cheaper cost to them. If they get their way, everything will get worse, while they make more money. That's the value proposition at play here.
grahar64 1y ago
5% success is actually way higher than I thought it would be. At that rate I suppose there will be actually profitable AI companies with VC subsidies
- whymauri 1y ago
  5% success rate might mean: if you get it to work, you are capturing value that the other 95% are not. A lot of this must come down to execution. And there's a lot of snake oil out there at the execution layer.
  - Joel_Mckay 1y ago
    "So you're telling me there's a chance" https://www.youtube.com/watch?v=KX5jNnDMfxA https://www.youtube.com/watch?v=KX5jNnDMfxA 5% is not unexpected, as startup success rates are normally about 1:22 over 3 years. lol =3
strictnein 1y ago
> "“Every single Monday was called 'AI Monday.' You couldn’t have customer calls, you couldn’t work on budgets, you had to only work on AI projects.”" > "Vaughan saw that his team was not fully on board. His ultimate response? He replaced nearly 80% of the staff within a year" Being that this is Fortune magazine, it makes sense that they're portraying it this way, but reading between the lines there a little bit, it seems like the staff knew what would happen and wasn't keen on replacing themselves.
scotty79 1y ago
I remember when it was being said that computers in business had basically the same impact.
- ath3nd 1y ago
  Comparing an universal computing machine to what is essentially a fancy autocomplete is just bonkers.
  - scotty79 1y ago
    Calling fancy auto-complete a thing that can solve math and coding puzzles and translate between English and other languages, all better than most humans is just bonkers. Especially since those skills were outside of the grasp of "universal" computing machines for half a century. You could even say that comparing glorified ifs, and fors calculator to capabilities of even today's AIs is laughable.
ipnon 1y ago
This is proof LLMs are viable and productive in my opinion. The baseline rate for business failure over 5 years is around 90%, so they say. With how much hype surrounds LLM wrapper startups this is still an astounding amount of novel business model creation.
sam0x17 1y ago
I mean 5% not failing is pretty standard for any startup-driven thing.
lysecret 1y ago
Oh god what is this website it gives me a headache with all the pop-ups and auto playing videos.
sounds 1y ago
At this rate, how is it better than pure random chance? The article mentions 19-20 year old founders, focused on solving single user problems, were the successes. The sample size is 300 public AI deployments and an undisclosed number of private in-house AI projects. And the survey seems to only consider business applications, as compared with end-user applications like media and software. That's significant but not definitive. Isn't it more likely that existing problems with low hanging fruit, perhaps unpopular answers, that could be solved by leaning on "AI". And perhaps "AI" wasn't the key to success?
layer8 1y ago
The MIT report linked in the article is giving a 404 for some reason. Here is the web archive version: https://web.archive.org/web/20250818145714if_/https://nanda.media.mit.edu/ai_report_2025.pdf https://web.archive.org/web/20250818145714if_/https://nanda....
- amirkabbara 1y ago
  https://github.com/Papr-ai/papers/blob/main/v0.1%20State%20of%20AI%20in%20Business%202025%20Report.pdf https://github.com/Papr-ai/papers/blob/main/v0.1%20State%20o...
  - synctext 1y ago
    The MIT NANDA lab seems to have a link rot problem. Their cardinal code repo is also 404. The NANDA Lab also does coding, their publication at AAAI 2025 is titled: "CoDream: Exchanging dreams instead of models for federated aggregation with heterogeneous models" [1]. However, the link to the Github repo is broken. Fascinating paper, sad about the missing code. [1] https://mitmedialab.github.io/codream.github.io/ https://mitmedialab.github.io/codream.github.io/
syngrog66 1y ago
The title led me to assume it was about the aircraft type of pilot.
sitzkrieg 1y ago
lots of bad partial solutions looking for problems companies rushed to implement
richardblythman 1y ago
This resonates. Upskilling to AI tools is perhaps the biggest problem of our day. One idea we have to tackle this problem is to bring onboarding/learning directly into the user's work environment, track struggles and offer targeted support, and create continuous feedback loops. If anyone has faced challenges with increasing activation and retention of users on pilots (or external-facing products), would love to chat and see how we can help .
ultrasaurus 1y ago
We've talked with a ton of AI companies and I was surprised how much of the challenges were the usual challenges in any project. Just amplified by the rush to do AI right now, but I haven't seen anything as bad as "You couldn’t have customer calls, you couldn’t work on budgets, you had to only work on AI projects.” Warning for gratuitous self promotion: https://humansignal.com/blog/9-criteria-for-successful-ai-projects/ https://humansignal.com/blog/9-criteria-for-successful-ai-pr...
amirkabbara 1y ago
actual report here: https://github.com/Papr-ai/papers/blob/main/v0.1%20State%20of%20AI%20in%20Business%202025%20Report.pdf https://github.com/Papr-ai/papers/blob/main/v0.1%20State%20o...
deleted 1y ago
[deleted]

230 points by amirkabbara 1y ago

https://mlq.ai/media/quarterly_decks/v0.1_State_of_AI_in_Business_2025_Report.pdf https://mlq.ai/media/quarterly_decks/v0.1_State_of_AI_in_Bus...

amirkabbara 1y ago

Why so bad?

longtimelistnr 1y ago
Because for the typical office - documents are strewn about on random network drives and are not formatted similarly. This combined with the inability to nail down 100% accuracy on even just internal doc search is just too much to overcome for non-tech industry offices. My office is mind blown if i use Gemini to extract data from a PDF and convert it to an .xlsx or .csv As a technically minded person but not a comp sci guy, refining document search is like staring into a void and every option uses different (confusing) terminology. This makes it extra difficult for me to both do my regular job AND learn the multiple names/ways to do the exact same thing between platforms. The only solution that has any reliability for me so far are Gemini instances where i upload only the files i wish to search and just keep it locked to a few questions per instance before it starts to hallucinate. My attempt at RAG search implementation was a disaster that left me more confused than anything.
- noddingham 1y ago
  Because you mentioned the use case specifically, I wanted to point you to the fact that Excel has been able to convert images to tables for a while now. Literally screenshot a table from your PDF and it will convert to table. Not trying to diminish any additional capabilities you're getting from Gemini, but this screenshot to table feature has been huge for my finance team. https://support.microsoft.com/en-us/office/insert-data-from-picture-3c1bb58d-2c59-4bc0-b04a-a671a6868fd7 https://support.microsoft.com/en-us/office/insert-data-from-...
- amirkabbara 1y ago
  try https://www.papr.ai https://www.papr.ai for RAG. built it to solve this problem
appease7727 1y ago
Turns out that garbage text has very little intrinsic value
nathan_compton 1y ago
I think one reason for this is that LLMs are sort of maximally if accidentally designed to fuck up our brains. Despite all the advancements in the last five years I see them as still, fundamentally, text transformation machines which have only very limited sort of intelligence. Yet because nothing in history has been able to generate language except humans, most of us are not prepared to make rational judgements about their capabilities and those of us that may be also often fail to do so. The fact that we live in an era where tech people have been so investor pilled that overstating the capabilities of technology is basically second nature does not help.
troupo 1y ago
It's in the name: generative AIs. There are very few use cases at companies where you need to generate something. You want to work with the company's often very private disparate data (with access controls etc.) You wouldn't even have enough data to train a custom LLM, much less use a generic one.
ARandumGuy 1y ago
Any consumer facing AI project has to contend with the fact that GenAI is predominantly associated with "slop." If you're not actively using an AI tool, most of your experience with GenAI is seeing social media or Youtube flooded with low quality AI content, or having to deal with useless AI customer support. This gives the impression that AI is just cheap garbage, and something that should be actively avoided.
morkalork 1y ago
In my experience is that LLMs get you 80%of the way to a solution almost immediately but that last 20% when it comes to missing knowledge, data, or accuracy is a complete tar pit and will wreck adoption. Especially since many vendors are selling products that are wrappers and provide generic, non-customised solutions. I hear the same from others doing trials with various AI tools as well.

JohnClark1337 1y ago

[dead]

trenchpilgrim 1y ago

What's the failure rates if technology pilots in general for comparison? For example, I heard that SAP has an 80-90% deployment failure rate back in the day, but don't have a citable source for it.

kqr 1y ago
Depends on industry I would think. In my previous industry it was something like 25 %, in my current industry it is closer to 80 %.
RaftPeople 1y ago
> I heard that SAP has an 80-90% deployment failure rate Something to keep in mind is that ERP "failure" is frequently defined as went over budget or over time, even if it ultimately completed and provided the desired functionality. It's a much smaller percentage of projects that are either cancelled or went live and significantly did not function as the business needed.
aprilthird2021 1y ago
That is not remotely true tbh. The company would have failed long ago if it were
- mike_hearn 1y ago
  Not if every manufacturing company in the world decided to use your software anyway. ERP rollouts can "fail" for lots of reasons that aren't to do with the software. They are usually business failures. Mostly, companies end up spending so much on trying to endlessly customize it to their idiosyncratic workflows that they exceed their project budgets and abandon the effort. In really bad cases like Birmingham they go live before actually finishing setup, and then lose control of their books and have to resort to hiring people to do the admin manually. There's a saying about SAP: at some point gaining competitive advantage in manufacturing/retail became all about who could make SAP deployment a success. This is no different to many other IT projects, most of them fail too. I think people who have never worked in an enterprise context don't realize that; it's not like working in the tech sector. In the tech industry if a project fails, it's probably because it was too ambitious and the tech itself just didn't work well. Or it was a startup whose tech worked, but they couldn't find PMF. But in normal, mature, profitable non-tech businesses a staggering number of business automation projects just fail for social or business reasons. AI deployments inside companies are going to be like that. The tech works. The business side problems are where the failures are going to happen. Reasons will include: • Not really knowing what they want the AI to do. • No way to measure improved productivity, so no way to decide if the API spend is worth it. • Concluding the only way to get a return is entirely replace people with AI and then having to re-hire them because the AI can't handle the last 5% of the work. • Non-tech executives doing deals to use models or tech stacks that aren't the right kind or good enough. etc

etothet 1y ago

https://archive.is/bdi7b https://archive.is/bdi7b

zahlman 1y ago

Am I the only one who looked at this shortened headline and wondered why anyone is allowing AIs to fly airplanes?

madcaptenor 1y ago
No. I also thought that even a 95% success rate wouldn't be good enough for airplanes.
- mr_toad 1y ago
  I just assumed it was developed by Boeing.
  - rigrassm 1y ago
    Thank you for starting my week with a good laugh!
- Culonavirus 1y ago
  It's very much enough for drones tho... all you need is a tiny Jensen's chip, moped engine, some boom boom play-doh and you're ready to rock. No remote control needed.
  - zero_shift 1y ago
    Drones are expensive. Solid six figures expensive. And they are used around or on things that are even more expensive. You wouldn't want ChatGPT piloting them.
    - Culonavirus 1y ago
      Under $50k for a Geran-2 level drone.
      - stockresearcher 1y ago

brettgriffin 1y ago

> Despite the rush to integrate powerful new models, about 5% of AI pilot programs achieve rapid revenue acceleration; the vast majority stall, delivering little to no measurable impact on P&L. This summer, I built two very sophisticated pieces of software. A financial ledger to power accrual accounting operations and a code generation framework that scaffolds a database from a defined data model to the frontend components and everything in between. I used ChatGPT substantially. I'm not sure how long it would have taken without generative AI, but in reality, I would have just given up out of frustration or exhaustion. From the outside, it would appear to any domain expert that at least three other people worked on these giving the pace at which they got completed. The completion of those two were seminal moments for me. I can't imagine how anyone, in any field of information systems, is not multiples more effective than they were five years ago. That directly affects a P&L and I can't think of anything in my career that is even remotely close to having that magnitude. I don't know what encapsulates an AI pilot in these orgs, and I'm sure they are massively more complex than anything I've done. But to hear 95% of these efforts don't have a demonstrable effect is just wild.

nemomarx 1y ago
I think they mean integrating AI into the business system directly and not using it to code things. I can see that having a more neutral impact
- brettgriffin 1y ago
  > Generic tools like ChatGPT excel for individuals because of their flexibility, but they stall in enterprise use since they don’t learn from or adapt to workflows, Challapally explained. Maybe I misunderstood this, but I took this to mean that people inside enterprises are struggling using tools like ChatGPT. They do point out that perhaps the tools are being deployed in the wrong areas: > The data also reveals a misalignment in resource allocation. More than half of generative AI budgets are devoted to sales and marketing tools, yet MIT found the biggest ROI in back-office automation—eliminating business process outsourcing, cutting external agency costs, and streamlining operations. But I've seen some amazing automation does in sales and marketing that directly affected sales efficiency and reduced sales and marketing expenses.

bilsbie 1y ago

I can’t help feeling that we’re rapidly heading towards the “trough of disillusionment”. (How should I invest if I have this thesis)

Davidzheng 1y ago
short nvidia?

deleted 1y ago

[deleted]

K0nserv 1y ago

I'm arriving at the conclusion that deployments of LLMs is most suitable in areas where the cost of false positives and, crucially, false negatives are low. If you cannot tolerate false negatives I don't see how you get around the inaccuracy of LLMs. As long as you can spot false positives and their rate is sufficiently low they are merely an annoyance. I think this is a good consideration before starting a project leveraging LLMs

infecto 1y ago
Has inaccuracies been an issue for any of the systems you have developed using LLMs? I hear your complaint quite a bit but it does not align with my experience. Definitely one shotting a chatbot around an esoteric problem introduces possible inaccuracies. If I get an LLM to interrogate a pdf or other document that error rate drops significantly and is mostly on the part of the structuring process and not the LLM. Genuinely curious what others have experienced but specifically those that are using LLMs for business workflows. It is not to say any system is perfect but for purpose driven data pipelines LLMs can be pretty great.
- K0nserv 1y ago
  Yes I've seen issues with both, but in part what's tricky about false negatives is also that you don't necessarily realise they are there. In the systems I've worked on we've made it simple for operators to verify the work the LLM has done, but this only guards against false positives, which are less problematic. I've had pretty good success using LLMs for coding and in some ways they are perfect for that. False positives are usually obvious and false negatives don't matter because as long as the LLM finds a solution, it's not a huge deal if there was a better way to do it. Even when the LLM cannot solve the problem at all, it usually produces some useful artifacts for the human to build on.
  - birn559 1y ago
    > as long as the LLM finds a solution, it's not a huge deal if there was a better way to do it It might not matter short term, but midterm such debt becomes a huge burden.

michaelfm1211 1y ago

> The data also reveals a misalignment in resource allocation. More than half of generative AI budgets are devoted to sales and marketing tools, yet MIT found the biggest ROI in back-office automation—eliminating business process outsourcing, cutting external agency costs, and streamlining operations. Makes sense. The people in charge of setting AI initiatives and policies are office people and managers who could be easily replaced by AI, but the people in charge not going to let themselves be replaced. Salesmen and engineers are the hardest to replace, yet they aren't in charge so they get replaced the fastest.

zoeysmithe 1y ago
I think this is being overly complimenting to AI. I think the most obvious reason is that for almost all business use cases its not very helpful. All these initiatives have the same problem. Staff asking 'how can this actually help me,' because they can't get it to help them other than polishing emails, polishing code, and writing summaries which is not what most people's jobs are. Then you have to proofread all of this because AI makes a lot of mistakes and poor assumptions, on top of hallucinations. I dont think Joe and Jane worker are purposely not using to protect their jobs, everyone wants ease at work, its just these LLM-based AI's dont offer much outside of some use cases. AI is vastly over-hyped and now we're in the part of the hype cycle where people are more comfortable saying to power, "This thing you love and think will raise your stock price is actually pretty terrible for almost all the things you said it would help with." AI has its place, but its not some kind of universal mind that will change everything and be applicable in significant and fundamentally changing ways outside of some narrow use cases. I'm on week 3 of making a video game (something I've never done before) with Claude/Chat and once I got past the 'tutorial level' design, these tools really struggle. I think even where an LLM would naturally be successful (structured logical languages), its still very underwhelming. I think we're just seeing people push back on hype and feeling empowered to say "This weird text autogenerator isn't helping me."
- HankStallone 1y ago
  Part of it is that the bosses often don't know what they want, so they leave the details up to marketing or whoever, so replacing marketing or whoever with AI would mean figuring out what they want. The boss can tell marketing, "Make a brochure for new product ABC," and marketing can run with that and present him with a mock-up, he can make a couple revisions, they shine it up based on those, and then they're done. To replace them completely with AI, he would have to provide a lot more guidance and it would take more iterations to get a correct result that he likes. It wouldn't be completely unlike the current process, but it would demand more of him, which wouldn't make him happy. Last week I was talking to my boss about a project I've been working on for him, and he asked whether AI could help me with it to save time. I pointed out that a lot of the holdup in the project has been his not knowing exactly what he wants (because he's not sure what the software we're working with can do until I do it and show it to him), and an AI can't tell him what he wants any more than I can. Sometimes you just have to do the work, and technology can't help you.

agloe_dreams 1y ago

Nobody actually wants half the useless tools companies are coming up with because most of the solutions are not really novel. They are just wrapping an LLM. It's kinda like what I realized with the meta Ray-Bans: I can have these things on my face, they can tell me the answer to virtually any question in 10 seconds or less. But I, as a human, rarely have questions to ask. When you walk in to your local grocery store - you generally know what you want and where to find it. A ton of companies are just gluing LLM text boxes into apps and then scratching their heads when people don't use them. Why? Because the customer wasn't the user - it was their boss and shareholders. It was all done to make someone else think 'woah, they are following the trend!'. The core issue with generative AI is that it all works best when focused in a narrow sense. There is like one or two really clever uses I've seen - disappointingly, one of them was Jira. The internal jargon dictionary tool was legitimately impressive. Will it make any more money? Probably not.

thewebguyd 1y ago
> There is like one or two really clever uses I've seen - disappointingly, one of them was Jira. The internal jargon dictionary tool was legitimately impressive. Will it make any more money? Probably not. Sounds like Microsoft 365 Copilot at my org. Sucks at nearly everything, but it actually makes a fantastic search engine for emails, teams convos, sharepoint docs, etc. Much better that Microsoft's own global search stuff. Outside of coding, that's the only other real world use case I've found for LLMs - "get me all the emails, chats, and documents related to this upcoming meeting" and it's pretty good at that. Though I'm not sure we should be killing the earth for better search, there are probably other, better ways to do it.
- ljf 1y ago
  Agreed - 95% of the questions I ask Copilot, I could answer myself by searching emails, Teams messages and files - BUT Copilot does a far far better job than me, and quicker. I went from barely using it, to using it daily. I wouldn't say it is a massive speed boost for me, but I'd miss it if it was taken away. Then the other 5% is the 'extra; it does for me, and gets me details I wouldn't have even known where to find. But it is just fancy search for me so far - but fancy search I see as valuable.

candiddevmike 1y ago

Actual report (State of AI in Business 2025): https://news.ycombinator.com/item?id=44941374 https://news.ycombinator.com/item?id=44941374

deleted 1y ago
[deleted]

airstrike 1y ago

Same source as https://news.ycombinator.com/item?id=44940944 https://news.ycombinator.com/item?id=44940944

onlyrealcuzzo 1y ago

These seems like a glass-is-half-empty view. 5% are succeeding. People are trying AI for just about everything right now. 5% is pretty damn good, when AI clearly has a lot of room to get better. The good models are quite expensive and slow. The fast & cheap models aren't that great - unless very specifically fine-tuned. Will it get better enough so that that growth rate in success pilots grows from 5% - 25% in 5 years or 20? Who knows, but it almost certainly will grow. It's hard to tell how much better the top foundation models will get over the next 5-10 years, but one thing that's certain is that the cost will go down substantially for the same quality over that time frame. Not to mention all the new use cases people will keep trying over that timeline. If in 10-years time, AI is succeeding in 2x as many use cases - that might not justify current valuations, but it will be a much better future - and necessary if we're planning on having ~25% of the population being retired / not working by then. Without AI replacing a lot of jobs, we're gonna have a tough time retiring all the people we promised retirements to.

zero_shift 1y ago
> 5% is pretty damn good, when AI clearly has a lot of room to get better. That depends if the AI successes depended much on the leading edge of LLM developments, or if actually most of the value was just "low hanging fruit". If the latter, that would imply the utility curve is levelling out, because new developments are not proving instrumental enough. I'm thinking of an S curve: slow improvements through the 2010s, then a burst of activity as the tech became good enough to do something "real", followed by more gradual wins in efficiency and accuracy.
- onlyrealcuzzo 1y ago
  I agree it's an S-curve, but it's anyone's guess where on the S we are. And regardless, I still see this as very positive for society - and don't care as much about whether or not this is an AI bubble or not.

kubb 1y ago

How much money can you pull out as a failed startup founder? About a mil? Maybe two? Seems realistic… People have to invent whatever seems reasonable while squinting given how much accumulation of capital there is. The guys with money are easy to fool. Just lie to them about your „product”, get the cash, get out of the rat race, smooth sailing. Of course easier said than done. I can’t lie this convincingly, I don’t have the con man skillset or connections. So I’m stuck in a 9 to 5. Zzz…

antisthenes 1y ago
> Of course easier said than done. I can’t lie this convincingly, I don’t have the con man skillset or connections. Isn't the idea that you're not a shitty human being enough in and of itself?
- kubb 1y ago
  I am. I'm working for a despicable company for money. And an incompetent one at that. I can't grab a bag and leave.

hendo3000 1y ago

There was an article on HN about the valuations of AI being out of touch with the question; what problem is being solved? We use generative imagery/video at my job and it's adding value. I see value being added for coders. There's real innovation happening, but I find it's mostly companies cutting corners making customer service even shittier than it already was.

ModernMech 1y ago
> real innovation happening, but I find it's mostly companies cutting corners There's a meme that I think fits: https://i.redd.it/20rpdamxef0f1.jpeg https://i.redd.it/20rpdamxef0f1.jpeg I think for a long time, cutting corners so that the number can go up next quarter has worked surprisingly well. Genuinely, I don't think a lot of corporations view offering a better product as a viable means of competing in the 2025 marketplace. For them, AI is not the next industrial revolution, it's the next overseas outsourcing; AI isn't a way to bring new value to customers, it's a way to bring roughly the same value (read worse) but at a much cheaper cost to them. If they get their way, everything will get worse, while they make more money. That's the value proposition at play here.

grahar64 1y ago

5% success is actually way higher than I thought it would be. At that rate I suppose there will be actually profitable AI companies with VC subsidies

whymauri 1y ago
5% success rate might mean: if you get it to work, you are capturing value that the other 95% are not. A lot of this must come down to execution. And there's a lot of snake oil out there at the execution layer.
- Joel_Mckay 1y ago
  "So you're telling me there's a chance" https://www.youtube.com/watch?v=KX5jNnDMfxA https://www.youtube.com/watch?v=KX5jNnDMfxA 5% is not unexpected, as startup success rates are normally about 1:22 over 3 years. lol =3

strictnein 1y ago

> "“Every single Monday was called 'AI Monday.' You couldn’t have customer calls, you couldn’t work on budgets, you had to only work on AI projects.”" > "Vaughan saw that his team was not fully on board. His ultimate response? He replaced nearly 80% of the staff within a year" Being that this is Fortune magazine, it makes sense that they're portraying it this way, but reading between the lines there a little bit, it seems like the staff knew what would happen and wasn't keen on replacing themselves.

scotty79 1y ago

I remember when it was being said that computers in business had basically the same impact.

ath3nd 1y ago
Comparing an universal computing machine to what is essentially a fancy autocomplete is just bonkers.
- scotty79 1y ago
  Calling fancy auto-complete a thing that can solve math and coding puzzles and translate between English and other languages, all better than most humans is just bonkers. Especially since those skills were outside of the grasp of "universal" computing machines for half a century. You could even say that comparing glorified ifs, and fors calculator to capabilities of even today's AIs is laughable.

ipnon 1y ago

This is proof LLMs are viable and productive in my opinion. The baseline rate for business failure over 5 years is around 90%, so they say. With how much hype surrounds LLM wrapper startups this is still an astounding amount of novel business model creation.

sam0x17 1y ago

I mean 5% not failing is pretty standard for any startup-driven thing.

lysecret 1y ago

Oh god what is this website it gives me a headache with all the pop-ups and auto playing videos.

sounds 1y ago

At this rate, how is it better than pure random chance? The article mentions 19-20 year old founders, focused on solving single user problems, were the successes. The sample size is 300 public AI deployments and an undisclosed number of private in-house AI projects. And the survey seems to only consider business applications, as compared with end-user applications like media and software. That's significant but not definitive. Isn't it more likely that existing problems with low hanging fruit, perhaps unpopular answers, that could be solved by leaning on "AI". And perhaps "AI" wasn't the key to success?

layer8 1y ago

The MIT report linked in the article is giving a 404 for some reason. Here is the web archive version: https://web.archive.org/web/20250818145714if_/https://nanda.media.mit.edu/ai_report_2025.pdf https://web.archive.org/web/20250818145714if_/https://nanda....

amirkabbara 1y ago
https://github.com/Papr-ai/papers/blob/main/v0.1%20State%20of%20AI%20in%20Business%202025%20Report.pdf https://github.com/Papr-ai/papers/blob/main/v0.1%20State%20o...
- synctext 1y ago
  The MIT NANDA lab seems to have a link rot problem. Their cardinal code repo is also 404. The NANDA Lab also does coding, their publication at AAAI 2025 is titled: "CoDream: Exchanging dreams instead of models for federated aggregation with heterogeneous models" [1]. However, the link to the Github repo is broken. Fascinating paper, sad about the missing code. [1] https://mitmedialab.github.io/codream.github.io/ https://mitmedialab.github.io/codream.github.io/

syngrog66 1y ago

The title led me to assume it was about the aircraft type of pilot.

sitzkrieg 1y ago

lots of bad partial solutions looking for problems companies rushed to implement

richardblythman 1y ago

This resonates. Upskilling to AI tools is perhaps the biggest problem of our day. One idea we have to tackle this problem is to bring onboarding/learning directly into the user's work environment, track struggles and offer targeted support, and create continuous feedback loops. If anyone has faced challenges with increasing activation and retention of users on pilots (or external-facing products), would love to chat and see how we can help .

ultrasaurus 1y ago

We've talked with a ton of AI companies and I was surprised how much of the challenges were the usual challenges in any project. Just amplified by the rush to do AI right now, but I haven't seen anything as bad as "You couldn’t have customer calls, you couldn’t work on budgets, you had to only work on AI projects.” Warning for gratuitous self promotion: https://humansignal.com/blog/9-criteria-for-successful-ai-projects/ https://humansignal.com/blog/9-criteria-for-successful-ai-pr...

amirkabbara 1y ago

actual report here: https://github.com/Papr-ai/papers/blob/main/v0.1%20State%20of%20AI%20in%20Business%202025%20Report.pdf https://github.com/Papr-ai/papers/blob/main/v0.1%20State%20o...

deleted 1y ago

[deleted]