4 ms·

> We're seeing exactly the same thing with AI, as there is massive investment creating a bubble without a payoff. ... And so far there's no evidence that all

by keeda 6mo ago

> We're seeing exactly the same thing with AI, as there is massive investment creating a bubble without a payoff. ... And so far there's no evidence that all this investment has generated more profit for the users of AI. If you look around a bit, you will find evidence for both. Recent data finds pretty high success in GenAI adoption even as "formal ROI measurement" -- i.e. not based on "vibes" -- becomes common: https://knowledge.wharton.upenn.edu/special-report/2025-ai-adoption-report/ https://knowledge.wharton.upenn.edu/special-report/2025-ai-a... (tl;dr: about 75% report positive RoI.) The trustworthiness, salience and nuances of this report is worth discussing, but unfortunately reports like this gets no airtime in the HN and the media echo chamber. Preliminary evidence, but given this weird, entirely unprecedented technology is about 3+ years old and people are still figuring it out (something that report calls out) this is significant.

0xbadcafebee 6mo ago
75% report positive ROI (and the VPs are much more "optimistic" than the middle managers who are closer to the work) - but how much ROI? 1%? The fact that they don't quote a figure at all is pretty telling. And that's the ROI of the people buying the AI services, which are often heavily subsidized. If it costs a billion dollars to give a mid-sized company a 1% ROI, that doesn't sound sustainable. I would love to see another report that isn't a year old with actual ROI figures...
- chatmasta 6mo ago
  It’s not easy to quantify because you’re basically substituting or augmenting labor. How do you quantify an ROI on employees? You can look at profit of a project they’re hired to execute. But with AI, it’s mixed with the employees, so how do you distinguish the ROI of the two? With time, we might be able to make comparisons, but outside of very specific scenarios it’s difficult to quantify.
  - lazide 6mo ago
    Everyone I’ve seen try has had negative actual ROI. All the middle managers are afraid to say anything though, so go go go.
- keeda 6mo ago
  Good questions! I have only skimmed through the report but slide 45 onwards of the full report has some vague numbers: https://ai.wharton.upenn.edu/wp-content/uploads/2025/10/2025-Wharton-GBK-AI-Adoption-Report_Full-Report.pdf https://ai.wharton.upenn.edu/wp-content/uploads/2025/10/2025... Can't say why they don't report exact numbers, but it may be because a) of confidentiality and b) RoI is very context dependent and c) there is a wide spectrum of RoI by different dimensions, including some 9% even reporting negative RoI. This may make it hard to cite a single number, but the majority report "moderate" to "significant" RoI, whatever that means to them. I'll add that I've seen mentions of similar reports from other sources like McKinsey and co. e.g. this one that claims actual revenue increase: https://www.mckinsey.com/featured-insights/week-in-charts/gen-ais-roi https://www.mckinsey.com/featured-insights/week-in-charts/ge... -- I tend not to take these reports at face value, but I'm seeing multiple of them from various sources that tend to align. As an aside, I just wanted to say, these are the kinds of discussions I was hoping to see here!
SlinkyOnStairs 6mo ago
> The trustworthiness, salience and nuances of this report is worth discussing, but unfortunately reports like this gets no airtime in the HN and the media echo chamber. It honestly just isn't that interesting. (Being most notable for people misunderstanding and misrepresenting the chart on page 46 of the report as being "ROI" rather than "ROI measurement") In terms of ROI figures, it's really just a survey with the question "Based on internal conversations with colleagues and senior leadership, what has been the return on investment (ROI) from your organization's Gen AI initiatives to date?". This doesn't mean much. It's not even dubiously-measured ROI data, it's not ROI data at all, it's just what the leadership thinks is true. And that's a worrying thing to rely on, as it's well documented (and measured by the report's next question) that there's a significant discrepancy in how high level leadership and low-level leadership/ICs rate AI "ROI". One of the main explanations of that discrepancy being Goodhart's law. A large amount of companies are simply demanding AI productivity as a "target" now, with accusations of "worker sabotage" being thrown around readily. That makes good economy-wide data on AI ROI very hard to get.
- keeda 6mo ago
  That's fair, it is survey based but it is apparently based on formal internal measurements. The full report (https://ai.wharton.upenn.edu/wp-content/uploads/2025/10/2025-Wharton-GBK-AI-Adoption-Report_Full-Report.pdf https://ai.wharton.upenn.edu/wp-content/uploads/2025/10/2025... -- slides 43 onwards) mentions that for 75% of them have "integrated formal ROI measurement." There is little discussion of what that means, however, but we really can't expect concrete numbers for what is going to be sensitive business data,and given that the report tracks it across multiple industries and functions ranging from IT to operations to legal to sales, it may be hard to put into sensible numbers, or how the measurements may be flawed or biased.

by keeda 6mo ago

0xbadcafebee 6mo ago

75% report positive ROI (and the VPs are much more "optimistic" than the middle managers who are closer to the work) - but how much ROI? 1%? The fact that they don't quote a figure at all is pretty telling. And that's the ROI of the people buying the AI services, which are often heavily subsidized. If it costs a billion dollars to give a mid-sized company a 1% ROI, that doesn't sound sustainable. I would love to see another report that isn't a year old with actual ROI figures...

chatmasta 6mo ago
It’s not easy to quantify because you’re basically substituting or augmenting labor. How do you quantify an ROI on employees? You can look at profit of a project they’re hired to execute. But with AI, it’s mixed with the employees, so how do you distinguish the ROI of the two? With time, we might be able to make comparisons, but outside of very specific scenarios it’s difficult to quantify.
- lazide 6mo ago
  Everyone I’ve seen try has had negative actual ROI. All the middle managers are afraid to say anything though, so go go go.
keeda 6mo ago
Good questions! I have only skimmed through the report but slide 45 onwards of the full report has some vague numbers: https://ai.wharton.upenn.edu/wp-content/uploads/2025/10/2025-Wharton-GBK-AI-Adoption-Report_Full-Report.pdf https://ai.wharton.upenn.edu/wp-content/uploads/2025/10/2025... Can't say why they don't report exact numbers, but it may be because a) of confidentiality and b) RoI is very context dependent and c) there is a wide spectrum of RoI by different dimensions, including some 9% even reporting negative RoI. This may make it hard to cite a single number, but the majority report "moderate" to "significant" RoI, whatever that means to them. I'll add that I've seen mentions of similar reports from other sources like McKinsey and co. e.g. this one that claims actual revenue increase: https://www.mckinsey.com/featured-insights/week-in-charts/gen-ais-roi https://www.mckinsey.com/featured-insights/week-in-charts/ge... -- I tend not to take these reports at face value, but I'm seeing multiple of them from various sources that tend to align. As an aside, I just wanted to say, these are the kinds of discussions I was hoping to see here!

SlinkyOnStairs 6mo ago

> The trustworthiness, salience and nuances of this report is worth discussing, but unfortunately reports like this gets no airtime in the HN and the media echo chamber. It honestly just isn't that interesting. (Being most notable for people misunderstanding and misrepresenting the chart on page 46 of the report as being "ROI" rather than "ROI measurement") In terms of ROI figures, it's really just a survey with the question "Based on internal conversations with colleagues and senior leadership, what has been the return on investment (ROI) from your organization's Gen AI initiatives to date?". This doesn't mean much. It's not even dubiously-measured ROI data, it's not ROI data at all, it's just what the leadership thinks is true. And that's a worrying thing to rely on, as it's well documented (and measured by the report's next question) that there's a significant discrepancy in how high level leadership and low-level leadership/ICs rate AI "ROI". One of the main explanations of that discrepancy being Goodhart's law. A large amount of companies are simply demanding AI productivity as a "target" now, with accusations of "worker sabotage" being thrown around readily. That makes good economy-wide data on AI ROI very hard to get.

keeda 6mo ago
That's fair, it is survey based but it is apparently based on formal internal measurements. The full report (https://ai.wharton.upenn.edu/wp-content/uploads/2025/10/2025-Wharton-GBK-AI-Adoption-Report_Full-Report.pdf https://ai.wharton.upenn.edu/wp-content/uploads/2025/10/2025... -- slides 43 onwards) mentions that for 75% of them have "integrated formal ROI measurement." There is little discussion of what that means, however, but we really can't expect concrete numbers for what is going to be sensitive business data,and given that the report tracks it across multiple industries and functions ranging from IT to operations to legal to sales, it may be hard to put into sensible numbers, or how the measurements may be flawed or biased.