3 ms·

> Usage does not correlate with valued output or ROI This sounds possible but how can we know? It could just as well correlate. Effort of all kinds correlates

by blfr 2mo ago

> Usage does not correlate with valued output or ROI This sounds possible but how can we know? It could just as well correlate. Effort of all kinds correlates with success even when it's not obvious or not... linear.

danggggg 2mo ago
[dead]
- blfr 2mo ago
  It's hard to measure ROI so we generally wouldn't. Even McKinseys and BCGs have a hard time showing ROIs. Sure, you could say all digital transformations are nebulous snake oil and don't deliver but I just don't think this is the case.
  - deleted 2mo ago
    [deleted]
cmiles8 2mo ago
If that was true then there wouldn’t be a giant ROI crisis fueling calls of an AI bubble burst imminent. If the answer was “the ROI is clear” then folks would just say that.
- stuartjohnson12 2mo ago
  The debate about ROI isn't about whether everyone is going to throw their hands up and stop using AI tooling - the debate is about where that value is going to be captured. It's pretty clear that using some degree AI assistance for programming is on the whole faster and better than writing code by hand for the majority of applications, and it's clear that the degree to which it's better than writing code by hand is increasing quickly. I'm sure someone is going to reply to and say "actually it's not clear and the tech debt is about to explode everything real soon now" or something to that effect, but it's hard for me to explain this theoretically when it is so obvious in practice in my day to day.
  - kyllo 2mo ago
    On the value capture question, it's really not clear that the ability to write code at superhuman speed actually translates to an increase in the speed with which we can create new, better technology products that customers will pay more money for, vs. just enabling layoffs.
    - Majromax 2mo ago
      You could say the same thing about compilers versus assemblers, high-level languages versus low-level ones, and services and libraries versus monolithic programs. All other things being equal, increasing the speed of some part of the development process will increase the overall pace of development. However, By Amdahl's law that increase will be sublinear, and that is why we should take "pull requests" as an imperfect metric. We also don't get to pick the form that 'better technology products' take. While we'd probably like to keep cost(/effort) and complexity constant and increase robustness and performance, the market equilibrium might be 'worse is better' and reward whiz-bang features and lower effort.
      - porridgeraisin 2mo ago
        Yep. Also, large companies are still experimenting on how to integrate LLMs into their workflows. Due to the fast cadence of releases, people forget that LLMs became robust (regardless of the capability level/parameter count/data size) enough to use semi-reliably in company-specific ways only 1 year ago. And bigger the org, the slower the process. I don't expect it to settle and get productive used across the majority of very large companies for another year atleast. High prices are mostly a result of DC capacity. As more and more DCs get built out, prices will drop. At a unit level the inference business is extremely sound regardless.
      - kyllo 2mo ago
        > You could say the same thing about compilers versus assemblers, high-level languages versus low-level ones, and services and libraries versus monolithic programs. Right, but we didn't invest a trillion dollars of capital into any of those things in the span of a couple years, thus forcing them to capture value and show a return on such a massive investment.

by blfr 2mo ago

danggggg 2mo ago

[dead]

blfr 2mo ago
It's hard to measure ROI so we generally wouldn't. Even McKinseys and BCGs have a hard time showing ROIs. Sure, you could say all digital transformations are nebulous snake oil and don't deliver but I just don't think this is the case.
- deleted 2mo ago
  [deleted]

cmiles8 2mo ago

If that was true then there wouldn’t be a giant ROI crisis fueling calls of an AI bubble burst imminent. If the answer was “the ROI is clear” then folks would just say that.

stuartjohnson12 2mo ago
The debate about ROI isn't about whether everyone is going to throw their hands up and stop using AI tooling - the debate is about where that value is going to be captured. It's pretty clear that using some degree AI assistance for programming is on the whole faster and better than writing code by hand for the majority of applications, and it's clear that the degree to which it's better than writing code by hand is increasing quickly. I'm sure someone is going to reply to and say "actually it's not clear and the tech debt is about to explode everything real soon now" or something to that effect, but it's hard for me to explain this theoretically when it is so obvious in practice in my day to day.
- kyllo 2mo ago
  On the value capture question, it's really not clear that the ability to write code at superhuman speed actually translates to an increase in the speed with which we can create new, better technology products that customers will pay more money for, vs. just enabling layoffs.
  - Majromax 2mo ago
    You could say the same thing about compilers versus assemblers, high-level languages versus low-level ones, and services and libraries versus monolithic programs. All other things being equal, increasing the speed of some part of the development process will increase the overall pace of development. However, By Amdahl's law that increase will be sublinear, and that is why we should take "pull requests" as an imperfect metric. We also don't get to pick the form that 'better technology products' take. While we'd probably like to keep cost(/effort) and complexity constant and increase robustness and performance, the market equilibrium might be 'worse is better' and reward whiz-bang features and lower effort.
    - porridgeraisin 2mo ago
      Yep. Also, large companies are still experimenting on how to integrate LLMs into their workflows. Due to the fast cadence of releases, people forget that LLMs became robust (regardless of the capability level/parameter count/data size) enough to use semi-reliably in company-specific ways only 1 year ago. And bigger the org, the slower the process. I don't expect it to settle and get productive used across the majority of very large companies for another year atleast. High prices are mostly a result of DC capacity. As more and more DCs get built out, prices will drop. At a unit level the inference business is extremely sound regardless.
    - kyllo 2mo ago
      > You could say the same thing about compilers versus assemblers, high-level languages versus low-level ones, and services and libraries versus monolithic programs. Right, but we didn't invest a trillion dollars of capital into any of those things in the span of a couple years, thus forcing them to capture value and show a return on such a massive investment.