3 ms·

> We’re talking about each developer gradually boosting their productivity by a multiplier of ~5x by Q4 2025 (allowing for ramp-up time), for an additional amor

> We’re talking about each developer gradually boosting their productivity by a multiplier of ~5x by Q4 2025 (allowing for ramp-up time), for an additional amortized cost of only maybe $50k/year the first year. Who wouldn’t go for that deal? OK, I'll take the other side of that bet. If in Q4 '25 devs using cursor or whatever are 5x as productive as me using emacs, I'll give this AI stuff another chance. But I'm pretty sure it won't happen.

victorbjorklund 2y ago
You should probably compare the same dev using AI vs not using AI. Otherwise you can use that argument for anything. Notepad is a better IDE than emacs if we compare a really good dev using notepad vs a shitty dev using emacs.
- jcgrillo 2y ago
  Ok, if I notice other devs suddenly getting 5x as productive I'll give it a try, but so far no such effect has been demonstrated. It seems like a pretty straightforward research question, and you'd think if there was any demonstrable effect the companies selling these things would use such research to market their products. So where is it?
  - fragmede 2y ago
    There's a dead body lying there but you won't believe they're actually dead until someone creates a video reenactment of the killer stabbing them and shoves that video in your face? Speaking of videos, https://youtu.be/opi1s_5Dm-c https://youtu.be/opi1s_5Dm-c is probably below your level and it's kinda long.
    - jcgrillo 2y ago
      I don't get it. What is the dead body thing about?
      - fragmede 2y ago
        There are people building things and making money using these new techniques, but because they haven't stopped to produce research studies and white papers about their work, you want to presume it doesn't exist.
        jcgrillo 2y ago
        Yes, without evidence there's no effect. It's telling that nowhere on cursor.ai's website do they appear to claim using their tool will actually make you, or crucially your team, more efficient.
        fragmede 2y ago
        evidencthe absence of evidence doesn't mean there's no effect
        jcgrillo 2y ago
        There is no trustworthy evidence that I'm aware of to support the claim that "using an LLM IDE makes developers 5x more effective". Do you disagree? Testimonials don't cut it, because individuals are prone to hysteria, delusions, and just making stuff up. All I'm asking for is a few trustworthy case studies showing unambiguously that it's worth my time and money to pay attention to this genai stuff. That shouldn't be hard for a trillion dollar industry, right? Like show me this isn't just hysterical hype. Don't tell me, show me.
        fragmede 2y ago
        What does it even mean to be 5x more effective? Because that's the thing. Testimonials is where we're at because how do you measure "more effective"? I'm not making any claim on a number, 5x 10x? .2x because you have to go back and fix the shitty code it wrote? We don't have a good way to measure programmer productivity. We all know lines of code is ridiculous. And how do you account for how much brain space it takes up. If the LLM lets someone bang out useful code in between meetings all day, vs needing some truly quiet time for some deep deep work to properly get into the right zone... how do you metricize that and show, what, lines of code and interruptions per hour on a graph to have a line that goes up and to the right? It turns out that it's actually a really hard problem to measure, so, even for a trillion dollar industry, it is hard. What did you think of that video?

victorbjorklund 2y ago

You should probably compare the same dev using AI vs not using AI. Otherwise you can use that argument for anything. Notepad is a better IDE than emacs if we compare a really good dev using notepad vs a shitty dev using emacs.

jcgrillo 2y ago
Ok, if I notice other devs suddenly getting 5x as productive I'll give it a try, but so far no such effect has been demonstrated. It seems like a pretty straightforward research question, and you'd think if there was any demonstrable effect the companies selling these things would use such research to market their products. So where is it?
- fragmede 2y ago
  There's a dead body lying there but you won't believe they're actually dead until someone creates a video reenactment of the killer stabbing them and shoves that video in your face? Speaking of videos, https://youtu.be/opi1s_5Dm-c https://youtu.be/opi1s_5Dm-c is probably below your level and it's kinda long.
  - jcgrillo 2y ago
    I don't get it. What is the dead body thing about?
    - fragmede 2y ago
      There are people building things and making money using these new techniques, but because they haven't stopped to produce research studies and white papers about their work, you want to presume it doesn't exist.
      - jcgrillo 2y ago
        Yes, without evidence there's no effect. It's telling that nowhere on cursor.ai's website do they appear to claim using their tool will actually make you, or crucially your team, more efficient.
        fragmede 2y ago
        evidencthe absence of evidence doesn't mean there's no effect
        jcgrillo 2y ago
        There is no trustworthy evidence that I'm aware of to support the claim that "using an LLM IDE makes developers 5x more effective". Do you disagree? Testimonials don't cut it, because individuals are prone to hysteria, delusions, and just making stuff up. All I'm asking for is a few trustworthy case studies showing unambiguously that it's worth my time and money to pay attention to this genai stuff. That shouldn't be hard for a trillion dollar industry, right? Like show me this isn't just hysterical hype. Don't tell me, show me.
        fragmede 2y ago
        What does it even mean to be 5x more effective? Because that's the thing. Testimonials is where we're at because how do you measure "more effective"? I'm not making any claim on a number, 5x 10x? .2x because you have to go back and fix the shitty code it wrote? We don't have a good way to measure programmer productivity. We all know lines of code is ridiculous. And how do you account for how much brain space it takes up. If the LLM lets someone bang out useful code in between meetings all day, vs needing some truly quiet time for some deep deep work to properly get into the right zone... how do you metricize that and show, what, lines of code and interruptions per hour on a graph to have a line that goes up and to the right? It turns out that it's actually a really hard problem to measure, so, even for a trillion dollar industry, it is hard. What did you think of that video?