3 ms·

> I'm happy to let people think that AI does not yield productivity gains. vs. --- start quote --- In a randomised controlled trial – the first of its kind –

by troupo 1y ago

> I'm happy to let people think that AI does not yield productivity gains. vs. --- start quote --- In a randomised controlled trial – the first of its kind – experienced computer programmers could use AI tools to help them write code. --- end quote --- Your quote is very representative of the magical wishful thinking most people have about AI: https://dmitriid.com/everything-around-llms-is-still-magical-and-wishful-thinking https://dmitriid.com/everything-around-llms-is-still-magical...

refulgentis 1y ago
(not op) Gosh, I was conflicted, then you pulled out that sentence and I was convinced. :) Alternatively: When faced with a contradiction, first, check your premises. I don't want to belabor the point too much, there's little common ground if we're at all or nothing thinking - "the study proved AI is net-negative because of this pull quote" isn't discussion.
simonw 1y ago
"Your quote is very representative of the magical wishful thinking most people have about AI" Your comment here is very representative of how quickly people who are AI skeptics will jump on anything that supports their skepticism.
- ofjcihen 1y ago
  I don’t think the snark is warranted here. The person you’re responding to is linking an article dealing with that issue and the title is what they mentioned. Edit: SimonW? Really? I didn’t see the name but I didn’t expect you to be like that.
  - refulgentis 1y ago
    Not OP, not sure what you mean but curious. - Snark? - Is "the issue" that anyone who claims any productivity gain is using magical thinking? - How does the linked article "deal with" "the issue"? - What title did they mention? - What did they link to that has that title? > Edit: SimonW? Really? I didn’t see the name but I didn’t expect you to be like that. Like what? I think you're getting a bit emotional & personal here, I don't read anything remotely inappropriate into Simon's comment. Been here 15 years. OP's was odd for HN in that it admits 0 argument: if you think you have productivity gains, it's magical thinking.
    - ofjcihen 1y ago
      So I understand how people can get emotional about this topic but really quick let’s calm down and reevaluate what I said. My comment was in response to Simon’s reply to a user who posted an article. The title of the article they posted addresses magical thinking in AI. Now whether that’s an opinion you share or not is not the point. Simon responded as if the user was only calling any perceived gains from AI as magical thinking which is not the case. I’ll let you come back to that when you feel like it. Altogether, though it’s just disappointing to see someone who’s work I read often jumping to an emotional response when it’s not warranted.
      - fragmede 1y ago
        Never meet your heros. Then again, if the worst is a bit of emotionally charged snark, of which we are all guilty of at some point, I think I'll live.
        ofjcihen 1y ago
        Very true and to be fair Simon’s response with his perspective is valid and understandable.
  - simonw 1y ago
    I stand by what I said. I don't think the response from troupo that nayshins's personal experience is invalidated by a "randomised controlled trial" was well argued, so I imitated what I saw as their snarky wording with my own reworded version of it. I do take the "AI isn't actually a productivity boost" thing a little bit personally these days, because the logical conclusion for that is that I've been deluding myself for the past two years and I'm effectively a victim of "magical thinking". (That said, I did actually go to delete my comment shortly after posting it because I didn't think it added anything to the conversation, but it had already drawn a reply so I left it there.)
    - ofjcihen 1y ago
      That’s totally fair and I get it. Working in security I often feel the same way and let’s be fair in the grand scheme of things it’s not that big of a deal.
    - troupo 1y ago
      > because the logical conclusion for that is that I've been deluding myself for the past two years and I'm effectively a victim of "magical thinking". You may just as well have. I, for one, am absolutely ready to re-evaluate any and all approaches I have with AI to see if I am actually more productive or not. But moreover, your own singular experience with your own code and projects may make you more productive. We don't know if it does because we don't have a baseline against which to measure. But even moreover over that moreover is that we don't even have a question "does a single senior engineer's experience with his own code and approaches can be generalised over the entire population of programmers?" Skeptics say: no (and now have some proof of that). Optimists loudly say: yes, of course, and dismiss everyone who dares contradict out of hand.
- troupo 1y ago
  The person above literally pitches an unsupported belief against a study (however flawed it may be). If that doesn't support my skepticism, I don't know what does.
  - simonw 1y ago
    When you've lived with AI boosting your productivity for a year or more it's pretty easy to believe your own experience over even a well-constructed "randomised controlled trial".
    - hooverd 1y ago
      When you've lived with -AI- stimulants boosting your productivity for a year or more it's pretty easy to believe your own experience over even a well-constructed "randomised controlled trial". we don't demand every developer pop Adderall though
      - NoGravitas 1y ago
        Don't give the management any ideas.
    - troupo 1y ago
      > When you've lived with AI boosting your productivity for a year or more it's pretty easy to believe your own experience over even a well-constructed "randomised controlled trial". Me: The person above literally pitches an unsupported belief against a study You: it's pretty easy to believe your own experience over even a well-constructed "randomised controlled trial". Really? Really?!! As for "boosting your productivity", it's also what I'm talking about in the article I linked: --- start quote --- For every description of how LLMs work or don't work we know only some, but not all of the following: - Do we know which projects people work on? No - Do we know which codebases (greenfield, mature, proprietary etc.) people work on? No - Do we know the level of expertise the people have? No. Is the expertise in the same domain, codebase, language that they apply LLMs to? We don't know. - How much additional work did they have reviewing, fixing, deploying, finishing etc.? We don't know. Even if you have one person describing all of the above, you will not be able to compare their experience to anyone else's because you have no idea what others answer for any of those bullet points. --- end quote --- So what happens when we actually control and measure those variables? Wait, don't answer: "no, it's easier to believe yourself over a study". See? Skeptics don't even have to "jump on anything that supports their skepticism." Even you supply them with material.
- jgalt212 1y ago
  The converse is also true.
pydry 1y ago
ive watched a lot of people code with cursor, etc. and i noticed that they seem to get a rush when it occasionally does something amazing that more than offsets their disappointment when it (more often) screws up. the psychological effect reminds me a bit of slot machines, which provide you with enough intermittent wins to make you feel like you're winning while youre lose. I think this might be linked to that study that found experienced oss devs who thought they were faster when they were in actual fact 20% slower.

by troupo 1y ago

refulgentis 1y ago

(not op) Gosh, I was conflicted, then you pulled out that sentence and I was convinced. :) Alternatively: When faced with a contradiction, first, check your premises. I don't want to belabor the point too much, there's little common ground if we're at all or nothing thinking - "the study proved AI is net-negative because of this pull quote" isn't discussion.

simonw 1y ago

"Your quote is very representative of the magical wishful thinking most people have about AI" Your comment here is very representative of how quickly people who are AI skeptics will jump on anything that supports their skepticism.

ofjcihen 1y ago
I don’t think the snark is warranted here. The person you’re responding to is linking an article dealing with that issue and the title is what they mentioned. Edit: SimonW? Really? I didn’t see the name but I didn’t expect you to be like that.
- refulgentis 1y ago
  Not OP, not sure what you mean but curious. - Snark? - Is "the issue" that anyone who claims any productivity gain is using magical thinking? - How does the linked article "deal with" "the issue"? - What title did they mention? - What did they link to that has that title? > Edit: SimonW? Really? I didn’t see the name but I didn’t expect you to be like that. Like what? I think you're getting a bit emotional & personal here, I don't read anything remotely inappropriate into Simon's comment. Been here 15 years. OP's was odd for HN in that it admits 0 argument: if you think you have productivity gains, it's magical thinking.
  - ofjcihen 1y ago
    So I understand how people can get emotional about this topic but really quick let’s calm down and reevaluate what I said. My comment was in response to Simon’s reply to a user who posted an article. The title of the article they posted addresses magical thinking in AI. Now whether that’s an opinion you share or not is not the point. Simon responded as if the user was only calling any perceived gains from AI as magical thinking which is not the case. I’ll let you come back to that when you feel like it. Altogether, though it’s just disappointing to see someone who’s work I read often jumping to an emotional response when it’s not warranted.
    - fragmede 1y ago
      Never meet your heros. Then again, if the worst is a bit of emotionally charged snark, of which we are all guilty of at some point, I think I'll live.
      - ofjcihen 1y ago
        Very true and to be fair Simon’s response with his perspective is valid and understandable.
- simonw 1y ago
  I stand by what I said. I don't think the response from troupo that nayshins's personal experience is invalidated by a "randomised controlled trial" was well argued, so I imitated what I saw as their snarky wording with my own reworded version of it. I do take the "AI isn't actually a productivity boost" thing a little bit personally these days, because the logical conclusion for that is that I've been deluding myself for the past two years and I'm effectively a victim of "magical thinking". (That said, I did actually go to delete my comment shortly after posting it because I didn't think it added anything to the conversation, but it had already drawn a reply so I left it there.)
  - ofjcihen 1y ago
    That’s totally fair and I get it. Working in security I often feel the same way and let’s be fair in the grand scheme of things it’s not that big of a deal.
  - troupo 1y ago
    > because the logical conclusion for that is that I've been deluding myself for the past two years and I'm effectively a victim of "magical thinking". You may just as well have. I, for one, am absolutely ready to re-evaluate any and all approaches I have with AI to see if I am actually more productive or not. But moreover, your own singular experience with your own code and projects may make you more productive. We don't know if it does because we don't have a baseline against which to measure. But even moreover over that moreover is that we don't even have a question "does a single senior engineer's experience with his own code and approaches can be generalised over the entire population of programmers?" Skeptics say: no (and now have some proof of that). Optimists loudly say: yes, of course, and dismiss everyone who dares contradict out of hand.
troupo 1y ago
The person above literally pitches an unsupported belief against a study (however flawed it may be). If that doesn't support my skepticism, I don't know what does.
- simonw 1y ago
  When you've lived with AI boosting your productivity for a year or more it's pretty easy to believe your own experience over even a well-constructed "randomised controlled trial".
  - hooverd 1y ago
    When you've lived with -AI- stimulants boosting your productivity for a year or more it's pretty easy to believe your own experience over even a well-constructed "randomised controlled trial". we don't demand every developer pop Adderall though
    - NoGravitas 1y ago
      Don't give the management any ideas.
  - troupo 1y ago
    > When you've lived with AI boosting your productivity for a year or more it's pretty easy to believe your own experience over even a well-constructed "randomised controlled trial". Me: The person above literally pitches an unsupported belief against a study You: it's pretty easy to believe your own experience over even a well-constructed "randomised controlled trial". Really? Really?!! As for "boosting your productivity", it's also what I'm talking about in the article I linked: --- start quote --- For every description of how LLMs work or don't work we know only some, but not all of the following: - Do we know which projects people work on? No - Do we know which codebases (greenfield, mature, proprietary etc.) people work on? No - Do we know the level of expertise the people have? No. Is the expertise in the same domain, codebase, language that they apply LLMs to? We don't know. - How much additional work did they have reviewing, fixing, deploying, finishing etc.? We don't know. Even if you have one person describing all of the above, you will not be able to compare their experience to anyone else's because you have no idea what others answer for any of those bullet points. --- end quote --- So what happens when we actually control and measure those variables? Wait, don't answer: "no, it's easier to believe yourself over a study". See? Skeptics don't even have to "jump on anything that supports their skepticism." Even you supply them with material.
jgalt212 1y ago
The converse is also true.

pydry 1y ago

ive watched a lot of people code with cursor, etc. and i noticed that they seem to get a rush when it occasionally does something amazing that more than offsets their disappointment when it (more often) screws up. the psychological effect reminds me a bit of slot machines, which provide you with enough intermittent wins to make you feel like you're winning while youre lose. I think this might be linked to that study that found experienced oss devs who thought they were faster when they were in actual fact 20% slower.