4 ms·

Wow these are extremely interesting results, specially this part: > This gap between perception and reality is striking: developers expected AI to speed them u

by pera 1y ago

Wow these are extremely interesting results, specially this part: > This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%. I wonder what could explain such large difference between estimation/experience vs reality, any ideas? Maybe our brains are measuring mental effort and distorting our experience of time?

alfalfasprout 1y ago
I would speculate that it's because there's been a huge concerted effort to make people want to believe that these tools are better than they are. The "economic experts" and "ml experts" are in many cases effectively the same group-- companies pushing AI coding tools have a vested interest in people believing they're more useful than they are. Executives take this at face value and broadly promise major wins. Economic experts take this at face value and use this for their forecasts. This propagates further, and now novices and casual individuals begin to believe in the hype. Eventually, as an experienced engineer it moves the "baseline" expectation much higher. Unfortunately this is very difficult to capture empirically.
longwave 1y ago
I also wonder how many of the numerous AI proponents in HN comments are subject to the same effect. Unless they are truly measuring their own performance, is AI really making them more productive?
- malfist 1y ago
  How would you even measure your own performance? You can go and redo something, forgetting everything you did along the way the first time
  - jwrallie 1y ago
    You could go the same way as the study, flip a coin to use AI or not, write down the task you just did, the time you thought the task took you and the actual clock time. Repeat and self-evaluate.
    - malfist 1y ago
      Sample size of 16 is already hard enough to draw conclusions from. Sample size of 1 is even worse.
      - JelteF 1y ago
        It's the most representative sample size if you're interested in your own performance though. I really don't care if other people are more productive with AI, if I'm the outlier that's not then I'd want to know.
      - sarchertech 1y ago
        Sample of 16 is plenty if the effect is big enough. It’s also not a sample size of 1, it’s a sample size of however many tasks you do because you don’t care about measuring the effect AI has on anyone but yourself if you’re trying to discern how it impacts you.
chamomeal 1y ago
It’s funny cause I sometimes have the opposite experience. I tried to use Claude code today to make a demo app to show off a small library I’m working on. I needed it to set up some very boilerplatey example app stuff. It was fun to watch, it’s super polished and sci-fi-esque. But after 15 minutes I felt braindead and was bored out of my mind lol
evanelias 1y ago
Here's a scary thought, which I'm admittedly basing on absolutely nothing scientific: What if agentic coding sessions are triggering a similar dopamine feedback loop as social media apps? Obviously not to the same degree as social media apps, I mean coding for work is still "work"... but there's maybe some similarity in getting iterative solutions from the agent, triggering something in your brain each time, yes? If that was the case, wouldn't we expect developers to have an overly positive perception of AI because they're literally becoming addicted to it?
- EarthLaunch 1y ago
  > The LLMentalist Effect: how chat-based Large Language Models replicate the mechanisms of a psychic’s con https://softwarecrisis.dev/letters/llmentalist/ https://softwarecrisis.dev/letters/llmentalist/ Plus there's a gambling mechanic: Push the button, sometimes get things for free.
  - lll-o-lll 1y ago
    This is very interesting and disturbing. We are outsourcing our decision making to an algorithmic “Mentalist” and will reap a terrible reward. I need to ween myself off the comforting teat of the chatbot psychic.
- csherratt 1y ago
  That's my suspicion to. My issue with this being a 'negative' thing is that I'm not sure it is. It works off of the same hunting / foraging instincts that keep us alive. If you feel addiction to something positive, it is bad? Social media is negative because it addicts you to mostly low quality filler content. Content that doesn't challenge you. You are reading shit posts instead of reading a book or doing something with better for you in the long run. One could argue that's true for AI, but I'm not confident enough to make such a statement.
  - evanelias 1y ago
    The study found AI caused a "significant slowdown" in developer efficiency though, so that doesn't seem positive!
fiddlerwoaroof 1y ago
I think just about every developer hack turns out this way: static vs dynamic types; keyboard shortcuts vs mice; etc. But I think it’s also possible to over-interpret these findings: using the tools that make your work enjoyable has important second-order effects even if they aren’t the productivity silver bullet everyone claims they are.
- robenkleene 1y ago
  This same conclusion has been actually observed practically verbatim about the keyboard vs. mouse https://www.asktog.com/TOI/toi06KeyboardVMouse1.html https://www.asktog.com/TOI/toi06KeyboardVMouse1.html from the post: - Test subjects consistently report that keyboarding is faster than mousing. - The stopwatch consistently proves mousing is faster than keyboarding.
  - fiddlerwoaroof 1y ago
    Yeah, that was one of the pieces of research I was thinking about. Another is this review of the evidence, such as it is, for static types: https://danluu.com/empirical-pl/ https://danluu.com/empirical-pl/ So far, Fred Brook’s “No Silver Bullet” remains undefeated.
qingcharles 1y ago
Part of it is that I feel I don't have to put as much mental energy into the coding part. I use my mental energy on the design and ideas, then kinda breeze through the coding now with AI at a much lower mental energy state than I would have when I was typing every single character of every line.
rsynnott 1y ago
> I wonder what could explain such large difference between estimation/experience vs reality, any ideas? This bit I wasn't at all surprised by, because this is _very common_. People who are doing a [magic thing] which they believe in often claim that it is improving things even where it empirically isn't; very, very common with fad diets and exercise regimens, say. You really can't trust subjects' claims of efficacy of something that's being tested on them, or that they're testing on themselves. And particularly for LLM tools, there is this strong sense amongst many fans that they are The Future, that anyone who doesn't get onboard is being Left Behind, and so forth. I'd assume a lot of users aren't thinking particularly rationally about them.

by pera 1y ago

alfalfasprout 1y ago

I would speculate that it's because there's been a huge concerted effort to make people want to believe that these tools are better than they are. The "economic experts" and "ml experts" are in many cases effectively the same group-- companies pushing AI coding tools have a vested interest in people believing they're more useful than they are. Executives take this at face value and broadly promise major wins. Economic experts take this at face value and use this for their forecasts. This propagates further, and now novices and casual individuals begin to believe in the hype. Eventually, as an experienced engineer it moves the "baseline" expectation much higher. Unfortunately this is very difficult to capture empirically.

longwave 1y ago

I also wonder how many of the numerous AI proponents in HN comments are subject to the same effect. Unless they are truly measuring their own performance, is AI really making them more productive?

malfist 1y ago
How would you even measure your own performance? You can go and redo something, forgetting everything you did along the way the first time
- jwrallie 1y ago
  You could go the same way as the study, flip a coin to use AI or not, write down the task you just did, the time you thought the task took you and the actual clock time. Repeat and self-evaluate.
  - malfist 1y ago
    Sample size of 16 is already hard enough to draw conclusions from. Sample size of 1 is even worse.
    - JelteF 1y ago
      It's the most representative sample size if you're interested in your own performance though. I really don't care if other people are more productive with AI, if I'm the outlier that's not then I'd want to know.
    - sarchertech 1y ago
      Sample of 16 is plenty if the effect is big enough. It’s also not a sample size of 1, it’s a sample size of however many tasks you do because you don’t care about measuring the effect AI has on anyone but yourself if you’re trying to discern how it impacts you.

chamomeal 1y ago

It’s funny cause I sometimes have the opposite experience. I tried to use Claude code today to make a demo app to show off a small library I’m working on. I needed it to set up some very boilerplatey example app stuff. It was fun to watch, it’s super polished and sci-fi-esque. But after 15 minutes I felt braindead and was bored out of my mind lol

evanelias 1y ago

Here's a scary thought, which I'm admittedly basing on absolutely nothing scientific: What if agentic coding sessions are triggering a similar dopamine feedback loop as social media apps? Obviously not to the same degree as social media apps, I mean coding for work is still "work"... but there's maybe some similarity in getting iterative solutions from the agent, triggering something in your brain each time, yes? If that was the case, wouldn't we expect developers to have an overly positive perception of AI because they're literally becoming addicted to it?

EarthLaunch 1y ago
> The LLMentalist Effect: how chat-based Large Language Models replicate the mechanisms of a psychic’s con https://softwarecrisis.dev/letters/llmentalist/ https://softwarecrisis.dev/letters/llmentalist/ Plus there's a gambling mechanic: Push the button, sometimes get things for free.
- lll-o-lll 1y ago
  This is very interesting and disturbing. We are outsourcing our decision making to an algorithmic “Mentalist” and will reap a terrible reward. I need to ween myself off the comforting teat of the chatbot psychic.
csherratt 1y ago
That's my suspicion to. My issue with this being a 'negative' thing is that I'm not sure it is. It works off of the same hunting / foraging instincts that keep us alive. If you feel addiction to something positive, it is bad? Social media is negative because it addicts you to mostly low quality filler content. Content that doesn't challenge you. You are reading shit posts instead of reading a book or doing something with better for you in the long run. One could argue that's true for AI, but I'm not confident enough to make such a statement.
- evanelias 1y ago
  The study found AI caused a "significant slowdown" in developer efficiency though, so that doesn't seem positive!

fiddlerwoaroof 1y ago

I think just about every developer hack turns out this way: static vs dynamic types; keyboard shortcuts vs mice; etc. But I think it’s also possible to over-interpret these findings: using the tools that make your work enjoyable has important second-order effects even if they aren’t the productivity silver bullet everyone claims they are.

robenkleene 1y ago
This same conclusion has been actually observed practically verbatim about the keyboard vs. mouse https://www.asktog.com/TOI/toi06KeyboardVMouse1.html https://www.asktog.com/TOI/toi06KeyboardVMouse1.html from the post: - Test subjects consistently report that keyboarding is faster than mousing. - The stopwatch consistently proves mousing is faster than keyboarding.
- fiddlerwoaroof 1y ago
  Yeah, that was one of the pieces of research I was thinking about. Another is this review of the evidence, such as it is, for static types: https://danluu.com/empirical-pl/ https://danluu.com/empirical-pl/ So far, Fred Brook’s “No Silver Bullet” remains undefeated.

qingcharles 1y ago

Part of it is that I feel I don't have to put as much mental energy into the coding part. I use my mental energy on the design and ideas, then kinda breeze through the coding now with AI at a much lower mental energy state than I would have when I was typing every single character of every line.

rsynnott 1y ago

> I wonder what could explain such large difference between estimation/experience vs reality, any ideas? This bit I wasn't at all surprised by, because this is _very common_. People who are doing a [magic thing] which they believe in often claim that it is improving things even where it empirically isn't; very, very common with fad diets and exercise regimens, say. You really can't trust subjects' claims of efficacy of something that's being tested on them, or that they're testing on themselves. And particularly for LLM tools, there is this strong sense amongst many fans that they are The Future, that anyone who doesn't get onboard is being Left Behind, and so forth. I'd assume a lot of users aren't thinking particularly rationally about them.