5 ms·

Good for them to design and publish this - I doubt you'd see anything like this from the other labs. The loss of competency seems pretty obvious but it's good

Good for them to design and publish this - I doubt you'd see anything like this from the other labs. The loss of competency seems pretty obvious but it's good to have data. What is also interesting to me is that the AI assisted group accomplished the task a bit faster but it wasn't statistically significant. Which seems to align with other findings that AI can make you 'feel' like you're working faster but that perception isn't always matched by the reality. So you're trading learning and eroding competency for a productivity boost which isn't always there.

brookst 9mo ago
I wish they had attempted to measure product management skill. My hypothesis is that the AI users gained less in coding skill, but improved in spec/requirement writing skills. But there’s no data, so it’s just my speculation. Intuitively, I think AI is shifting entry level programmers to focus on expressing requirements clearly, which may not be all that bad of a thing.
- SJMG 9mo ago
  > I wish they had attempted to measure product management skill. We're definitely getting better at writing specs. The issue is the labor bottleneck is competent senior engineers, not juniors, not PMs, not box-and-arrow staff engineers. > I think AI is shifting entry level programmers to focus on expressing requirements clearly This is what the TDD advocates were saying years ago.
- empath75 9mo ago
  What AI development has done for my team is the following: Dramatically improved Jira usage -- better, more descriptive tickets with actionable user stories and clearly expressed requirements. Dramatically improved github PRs. Dramatically improved test coverage. Dramatically improved documentation, not just in code but in comments. Basically all _for free_, while at the same time probably doubling or tripling our pace at closing issues, including some issues in our backlog that had lingered for months because they were annoying and nobody felt like working on them, but were easy for claude to knock out.
  - theshrike79 8mo ago
    I keep describing this as the environmental protection meme, "but what if we make the world a better place - for nothing!" Even if AI goes away tomorrow, we'll still have better tooling, documentation and processes just because we HAD to implement them to use AIs more efficiently.
  - Jensson 9mo ago
    > Dramatically improved Jira usage -- better, more descriptive tickets with actionable user stories and clearly expressed requirements. Dramatically improved github PRs. Dramatically improved test coverage. Dramatically improved documentation, not just in code but in comments. > Basically all _for free_ Not for free, the cost is that all of those are now written by AI so not really vetted any longer. Or do you really think your team is just using AI for code?
  - WD-42 9mo ago
    I'd be willing to bet that your AI written issues, docs, etc look impressive initially but are extremely low signal to noise. You might be checking some boxes (docstrings, etc) but I do not envy anyone on your team that needs to actually read any of that stuff in the future to solve an actual problem.
    - thunky 9mo ago
      Right because developers are famous for their 100% perfect hand-crafted docs.
epolanski 9mo ago
> The loss of competency seems pretty obvious but it's good to have data That's not what the study says. It says that most users reflect your statement while there is a smaller % that benefits and learns more and faster. Generalizations are extremely dangerous. What the article says simply reflect that most people don't care that much and default to the path of least resistance, which is common every day knowledge, but we very well know this does not apply to everyone.
- AstroBen 9mo ago
  Relevant quote from their conclusion: > Among participants who use AI, we find a stark divide in skill formation outcomes between high-scoring interaction patterns (65%-86% quiz score) vs low-scoring interaction patterns (24%-39% quiz score). The high scorers only asked AI conceptual questions instead of code generation or asked for explanations to accompany generated code; these usage patterns demonstrate a high level of cognitive engagement. This is very much my experience. AI is incredibly useful as a personal tutor
  - ambicapter 9mo ago
    A personal tutor who you remain skeptical of, and constantly try to disprove in order to perfect your understanding.
    - epolanski 9mo ago
      I see it more of a replacement for Google and digging GitHub issues. It can also replace chats for 80% of questions. Not much as a tutor.
    - marcosdumay 9mo ago
      A tutor that can guide you through jargon and give you references. If "skepticism" is even something you have to think about, you are already outside of the optimum path.
      - nativeit 9mo ago
AstroBen 9mo ago
Interestingly if you look at the breakdown by years of experience, it shows the 1-3 year junior group being faster, 4+ years no difference I wonder if we're going to have a future where the juniors never gain the skills and experience to work well by themselves, and instead become entirely reliant on AI, assuming that's the only way
- pesus 9mo ago
  I think we're going to see a small minority of juniors who managed to ignore the hype/peer pressure/easy path and actually learned to code have a huge advantage over the others.
  - DrewADesign 9mo ago
    Which isn’t saying much if efficiency gains tank the demand for developers, which will then tank everybody’s salary. The actual efficiency gains are debatable, but even if we’re talking about a 20% gain, that could be a few FTEs for a small team.
shimman 9mo ago
It's research from a company that gains from selling said tools they researched. Why does it have to be repeated that this is a massive conflict of interests and until this "research" has been verified multiple times by parties with zero conflict of interests it's best to be highly skeptical of anything it claims? This is up there with believing tobacco companies health "research" from the 30s, 40s, 50s, 60s, 70s, 80s, and 90s.
- keeda 9mo ago
  I mean, they're literally pointing out the negative effects of AI-assisted coding? > We found that using AI assistance led to a statistically significant decrease in mastery. On a quiz that covered concepts they’d used just a few minutes before, participants in the AI group scored 17% lower than those who coded by hand, or the equivalent of nearly two letter grades. Using AI sped up the task slightly, but this didn’t reach the threshold of statistical significance. This also echoes other research from a few years ago that had similar findings: https://news.ycombinator.com/item?id=46822158 https://news.ycombinator.com/item?id=46822158
  - shimman 9mo ago
    Dude you falling for so obvious corpo-psyops is so sad. Tobacco companies literally published research that said cigarettes were dangerous too, that didn't stop them from lying to Congress and saying cigarettes weren't totally safe. Some of you are the reason why there needs to be a new luddite movement (fun fact, the luddites were completely correct in their movements; they fought against oppressive factory owners that treated their fellow humans terrible, smashing the very same machines they used themselves. Entrepreneurs were literally ushering in a new hell on Earth where their factors were killing so many orphans (because many people refused to work in such places originally, until forced by dying in the streets or dying from their labor in such places) they had to ship the bodies of children across towns to not draw suspicion). Until the entrepreneurs started killing them and convincing the king reagent to kill them with the state, they had massive support. Support so high that when suspected luddites were escaping from the "police" you could hear entire towns cheering them on helping them escape). People rightfully hate this stuff and you refuse to see, the evidence says it's terrible but hey let's still sell it anyway what's the worse that can happen?
cal_dent 9mo ago
Anthropic's way into regulatory capture seems to be to pretend they're the benevolent adults in the room. It'll probably work too.
austin-cheney 9mo ago
I agree with the Ray Dalio perspective on this. AI is not a creative force. It is only a different form of automation. So, the only value to AI is to get to know your habits. As an example have it write test cases in your code style so you don't have to. That is it. If you sucked before using AI you are going to suck with AI. The compounded problem there is that you won't see just how bad you suck at what you do, because AI will obscure your perspective through its output, like an echo chamber of stupid. You are just going to suck much faster and feel better about it. Think of it as steroids for Dunning-Kruger. https://www.youtube.com/shorts/0LeJ6xn35gc https://www.youtube.com/shorts/0LeJ6xn35gc https://www.youtube.com/shorts/vXecG_KajLI https://www.youtube.com/shorts/vXecG_KajLI
- psyclobe 9mo ago
  > If you sucked before using AI > you are going to suck with AI This.
  - whattheheckheck 9mo ago
    Its 2026 and people still post this. Instead of an upvote?

brookst 9mo ago

I wish they had attempted to measure product management skill. My hypothesis is that the AI users gained less in coding skill, but improved in spec/requirement writing skills. But there’s no data, so it’s just my speculation. Intuitively, I think AI is shifting entry level programmers to focus on expressing requirements clearly, which may not be all that bad of a thing.

SJMG 9mo ago
> I wish they had attempted to measure product management skill. We're definitely getting better at writing specs. The issue is the labor bottleneck is competent senior engineers, not juniors, not PMs, not box-and-arrow staff engineers. > I think AI is shifting entry level programmers to focus on expressing requirements clearly This is what the TDD advocates were saying years ago.
empath75 9mo ago
What AI development has done for my team is the following: Dramatically improved Jira usage -- better, more descriptive tickets with actionable user stories and clearly expressed requirements. Dramatically improved github PRs. Dramatically improved test coverage. Dramatically improved documentation, not just in code but in comments. Basically all _for free_, while at the same time probably doubling or tripling our pace at closing issues, including some issues in our backlog that had lingered for months because they were annoying and nobody felt like working on them, but were easy for claude to knock out.
- theshrike79 8mo ago
  I keep describing this as the environmental protection meme, "but what if we make the world a better place - for nothing!" Even if AI goes away tomorrow, we'll still have better tooling, documentation and processes just because we HAD to implement them to use AIs more efficiently.
- Jensson 9mo ago
  > Dramatically improved Jira usage -- better, more descriptive tickets with actionable user stories and clearly expressed requirements. Dramatically improved github PRs. Dramatically improved test coverage. Dramatically improved documentation, not just in code but in comments. > Basically all _for free_ Not for free, the cost is that all of those are now written by AI so not really vetted any longer. Or do you really think your team is just using AI for code?
- WD-42 9mo ago
  I'd be willing to bet that your AI written issues, docs, etc look impressive initially but are extremely low signal to noise. You might be checking some boxes (docstrings, etc) but I do not envy anyone on your team that needs to actually read any of that stuff in the future to solve an actual problem.
  - thunky 9mo ago
    Right because developers are famous for their 100% perfect hand-crafted docs.

epolanski 9mo ago

> The loss of competency seems pretty obvious but it's good to have data That's not what the study says. It says that most users reflect your statement while there is a smaller % that benefits and learns more and faster. Generalizations are extremely dangerous. What the article says simply reflect that most people don't care that much and default to the path of least resistance, which is common every day knowledge, but we very well know this does not apply to everyone.

AstroBen 9mo ago
Relevant quote from their conclusion: > Among participants who use AI, we find a stark divide in skill formation outcomes between high-scoring interaction patterns (65%-86% quiz score) vs low-scoring interaction patterns (24%-39% quiz score). The high scorers only asked AI conceptual questions instead of code generation or asked for explanations to accompany generated code; these usage patterns demonstrate a high level of cognitive engagement. This is very much my experience. AI is incredibly useful as a personal tutor
- ambicapter 9mo ago
  A personal tutor who you remain skeptical of, and constantly try to disprove in order to perfect your understanding.
  - epolanski 9mo ago
    I see it more of a replacement for Google and digging GitHub issues. It can also replace chats for 80% of questions. Not much as a tutor.
  - marcosdumay 9mo ago
    A tutor that can guide you through jargon and give you references. If "skepticism" is even something you have to think about, you are already outside of the optimum path.
    - nativeit 9mo ago

AstroBen 9mo ago

Interestingly if you look at the breakdown by years of experience, it shows the 1-3 year junior group being faster, 4+ years no difference I wonder if we're going to have a future where the juniors never gain the skills and experience to work well by themselves, and instead become entirely reliant on AI, assuming that's the only way

pesus 9mo ago
I think we're going to see a small minority of juniors who managed to ignore the hype/peer pressure/easy path and actually learned to code have a huge advantage over the others.
- DrewADesign 9mo ago
  Which isn’t saying much if efficiency gains tank the demand for developers, which will then tank everybody’s salary. The actual efficiency gains are debatable, but even if we’re talking about a 20% gain, that could be a few FTEs for a small team.

shimman 9mo ago

It's research from a company that gains from selling said tools they researched. Why does it have to be repeated that this is a massive conflict of interests and until this "research" has been verified multiple times by parties with zero conflict of interests it's best to be highly skeptical of anything it claims? This is up there with believing tobacco companies health "research" from the 30s, 40s, 50s, 60s, 70s, 80s, and 90s.

keeda 9mo ago
I mean, they're literally pointing out the negative effects of AI-assisted coding? > We found that using AI assistance led to a statistically significant decrease in mastery. On a quiz that covered concepts they’d used just a few minutes before, participants in the AI group scored 17% lower than those who coded by hand, or the equivalent of nearly two letter grades. Using AI sped up the task slightly, but this didn’t reach the threshold of statistical significance. This also echoes other research from a few years ago that had similar findings: https://news.ycombinator.com/item?id=46822158 https://news.ycombinator.com/item?id=46822158
- shimman 9mo ago
  Dude you falling for so obvious corpo-psyops is so sad. Tobacco companies literally published research that said cigarettes were dangerous too, that didn't stop them from lying to Congress and saying cigarettes weren't totally safe. Some of you are the reason why there needs to be a new luddite movement (fun fact, the luddites were completely correct in their movements; they fought against oppressive factory owners that treated their fellow humans terrible, smashing the very same machines they used themselves. Entrepreneurs were literally ushering in a new hell on Earth where their factors were killing so many orphans (because many people refused to work in such places originally, until forced by dying in the streets or dying from their labor in such places) they had to ship the bodies of children across towns to not draw suspicion). Until the entrepreneurs started killing them and convincing the king reagent to kill them with the state, they had massive support. Support so high that when suspected luddites were escaping from the "police" you could hear entire towns cheering them on helping them escape). People rightfully hate this stuff and you refuse to see, the evidence says it's terrible but hey let's still sell it anyway what's the worse that can happen?

cal_dent 9mo ago

Anthropic's way into regulatory capture seems to be to pretend they're the benevolent adults in the room. It'll probably work too.

austin-cheney 9mo ago

I agree with the Ray Dalio perspective on this. AI is not a creative force. It is only a different form of automation. So, the only value to AI is to get to know your habits. As an example have it write test cases in your code style so you don't have to. That is it. If you sucked before using AI you are going to suck with AI. The compounded problem there is that you won't see just how bad you suck at what you do, because AI will obscure your perspective through its output, like an echo chamber of stupid. You are just going to suck much faster and feel better about it. Think of it as steroids for Dunning-Kruger. https://www.youtube.com/shorts/0LeJ6xn35gc https://www.youtube.com/shorts/0LeJ6xn35gc https://www.youtube.com/shorts/vXecG_KajLI https://www.youtube.com/shorts/vXecG_KajLI

psyclobe 9mo ago
> If you sucked before using AI > you are going to suck with AI This.
- whattheheckheck 9mo ago
  Its 2026 and people still post this. Instead of an upvote?