6 ms·

> even at the highest levels of skill, it's not just a juniors-only phenomenon AI has the most effect for people with less experience or low performance. It ha

> even at the highest levels of skill, it's not just a juniors-only phenomenon AI has the most effect for people with less experience or low performance. It has less of an effect for people on the high end. It is indeed closing the skill gap and it does so by elevating the lower side of it. This is important to know because it helps explain why people react as they do. Those who feel the most lift will be vocal about AI being good while those that don't are confused by anyone thinking AI is helpful at all. It is not common for people on the high skill side to experience a big lift except for when they use AI for the tedious stuff that they don't really want to do. This is a sweetspot because all of the competence is there, but the willingness to do the work is not. I have heard Dr Lilach Mollick, dir of Pedagogy at Wharton, say this has been shown numerous times. People who follow her husband, Ethan, are probably aware already.

CuriouslyC 1y ago
So, basically, you think all the pro-AI folks are "bad," and defensive because they feel like anti-AI folks are attacking the thing that makes them not bad? Hard to want to engage with that. I'll bite anyhow. AI is very, very good at working at short length-scales. It tends to be worse at working at longer length-scales (Gemini is a bit of an outlier here but even so, it holds). People who are hyper-competent/elite-skill in their domain who achieve force multiplication with gen-AI understand this, and know how to decompose challenging long length-scale problems into a number of smaller short-length scale problems efficiently. This isomorphic transform allows AI to tackle the original problem in a way that it's maximally efficient at, thus side-stepping their inherent weaknesses. You can think of this sort of like mathematical transformations that make data analysis easier.
- mewpmewp2 1y ago
  Yeah, exactly, shorter files, good context for the AI agents, good rules for the AI agents how to navigate around the codebase, reuse functions, components, keep everything always perfectly organized, with no effort from the person themselves. It is truly amazing to witness. No people can match that ability to reuse and organize.
- johnnyanmac 1y ago
  >So, basically, you think all the pro-AI folks are "bad," and defensive because they feel like anti-AI folks are attacking the thing that makes them not bad? I break the pro-AI crowd into 3 main categories and 2 sub categories: 1. those who don't really know how to code, but AI lets them output something more than what they could do on their own. This seems to be what the GP is focused on 2. The ones who can code but are financially invested to hype up the bubble. Fairly self explanatory; the market is rough and if you're getting paid the big bucks to evangelize, it's clear where the interests lie. 3. Executives and product teams that have no actual engagement with AI, but know bringing it up excites investors. a hybrid of 1 and 2, but they aren't necessarily pretending they use it themselves. It's the latest means to an end (the end being money). and then the smaller sects: 1. those who genuinely feel AI is the future and is simply prepping for it and trying to adapt their workflow and knowledged based around it. They may feel it can already replace people, or may feel it's a while out but progressing that way. These are probable the most honest party, but I personally feel they miss a critical aspect: what is used currently as the backbone for AI may radically change by the time it is truly viable. 2. those who are across the spectrum of AI, but see it as a means to properly address the issue of copyright. If AI wins, they get their true goal of being able to yoink many more properties without regulations to worry about. >People who are hyper-competent/elite-skill in their domain who achieve force multiplication with gen-AI understand this, are their real examples of this? The main issue I see is that people seem to judge "competency" based on speed and output. But not on the quality, maintainability, nor conciseness of such output. If we just needed engineers to slap together something that "works", we could be "more productive".
  - CuriouslyC 1y ago
    Well, just to give you context on my position, because I don't feel I fit into any of those molds: I was already a very high performer before AI, leading teams, aligning product vision and technical capabilities, architecting systems and implementing at top-of-stack velocity. I have been involved in engineering around AI/ML since 2008, so I have pretty good understanding of the complexities/inconsistencies of model behavior. When I observed the ability of GPT3.5 to often generate working (if poorly written, in general) code, I knew this was a powerful tool that would eventually totally reshape development once it matured, but that I had to understand its capabilities and non-uniform expertise boundary to take advantage of its strengths without having to suffer its weaknesses. I basically threw myself fully into mastering the "art" of using LLMs, both in terms of prompting and knowing when/how to use them, and while I saw immediate gains, it wasn't until Gemini Pro 2.5 that I saw the capabilities in place for a fully agentic workflow. I've been actively polishing my agentic workflow since Gemini 2.5's release, and now I'm at the point where I write less than 10% of my own code. Overall my hand written code is still significantly "neater/tighter" than that produced by LLMs, but I'm ok with the LLM nailing the high level patterns I outline and being "good enough" (which I encourage via detailed system prompts and enforce via code review, though I often have AI rewrite its own code given my feedback rather than manually edit it). I liken it to assembly devs who could crush the compiler in performance (not as much of a thing now in general, but it used to be), who still choose to write most of the system in c/c++ and only implement the really hot loops in assembly because that's just the most efficient way to work.
mrweasel 1y ago
> It is indeed closing the skill gap and it does so by elevating the lower side of it. That's my "criticism", it's not closing the skill gap. Your skills haven't change, your output has. If you're using AI conservatively I'd say you're right, it can remove all the tedious work, which is great, but you'll still need to check that it's correct. I'm more and more coming to the idea that for e.g. some coding jobs, CoPilot, Claude, whatever can be pretty helpful. I don't need to write a generic API call, handle all the error codes and hook up messages to the user, the robot can do that. I'll check and validate the code anyway. Where I'm still not convinced is for communicating with other humans. Writing is hard, communication is worse. If you still have basic spelling errors, after decades of using a spellchecker, I doubt that your ability to communicate clearly will change even with an LLM helping you. Same with arts. If you can't draw, no amount of prompting is going to change that. Which is fine, if you only care about the output, but you still don't have the skills. My concern is the uncritical application of LLMs to all aspects of peoples daily life. If you can use an LLM to do your job faster, fine. If you can't do it without an LLM, you shouldn't be doing it with one.
- az09mugen 1y ago
  "If you can use an LLM to do your job faster, fine. If you can't do it without an LLM, you shouldn't be doing it with one." This. People need to take responsability for what they produce. It's too easy and especially irresponsible to delegate blindly everything to AI.
surgical_fire 1y ago
> AI has the most effect for people with less experience or low performance. It has less of an effect for people on the high end. I actually think that it benefits high performance workers as AI can do a lot of heavy lifting that frees them to focus on things where their skills make a difference. Also, for less skilled or less experienced developers, they will have a harder time spotting the mistakes and inconsistencies generated by AI. This can actually become a productivity sink.
- windows2020 1y ago
  Reading other people's code is often more challenging than writing it yourself.
  - mewpmewp2 1y ago
    With AI code I find it very easy to understand as I prompted it, I know what to expect, I know what it will likely do. Far easier than other people code.
- jessoteric 1y ago
  the main issue is that you end up looking down the barrel of begging claude, for the fifth time this session, to do it right- or just do it yourself in half the total time you've wasted so far. at least, this is what i typically end up with.
  - surgical_fire 1y ago
    Typically, I've been asking it to do "heavy lifting" for me. It generally generates defective code, but it doesn't really matter all that much, it is still useful that it is mostly right, and I only need to make a few adjustments. It saves me a lot of typing. Would I pay for it? Probably not. But it is included in my IntelliJ subscription, so why not? It is there already.
jaredklewis 1y ago
> I have heard Dr Lilach Mollick, dir of Pedagogy at Wharton, say this has been shown numerous times. People who follow her husband, Ethan, are probably aware already. I'd be curious to see the sources. Basically every study I have ever read making some claim about programming (efficacy of IDEs, TDD, static typing, pair programming, formal CS education, ai assistants, etc...) has been a house of cards that falls apart with even modest prodding. They are usually premised on one or more inherently flawed metrics like number of github issues, LoC or whatever. That would be somewhat forgivable since there are not really any good metrics to go on, but then the studies invariably make only a perfunctory effort to disentangle even the most obvious of confounding variables, making all the results kind of pointless. Would be happy if anyone here knew of good papers that would change my mind on this point.
- nyarlathotep_ 1y ago
  > Basically every study I have ever read making some claim about programming (efficacy of IDEs, TDD, static typing, pair programming, formal CS education, ai assistants, etc...) has been a house of cards that falls apart with even modest prodding. Isn't this true about most things in software? I mean is there anything quantifiable about "microservices" vs "monolith"? Test-driven development, containers, whatever? I mean all of these things are in some way good, in some contexts, but it seems impossible to quantify benefits of any of them. I'm a believer that most decisions made in software are somewhat arbitrary, driven by trends and popularity and it seems like little effort is expended to come to overarching, data-backed results. When they are, they're rare and, like you said, fall apart under investigation or scrutiny. I've always found this strange about software in general. Even every time there's a "we rewrote $THING in $NEW_LANG" and it improved memory use/speed/latency whatever, there's a chorus of (totally legitimate) criticism and inquiry about how things were measured, what attempts were made to optimize the original solutions, if changes were made along the way outside of the language choice that impacted performance etc etc.
  - jaredklewis 1y ago
    I think we agree. To be clear I am not arguing that tools and practices like TDD, microservices, ai assistants, and so on have no effect. They almost certainly have an effect (good or bad). It’s just the unfortunate reality that quantitatively measuring these effects in a meaningful way seems to basically be impossible (or at least I’ve never see it done). With enough resources I can’t think of any reason it shouldn’t be possible, but apparently those resources are not available because there are no good studies on these topics. Thus my skepticism of the “studies” referenced earlier in the thread.

CuriouslyC 1y ago

So, basically, you think all the pro-AI folks are "bad," and defensive because they feel like anti-AI folks are attacking the thing that makes them not bad? Hard to want to engage with that. I'll bite anyhow. AI is very, very good at working at short length-scales. It tends to be worse at working at longer length-scales (Gemini is a bit of an outlier here but even so, it holds). People who are hyper-competent/elite-skill in their domain who achieve force multiplication with gen-AI understand this, and know how to decompose challenging long length-scale problems into a number of smaller short-length scale problems efficiently. This isomorphic transform allows AI to tackle the original problem in a way that it's maximally efficient at, thus side-stepping their inherent weaknesses. You can think of this sort of like mathematical transformations that make data analysis easier.

mewpmewp2 1y ago
Yeah, exactly, shorter files, good context for the AI agents, good rules for the AI agents how to navigate around the codebase, reuse functions, components, keep everything always perfectly organized, with no effort from the person themselves. It is truly amazing to witness. No people can match that ability to reuse and organize.
johnnyanmac 1y ago
>So, basically, you think all the pro-AI folks are "bad," and defensive because they feel like anti-AI folks are attacking the thing that makes them not bad? I break the pro-AI crowd into 3 main categories and 2 sub categories: 1. those who don't really know how to code, but AI lets them output something more than what they could do on their own. This seems to be what the GP is focused on 2. The ones who can code but are financially invested to hype up the bubble. Fairly self explanatory; the market is rough and if you're getting paid the big bucks to evangelize, it's clear where the interests lie. 3. Executives and product teams that have no actual engagement with AI, but know bringing it up excites investors. a hybrid of 1 and 2, but they aren't necessarily pretending they use it themselves. It's the latest means to an end (the end being money). and then the smaller sects: 1. those who genuinely feel AI is the future and is simply prepping for it and trying to adapt their workflow and knowledged based around it. They may feel it can already replace people, or may feel it's a while out but progressing that way. These are probable the most honest party, but I personally feel they miss a critical aspect: what is used currently as the backbone for AI may radically change by the time it is truly viable. 2. those who are across the spectrum of AI, but see it as a means to properly address the issue of copyright. If AI wins, they get their true goal of being able to yoink many more properties without regulations to worry about. >People who are hyper-competent/elite-skill in their domain who achieve force multiplication with gen-AI understand this, are their real examples of this? The main issue I see is that people seem to judge "competency" based on speed and output. But not on the quality, maintainability, nor conciseness of such output. If we just needed engineers to slap together something that "works", we could be "more productive".
- CuriouslyC 1y ago
  Well, just to give you context on my position, because I don't feel I fit into any of those molds: I was already a very high performer before AI, leading teams, aligning product vision and technical capabilities, architecting systems and implementing at top-of-stack velocity. I have been involved in engineering around AI/ML since 2008, so I have pretty good understanding of the complexities/inconsistencies of model behavior. When I observed the ability of GPT3.5 to often generate working (if poorly written, in general) code, I knew this was a powerful tool that would eventually totally reshape development once it matured, but that I had to understand its capabilities and non-uniform expertise boundary to take advantage of its strengths without having to suffer its weaknesses. I basically threw myself fully into mastering the "art" of using LLMs, both in terms of prompting and knowing when/how to use them, and while I saw immediate gains, it wasn't until Gemini Pro 2.5 that I saw the capabilities in place for a fully agentic workflow. I've been actively polishing my agentic workflow since Gemini 2.5's release, and now I'm at the point where I write less than 10% of my own code. Overall my hand written code is still significantly "neater/tighter" than that produced by LLMs, but I'm ok with the LLM nailing the high level patterns I outline and being "good enough" (which I encourage via detailed system prompts and enforce via code review, though I often have AI rewrite its own code given my feedback rather than manually edit it). I liken it to assembly devs who could crush the compiler in performance (not as much of a thing now in general, but it used to be), who still choose to write most of the system in c/c++ and only implement the really hot loops in assembly because that's just the most efficient way to work.

mrweasel 1y ago

> It is indeed closing the skill gap and it does so by elevating the lower side of it. That's my "criticism", it's not closing the skill gap. Your skills haven't change, your output has. If you're using AI conservatively I'd say you're right, it can remove all the tedious work, which is great, but you'll still need to check that it's correct. I'm more and more coming to the idea that for e.g. some coding jobs, CoPilot, Claude, whatever can be pretty helpful. I don't need to write a generic API call, handle all the error codes and hook up messages to the user, the robot can do that. I'll check and validate the code anyway. Where I'm still not convinced is for communicating with other humans. Writing is hard, communication is worse. If you still have basic spelling errors, after decades of using a spellchecker, I doubt that your ability to communicate clearly will change even with an LLM helping you. Same with arts. If you can't draw, no amount of prompting is going to change that. Which is fine, if you only care about the output, but you still don't have the skills. My concern is the uncritical application of LLMs to all aspects of peoples daily life. If you can use an LLM to do your job faster, fine. If you can't do it without an LLM, you shouldn't be doing it with one.

az09mugen 1y ago
"If you can use an LLM to do your job faster, fine. If you can't do it without an LLM, you shouldn't be doing it with one." This. People need to take responsability for what they produce. It's too easy and especially irresponsible to delegate blindly everything to AI.

surgical_fire 1y ago

> AI has the most effect for people with less experience or low performance. It has less of an effect for people on the high end. I actually think that it benefits high performance workers as AI can do a lot of heavy lifting that frees them to focus on things where their skills make a difference. Also, for less skilled or less experienced developers, they will have a harder time spotting the mistakes and inconsistencies generated by AI. This can actually become a productivity sink.

windows2020 1y ago
Reading other people's code is often more challenging than writing it yourself.
- mewpmewp2 1y ago
  With AI code I find it very easy to understand as I prompted it, I know what to expect, I know what it will likely do. Far easier than other people code.
jessoteric 1y ago
the main issue is that you end up looking down the barrel of begging claude, for the fifth time this session, to do it right- or just do it yourself in half the total time you've wasted so far. at least, this is what i typically end up with.
- surgical_fire 1y ago
  Typically, I've been asking it to do "heavy lifting" for me. It generally generates defective code, but it doesn't really matter all that much, it is still useful that it is mostly right, and I only need to make a few adjustments. It saves me a lot of typing. Would I pay for it? Probably not. But it is included in my IntelliJ subscription, so why not? It is there already.

jaredklewis 1y ago

> I have heard Dr Lilach Mollick, dir of Pedagogy at Wharton, say this has been shown numerous times. People who follow her husband, Ethan, are probably aware already. I'd be curious to see the sources. Basically every study I have ever read making some claim about programming (efficacy of IDEs, TDD, static typing, pair programming, formal CS education, ai assistants, etc...) has been a house of cards that falls apart with even modest prodding. They are usually premised on one or more inherently flawed metrics like number of github issues, LoC or whatever. That would be somewhat forgivable since there are not really any good metrics to go on, but then the studies invariably make only a perfunctory effort to disentangle even the most obvious of confounding variables, making all the results kind of pointless. Would be happy if anyone here knew of good papers that would change my mind on this point.

nyarlathotep_ 1y ago
> Basically every study I have ever read making some claim about programming (efficacy of IDEs, TDD, static typing, pair programming, formal CS education, ai assistants, etc...) has been a house of cards that falls apart with even modest prodding. Isn't this true about most things in software? I mean is there anything quantifiable about "microservices" vs "monolith"? Test-driven development, containers, whatever? I mean all of these things are in some way good, in some contexts, but it seems impossible to quantify benefits of any of them. I'm a believer that most decisions made in software are somewhat arbitrary, driven by trends and popularity and it seems like little effort is expended to come to overarching, data-backed results. When they are, they're rare and, like you said, fall apart under investigation or scrutiny. I've always found this strange about software in general. Even every time there's a "we rewrote $THING in $NEW_LANG" and it improved memory use/speed/latency whatever, there's a chorus of (totally legitimate) criticism and inquiry about how things were measured, what attempts were made to optimize the original solutions, if changes were made along the way outside of the language choice that impacted performance etc etc.
- jaredklewis 1y ago
  I think we agree. To be clear I am not arguing that tools and practices like TDD, microservices, ai assistants, and so on have no effect. They almost certainly have an effect (good or bad). It’s just the unfortunate reality that quantitatively measuring these effects in a meaningful way seems to basically be impossible (or at least I’ve never see it done). With enough resources I can’t think of any reason it shouldn’t be possible, but apparently those resources are not available because there are no good studies on these topics. Thus my skepticism of the “studies” referenced earlier in the thread.