3 ms·

The title of this submission is misleading, that's not what they're saying. They said it doesn't show productivity gains for inexperienced developers still gain

• visarga 9mo ago
  The study measures if participants learn the library, but what they should study is if they learn effective coding agent patterns to use the library well. Learning the library is not going to be what we need in the future. > "We collect self-reported familiarity with AI coding tools, but we do not actually measure differences in prompting techniques." Many people drive cars without being able to explain how cars work. Or use devices like that. Or interact with people who's thinking they can't explain. Society works like that, it is functional, does not work by full understanding. We need to develop the functional part not the full understanding part. We can write C without knowing the machine code. You can often recognize a wrong note without being able to play the piece, spot a logical fallacy without being able to construct the valid argument yourself, catch a translation error with much less fluency than producing the translation would require. We need discriminative competence, not generative. For years I maintained a library for formatting dates and numbers (prices, ints, ids, phones), it was a pile of regex but I maintained hundreds of test cases for each type of parsing. And as new edge cases appeared, I added them to my tests, and iterated to keep the score high. I don't fully understand my own library, it emerged by scar accumulation. I mean, yes I can explain any line, but why these regexes in this order is a data dependent explanation I don't have anymore, all my edits run in loop with tests and my PRs are sent only when the score is good. Correctness was never grounded in understanding the implementation. Correctness was grounded in the test suite.
  • gjadi 9mo ago
    Interesting argument. But isn't the corrections of those errors that are valuable to society and get us a job? People can tell they found a bug or give a description about what they want from a software, yet it requires skills to fix the bugs and to build software. Though LLMs can speedup the process, expert human judgment is still required.
    • visarga 9mo ago
      I think the kind of judgement required here is to design ways to test the code without inspecting it manually line by line, that would be walking a motorcycle, and you would be only vibe-testing. That is why we have seen the FastRender browser and JustHTML parser - the testing part was solved upfront, so AI could go nuts implementing.
      • northfield27 9mo ago
        I partially agree, but I don’t think “design ways to test the code without inspecting it manually line by line” is a good strategy. Tests only cover cases you already know to look for. In my experience, many important edge cases are discovered by reading the implementation and noticing hidden assumptions or unintended interactions. When something goes wrong, understanding why almost always requires looking at the code, and that understanding is what informs better tests.
        • visarga 9mo ago
          Another possibility is to implement the same spec twice, and do differential testing, you can catch diverging assumptions and clarify them.
          • northfield27 9mo ago
            Isn't that too much work? Instead, just learning concepts with AI and then using HI (Human Intelligence) & AI to solve the problem at hand—by going through code line by line and writing tests - is a better approach productivity-, correctness-, efficiency-, and skill-wise. I can only think of LLMs as fast typists with some domain knowledge. Like typists of government/legal documents who know how to format documents but cannot practice law. Likewise, LLMs are code typists who can write good/decent/bad code but cannot practice software engineering - we need, and will need, a human for that.
    • another-dave 9mo ago
      I think there's different levels to look at it. If you know that you need O(n) "contains" checks and O(1) retrieval for items, for a given order of magnitude, it feels like you've all the pieces of the puzzle needed to make sure you keep the LLM on the straight and narrow, even if you didn't know off the top of your head that you should choose ArrayList. Or if you know that string manipulation might be memory intensive so you write automated tests around it for your order of magnitude, it probably doesn't really matter if you didn't know to choose StringBuilder. That feels different to e.g. not knowing the difference between an array list and linked list (or the concept of time/space complexity) in the first place.
  • discreteevent 9mo ago
    > Many people drive cars without being able to explain how cars work. But the fundamentals all cars behave the same way all the time. Imagine running a courier company where sometimes the vehicles take a random left turn. > Or interact with people who's thinking they can't explain Sure but they trust those service providers because they are reliable . And the reason that they are reliable is that the service providers can explain their own thinking to themselves. Otherwise their business would be chaos and nobody would trust them. How you approached your library was practical given the use case. But can you imagine writing a compiler like this? Or writing an industrial automation system? Not only would it be unreliable but it would be extremely slow. It's much faster to deal with something that has a consistent model that attempts to distill the essence of the problem, rather than patching on hack by hack in response to failed test after failed test.
  • 2sk21 9mo ago
    You can, most certainly, drive a car without understanding how it works. A pilot of an aircraft on the other hand needs a fairly detailed understanding of the subsystems in order to effectively fly it. I think being a programmer is closer to being an aircraft pilot than a car driver.
    • northfield27 9mo ago
      Agreed
    • iammjm 9mo ago
      Sure, if you are a pilot then that makes sense. But what if you are a company that uses planes to deliver goods? Like when the focus shifts from the thing itself to its output
• concats 9mo ago
  I agree. It's very missleading. Here's what the authors actually say: > AI assistance produces significant productivity gains across professional domains, particularly for novice workers. Yet how this assistance affects the development of skills required to effectively supervise AI remains unclear. Novice workers who rely heavily on AI to complete unfamiliar tasks may compromise their own skill acquisition in the process. We conduct randomized experiments to study how developers gained mastery of a new asynchronous programming library with and without the assistance of AI. We find that AI use impairs conceptual understanding, code reading, and debugging abilities, without delivering significant efficiency gains on average. Participants who fully delegated coding tasks showed some productivity improvements, but at the cost of learning the library. We identify six distinct AI interaction patterns, three of which involve cognitive engagement and preserve learning outcomes even when participants receive AI assistance. Our findings suggest that AI-enhanced productivity is not a shortcut to competence and AI assistance should be carefully adopted into workflows to preserve skill formation -- particularly in safety-critical domains.
  • danbruc 9mo ago
    That itself sounds contradictory to me. I assistance produces significant productivity gains across professional domains, particularly for novice workers. We find that AI use impairs conceptual understanding, code reading, and debugging abilities, without delivering significant efficiency gains on average. Are the two sentences talking about non-overlapping domains? Is there an important distinction between productivity and efficiency gains? Does one focus on novice users and one on experienced ones? Admittedly did not read the paper yet, might be clearer than the abstract.
    • torginus 9mo ago
      That doesn't really line up with my experience, I wanted to debug a CMake file recently, having done no such thing before - AI helped me walk through the potential issues, explaining what I got wrong. I learned a lot more in a short amount of time than I would've stumbling around on my own. Afaik its been known for a long time that the most effective way of learning a new skill, is to get private tutoring from an expert.
• omnicognate 9mo ago
  I agree the title should be changed, but as I commented on the dupe of this submission learning is not something that happens as a beginner, student or "junior" programmer and then stops. The job is learning, and after 25 years of doing it I learn more per day than ever.
  • mold_aid 9mo ago
    The study doesn't argue that you stopped learning.
    • omnicognate 9mo ago
      I didn't say it did. I just pointed out that learning effectively isn't only a concern for "inexperienced developers still gaining knowledge".
• emsign 9mo ago
  > They said it doesn't show productivity gains for inexperienced developers still gaining knowledge. But that's what "impairs learning" means.
  • deleted 9mo ago
    [deleted]