3 ms·

One concern Ive heard is that Llama can be a great accelerator for senior engineers who know what they want. However for junior engineers it could hold back the

One concern Ive heard is that Llama can be a great accelerator for senior engineers who know what they want. However for junior engineers it could hold back their learning as they just use whatever it gives without really understanding what or why something was done that way. It seems plausible to me. At least using stack overflow people get explanations of why you do x to get y, it's a bit copy paste, sure, but the pace brought learning. Now if you type a comment and get code, a junior engineer may not even read what's generated as 'theu don't need to, it usually works' .... Any ideas how we can ensure junior engineers do effectively learn and understand what is going on, while still getting the LLM benefit?

ripped_britches 2y ago
I’ve been coding for a long time and just became very proficient in dart this year just by using LLM and asking it to explain anything I don’t know. I haven’t been on Stack Overflow in 2 years and it’s not because I’m taking shortcuts by letting the LLM write my code, it’s because it is the best teacher / paired programmer that will just sit with you with endless patience. Especially using @docs with cursor and @directory context.
- zkry 2y ago
  > I’ve been coding for a long time... I think having been coding for a long time, I don't think you fall into the same category. Dart having paradigms not too different from other standard languages, a lot of these skills are probably transferable. I've seen beginners on the other hand using LLMs who couldn't even write a proper for-loop without AI assistance. They lack the fundamental ability to "run code in their head." This type of person I feel would be utterly limited by the capabilities of the AI model and would fail in lockstep with it.
  - brookst 2y ago
    This is kind of the classic “kids these days” argument: that because we understand something from what we consider the foundational level up to what we consider the goal level, anyone who comes along later and starts at higher levels will be limited / less capable / etc. It is true, but also irrelevant. Just like most programmers today do not need to understand CPU architecture or even assembly language, programmers in the future will not need to understand for loops the way we do. They will get good at writing LLM-optimized specifications that produce the desired results. And that will be fine, even if we old-timers bemoan that they don’t really program the way we do. Yes, the abstractions required will be inefficient. And we will always be able to say that when those kids solve our kinds of problems, our methods are better. Just like assembly programmers can look at simple python programs and be astounded at the complexity “required” to solve simple problems.
    - zkry 2y ago
      I agree with this take actually. I do imagine how programming in the future could be comprised of mostly interactions with LLMs. Such interactions would probably constrained enough to get the success rate of LLMs sufficiently high, maybe involving specialized DSLs made for LLMs. I do think the future may be more varied. Just like today where I look at kernel/systems/DB engineering and it seems almost arcane to me, I feel like there will be another stratum created, working on things where LLMs don't suffice. A lot of this will also depend on how far LLMs get. I would think that there would have to be more ChatGPT-like breakthroughs before this new type of developer can come.
    - rafaelmn 2y ago
      I feel like you underestimate how much effort goes into making CPUs reliable and how low level/well specified the problem of building a VM/compiler is compared to getting a natural language specification to executable program. Solving that reliably is basically AGI - I doubt there will be many humans in the loop if we reach that point.
      - brookst 2y ago
        I get CPU’s; I worked at Intel and cut my teeth on x86 assembly. But the fact that some people need to understand CPU architecture does not mean all people need to. The vast, vast majority of programmers do not need to understand CPUs or compilers today. That’s fine. It is also fine that many new programmers won’t even think in the form of functions and return values and types the way we do. I’m not saying traditional hard science tech is useless. I am saying it is not mandatory for everyone.
        rafaelmn 2y ago
        Yeah but what I'm saying is the level of engineering power that goes into making such relatively simple abstraction is huge and we have decades of experience. I think if AI ever gets to the point where it's so reliable for natural language -> code - we're into the AGI era and I don't see the role of programmers at all - bridging that layer successfully requires some very careful analysis and context awareness. Unless you think we're headed off in a direction where LLLms are gluing idiot proof boxes that are super inefficient but get the job done. I can sort of see that happening but in my experience reasoning through/debugging natural language specs is harder than going through equivalent code - I don't think we're getting much value here and adding a huge layer of inefficiency.
- rafaelmn 2y ago
  I am in a similar situation where I needed to jump in on a python Django codebase for a side gig I'm helping a friend out on. Since I have no interest in using Django in the future I decided to Claude my way trough the project - I did use Django like 10 years ago, am fairly competent with python and used rails recently - so I thought how bad can this be. By the time I had something cobbled together with Claude I decided to get a friend who's competent with Django to review my code and boy did I feel like an amateur. Not only did the code not use good practices/patterns (eg. not even using viewsets from DRF), I couldn't even follow in the conversation because I knew nothing about these concepts. So I spent a day reading docs and looking at a standard example of a idiomatic setup and it made my project a lot better. I've had this experience at multiple points in the project - not reading the API docs and letting Claude walk me through integration left me stuck when Claude failed, design decisions. So I would say Claude is useful for simple execution when you know what you are expecting, relying on it to learn sounds like a short term gain for problems down the line. At a point where LLMs can reliably lookup sources and reason trough something to explain it there will be no coding left, but I feel we aren't close to that with current tech.
lionkor 2y ago
Probably by not letting them check in code they don't understand. This creates a lot of overhead, but it turns out you simply can't skip that time investment. Somewhere along the junior->senior developer (in terms of skill, not time) is a big time cost of just sitting down and learning. You can pretend to skip it with LLMs, but you're just delaying having to understand it. So the benefit of LLMs is negligible in the long term, for beginner/junior programmers, as they essentially collect knowledge debt. Don't let your juniors use LLMs, or if you do, make sure they can explain every little detail about the code they have written. You don't have to be annoying about it - ask socratically.
- tails4e 2y ago
  I Agree. However in many cases the cat is out of the bag in terms of juniors using LLMs, it's almost ubiquitous these days .
blueboo 2y ago
Despite your observation that SO has explanations, SO can he and has also been used as a zero-learning crutch. Similarly, LLMs can be used as educational tools. In the end, learning and self-improvement needs some non-trivial motivation from the individual. The answer to your question is to show them it’s valuable to learn. If they find they’re completing their tasks adequately from AI assistance, then give them harder tasks. Meanwhile, model how your own learning effort is paying off. Note how if they are left with AI-trivial tasks and the benefit of learning remains abstract, we shouldn’t expect anything to change.
- tails4e 2y ago
  I agree, using LLMs the right way and SO the wrong way is definitely possible, but it's easier to use LLMs the wrong way and just blindly without understanding. A diligent person would use it as a learning tool for sure.

ripped_britches 2y ago

I’ve been coding for a long time and just became very proficient in dart this year just by using LLM and asking it to explain anything I don’t know. I haven’t been on Stack Overflow in 2 years and it’s not because I’m taking shortcuts by letting the LLM write my code, it’s because it is the best teacher / paired programmer that will just sit with you with endless patience. Especially using @docs with cursor and @directory context.

zkry 2y ago
> I’ve been coding for a long time... I think having been coding for a long time, I don't think you fall into the same category. Dart having paradigms not too different from other standard languages, a lot of these skills are probably transferable. I've seen beginners on the other hand using LLMs who couldn't even write a proper for-loop without AI assistance. They lack the fundamental ability to "run code in their head." This type of person I feel would be utterly limited by the capabilities of the AI model and would fail in lockstep with it.
- brookst 2y ago
  This is kind of the classic “kids these days” argument: that because we understand something from what we consider the foundational level up to what we consider the goal level, anyone who comes along later and starts at higher levels will be limited / less capable / etc. It is true, but also irrelevant. Just like most programmers today do not need to understand CPU architecture or even assembly language, programmers in the future will not need to understand for loops the way we do. They will get good at writing LLM-optimized specifications that produce the desired results. And that will be fine, even if we old-timers bemoan that they don’t really program the way we do. Yes, the abstractions required will be inefficient. And we will always be able to say that when those kids solve our kinds of problems, our methods are better. Just like assembly programmers can look at simple python programs and be astounded at the complexity “required” to solve simple problems.
  - zkry 2y ago
    I agree with this take actually. I do imagine how programming in the future could be comprised of mostly interactions with LLMs. Such interactions would probably constrained enough to get the success rate of LLMs sufficiently high, maybe involving specialized DSLs made for LLMs. I do think the future may be more varied. Just like today where I look at kernel/systems/DB engineering and it seems almost arcane to me, I feel like there will be another stratum created, working on things where LLMs don't suffice. A lot of this will also depend on how far LLMs get. I would think that there would have to be more ChatGPT-like breakthroughs before this new type of developer can come.
  - rafaelmn 2y ago
    I feel like you underestimate how much effort goes into making CPUs reliable and how low level/well specified the problem of building a VM/compiler is compared to getting a natural language specification to executable program. Solving that reliably is basically AGI - I doubt there will be many humans in the loop if we reach that point.
    - brookst 2y ago
      I get CPU’s; I worked at Intel and cut my teeth on x86 assembly. But the fact that some people need to understand CPU architecture does not mean all people need to. The vast, vast majority of programmers do not need to understand CPUs or compilers today. That’s fine. It is also fine that many new programmers won’t even think in the form of functions and return values and types the way we do. I’m not saying traditional hard science tech is useless. I am saying it is not mandatory for everyone.
      - rafaelmn 2y ago
        Yeah but what I'm saying is the level of engineering power that goes into making such relatively simple abstraction is huge and we have decades of experience. I think if AI ever gets to the point where it's so reliable for natural language -> code - we're into the AGI era and I don't see the role of programmers at all - bridging that layer successfully requires some very careful analysis and context awareness. Unless you think we're headed off in a direction where LLLms are gluing idiot proof boxes that are super inefficient but get the job done. I can sort of see that happening but in my experience reasoning through/debugging natural language specs is harder than going through equivalent code - I don't think we're getting much value here and adding a huge layer of inefficiency.
rafaelmn 2y ago
I am in a similar situation where I needed to jump in on a python Django codebase for a side gig I'm helping a friend out on. Since I have no interest in using Django in the future I decided to Claude my way trough the project - I did use Django like 10 years ago, am fairly competent with python and used rails recently - so I thought how bad can this be. By the time I had something cobbled together with Claude I decided to get a friend who's competent with Django to review my code and boy did I feel like an amateur. Not only did the code not use good practices/patterns (eg. not even using viewsets from DRF), I couldn't even follow in the conversation because I knew nothing about these concepts. So I spent a day reading docs and looking at a standard example of a idiomatic setup and it made my project a lot better. I've had this experience at multiple points in the project - not reading the API docs and letting Claude walk me through integration left me stuck when Claude failed, design decisions. So I would say Claude is useful for simple execution when you know what you are expecting, relying on it to learn sounds like a short term gain for problems down the line. At a point where LLMs can reliably lookup sources and reason trough something to explain it there will be no coding left, but I feel we aren't close to that with current tech.

lionkor 2y ago

Probably by not letting them check in code they don't understand. This creates a lot of overhead, but it turns out you simply can't skip that time investment. Somewhere along the junior->senior developer (in terms of skill, not time) is a big time cost of just sitting down and learning. You can pretend to skip it with LLMs, but you're just delaying having to understand it. So the benefit of LLMs is negligible in the long term, for beginner/junior programmers, as they essentially collect knowledge debt. Don't let your juniors use LLMs, or if you do, make sure they can explain every little detail about the code they have written. You don't have to be annoying about it - ask socratically.

tails4e 2y ago
I Agree. However in many cases the cat is out of the bag in terms of juniors using LLMs, it's almost ubiquitous these days .

blueboo 2y ago

Despite your observation that SO has explanations, SO can he and has also been used as a zero-learning crutch. Similarly, LLMs can be used as educational tools. In the end, learning and self-improvement needs some non-trivial motivation from the individual. The answer to your question is to show them it’s valuable to learn. If they find they’re completing their tasks adequately from AI assistance, then give them harder tasks. Meanwhile, model how your own learning effort is paying off. Note how if they are left with AI-trivial tasks and the benefit of learning remains abstract, we shouldn’t expect anything to change.

tails4e 2y ago
I agree, using LLMs the right way and SO the wrong way is definitely possible, but it's easier to use LLMs the wrong way and just blindly without understanding. A diligent person would use it as a learning tool for sure.