3 ms·

Yeah, I would say it's pretty variable, and it depends on what you mean by the word write. I've recently joined a startup whose stack is Ruby on Rails + Postgr

Yeah, I would say it's pretty variable, and it depends on what you mean by the word write. I've recently joined a startup whose stack is Ruby on Rails + PostgreSQL. Whilst I've used PostgreSQL, and am extremely familiar with relational databases (especially SQL Server), I've never been a Rubyist - never written a line of Ruby until very recently in fact - and certainly don't know Rails, although the MVC architecture and the way projects are structured feels very comfortable. We have what I'll describe as a prototype that I am in the process of reworking into a production app by fixing bugs, and making some pretty substantial functional improvements. I would say, out of the gate, 90%+ of the code I'm merging is initially written by an LLM for which I'm writing prompts... because I don't know Ruby or Rails (although I'm picking them up fast), and rather than scratch my head and spend a lot of time going down a Google and Stackoverflow black hole, it's just easier to tell the LLM what I want. But, of course, I tell it what I want like the software engineer I am, so I keep it on a short leash where everything is quite tightly specified, including what I'm looking for in terms of structure and architectural concerns. Then the code is fettled by me to a greater or lesser extent. Then I push and PR, and let Copilot review the code. Any good suggestions it makes I usually allow it to either commit directly or raise a PR for. I will often ask it to write automated tests for me. Once it's PRed everything, I then both review and test its code and, if it's good, merge into my PR, before running through our pipeline and merging everything. Is this quicker? Hmm. It might not be quicker than an experienced Rails developer would make progress, but it's certainly a lot quicker than I - a very inexperienced Rails developer - would make progress unaided, and that's quite an important value-add in itself. But yeah, if you look at it from a certain perspective, an LLM writes 90% of my code, but the reality is rather more nuanced, and so it's probably more like 50 - 70% that remains that way after I've got my grubby mitts on it.

AstroBen 8mo ago
This seems like a really short-sighted view. 6 months from now you'll be much more inexperienced than if you just went through the initial struggle (with an LLM's help!)
- WD-42 8mo ago
  What does the struggle with an LLMs help look like?
  - AstroBen 8mo ago
    "Explain this to me" until you're able to complete the task, instead of "do this for me"
    - WD-42 8mo ago
      If you break down the problem to small enough chunks, these basically become the same thing. The hardest part of new languages is the syntax and new APIs, so you end up getting the code anyway.
- bartread 8mo ago
  I'm not sure what you mean by this. As I mentioned in another comment, both ChatGPT and Claude to a pretty good job of explaining around the solution we're working on - because of the way I'm prompting them, no doubt - and it's not as if I haven't also worked through both Ruby and Rails tutorials. What I'd observe is that it's simply a different way of learning, but to me it seems like it's working at least decently well, and it means that I'm able to devote more time to thinking about and working on our roadmap and higher level issues without completely screwing my work/life balance - and, critically, time with family because the kids are growing up fast.
captain_coffee 8mo ago
So let me get this straight - you vibe code, make what you consider as necessary changes to the LLM-generated code, create PRs that get to be reviewed by another AI tool (Copilot), potentially make changes based on Copilot's suggestions and at the end, when you are satisfied with that particular PR you merge it yourself without having any other human reviewing it and then continue to the next PR. Did I get that right or did I miss anything?
- bartread 8mo ago
  Not quite: remember I said I fettled the code myself? This might involve rewriting, refactoring, reorganising, and I'll make changes to Copilot's PRs as well, but still the percentage of code written by LLM remains what I'd consider to be very high. I've been an engineer for more than 25 years so I know how to architect an application from the highest level down to the fine details, and I can obviously code in a variety of languages. Even with Ruby and Rails I know it well enough to read and understand the code already. Plus I obviously know HTML and JavaScript, and have used various CSS frameworks and overlays. (Though the truth is I basically hate fussing with CSS [and its various proxies], and I'm very far indeed from the king of UX or design anyway, so having an LLM to assist with that is a godsend.) The strict definition of vibe coding, as far as I've understood it, is that you don't touch or even really look at the code at all, and make all modifications, do debugging, etc., via prompting the LLM or using an agent. Vibe coding tends to break down in critical areas like security, or when the contours and wrinkles of the domain become too complex, so I'd never trust that approach for anything substantial or where security is a critical concern. For most organisations the data that I'm working with in our app is going to be second only to financial information in terms of sensitivity, and it's a complex domain, so the problems we're solving are simply not amenable to a pure vibe-coding approach. Going back to the original point on how much code LLMs are writing: with other languages that I'm more familiar with, where it might be quicker for me to write the code to do what I need myself rather than write a spec for the LLM, and modify afterwards, you would probably see quite a different picture.
  - captain_coffee 8mo ago
    Yes OK, but that was not the point that I was trying to make. You are developing in a language + framework that you are not familiar with, without having any human feedback in the process at all - see where I am going with this? Even with a quarter of a decade of experience you are still in what can basically be reffered to as uncharted territory.
WD-42 8mo ago
This is exactly how I use AI as well in codebases and languages I’m not familiar with. I’m a bit concerned we might be losing something without the google and stack overflow rabbit holes, and that’s the context surrounding the answer. Without digging through docs you don’t see what else is there. Without the comments on the SO answer you might miss some caveats. So while I’m faster than I would have been, I can’t help but wonder if I’m actually stunting my learning curve and might end up slower in the long term.
- bartread 8mo ago
  Yeah, I had that concern as well, but I tend to find that both ChatGPT and Claude do a pretty good job of explaining the issues around the solutions I'm using them for, so I think - maybe not all - but a good portion of that learning is still happening. No doubt this is a result of the way I prompt them, because it does involve a lot of discussion, iteration, and back and forth. Of course, you do have to know what you're doing well enough to spot a hallucination, and those do occur but, I mean, how often do you go down a blind alley or rethink your approach with coding, particularly with a new platform or framework, anyway? It doesn't feel like I'm wasting a huge amount of time here.

AstroBen 8mo ago

This seems like a really short-sighted view. 6 months from now you'll be much more inexperienced than if you just went through the initial struggle (with an LLM's help!)

WD-42 8mo ago
What does the struggle with an LLMs help look like?
- AstroBen 8mo ago
  "Explain this to me" until you're able to complete the task, instead of "do this for me"
  - WD-42 8mo ago
    If you break down the problem to small enough chunks, these basically become the same thing. The hardest part of new languages is the syntax and new APIs, so you end up getting the code anyway.
bartread 8mo ago
I'm not sure what you mean by this. As I mentioned in another comment, both ChatGPT and Claude to a pretty good job of explaining around the solution we're working on - because of the way I'm prompting them, no doubt - and it's not as if I haven't also worked through both Ruby and Rails tutorials. What I'd observe is that it's simply a different way of learning, but to me it seems like it's working at least decently well, and it means that I'm able to devote more time to thinking about and working on our roadmap and higher level issues without completely screwing my work/life balance - and, critically, time with family because the kids are growing up fast.

captain_coffee 8mo ago

So let me get this straight - you vibe code, make what you consider as necessary changes to the LLM-generated code, create PRs that get to be reviewed by another AI tool (Copilot), potentially make changes based on Copilot's suggestions and at the end, when you are satisfied with that particular PR you merge it yourself without having any other human reviewing it and then continue to the next PR. Did I get that right or did I miss anything?

bartread 8mo ago
Not quite: remember I said I fettled the code myself? This might involve rewriting, refactoring, reorganising, and I'll make changes to Copilot's PRs as well, but still the percentage of code written by LLM remains what I'd consider to be very high. I've been an engineer for more than 25 years so I know how to architect an application from the highest level down to the fine details, and I can obviously code in a variety of languages. Even with Ruby and Rails I know it well enough to read and understand the code already. Plus I obviously know HTML and JavaScript, and have used various CSS frameworks and overlays. (Though the truth is I basically hate fussing with CSS [and its various proxies], and I'm very far indeed from the king of UX or design anyway, so having an LLM to assist with that is a godsend.) The strict definition of vibe coding, as far as I've understood it, is that you don't touch or even really look at the code at all, and make all modifications, do debugging, etc., via prompting the LLM or using an agent. Vibe coding tends to break down in critical areas like security, or when the contours and wrinkles of the domain become too complex, so I'd never trust that approach for anything substantial or where security is a critical concern. For most organisations the data that I'm working with in our app is going to be second only to financial information in terms of sensitivity, and it's a complex domain, so the problems we're solving are simply not amenable to a pure vibe-coding approach. Going back to the original point on how much code LLMs are writing: with other languages that I'm more familiar with, where it might be quicker for me to write the code to do what I need myself rather than write a spec for the LLM, and modify afterwards, you would probably see quite a different picture.
- captain_coffee 8mo ago
  Yes OK, but that was not the point that I was trying to make. You are developing in a language + framework that you are not familiar with, without having any human feedback in the process at all - see where I am going with this? Even with a quarter of a decade of experience you are still in what can basically be reffered to as uncharted territory.

WD-42 8mo ago

This is exactly how I use AI as well in codebases and languages I’m not familiar with. I’m a bit concerned we might be losing something without the google and stack overflow rabbit holes, and that’s the context surrounding the answer. Without digging through docs you don’t see what else is there. Without the comments on the SO answer you might miss some caveats. So while I’m faster than I would have been, I can’t help but wonder if I’m actually stunting my learning curve and might end up slower in the long term.

bartread 8mo ago
Yeah, I had that concern as well, but I tend to find that both ChatGPT and Claude do a pretty good job of explaining the issues around the solutions I'm using them for, so I think - maybe not all - but a good portion of that learning is still happening. No doubt this is a result of the way I prompt them, because it does involve a lot of discussion, iteration, and back and forth. Of course, you do have to know what you're doing well enough to spot a hallucination, and those do occur but, I mean, how often do you go down a blind alley or rethink your approach with coding, particularly with a new platform or framework, anyway? It doesn't feel like I'm wasting a huge amount of time here.