3 ms·

I am an experienced programmer, and just recently started using ChatGPT and Cursor to help me code. Some things it does like magic, and it's hard to say what n-

I am an experienced programmer, and just recently started using ChatGPT and Cursor to help me code. Some things it does like magic, and it's hard to say what n-fold improvement there is. I'll put the lower limit on 3x and an upper, on certain tasks, at 20x. The project I am currently working on is took me about 16 hours to get to an MVP. A hobby project of similar size I did a few years ago took me about 80 hours. A lot of the work is NOT coding work that an LLM can help me with. 10x over everything is overstating it. However, if I can take my star developer who is already delivering significantly more value per dollar than my average guys and 3x him, that's quite a boost.

fwsgonzo 2y ago
I think I agree. It's maybe 2x for me after the project has been solidly established. However, it's very useful to have the AI during the beginning, when you're writing a lot of boilerplate or integrating with APIs that you don't know super well. The AI can surprise you and connect some dots.
handzhiev 2y ago
And it's not just the speed. The mental energy I save from having AI do some work vs. me doing it is allowing me to feel better, have mental energy for different things, be more productive, or just have some rest.
janalsncm 2y ago
Internet of Bugs did a decent side-by-side of four of them. https://youtu.be/-ONQvxqLXqE https://youtu.be/-ONQvxqLXqE They can get you 95% of the way there, the question is whether fixing the last 5% will take more time than doing all of it yourself. I presume this depends on your familiarity with the domain.
microtonal 2y ago
The lower bound is certainly lower than 3x. I do very domain-specific work (in Rust/Python) and usually the output is useless and ignoring the garbage suggestions can have a higher mental load than not using it. It does work well for boilerplate though and since boilerplate is the most annoying code to write, I'll tolerate it. I agree that the upper bound is pretty high, with more straightforward MVPs, it can generate large swabs of code for you from a description. Also for little DSLs that I haven't memorized, generating a pyproject.toml from a bunch of constraints is so much nicer/quicker than reading the docs again every few months.
bunderbunder 2y ago
You've got to consider second order effects, though. Where I'm working the star developers don't have a great culture of being conscientious about writing code their non-star colleagues can read and modify without too much trouble. (In fairness, I've worked exactly one place where that wasn't the case, and that was only because avoiding this problem was a personal vendetta for the CTO.) We gave them Copilot and it only compounded the problem. They started churning out code that's damn near unreadable at a furious pace. And that pushed the productivity of the people who are responsible for supporting and maintaining their code down toward zero. If you're only looking at how quickly people clear tickets, it looks like Copilot was an amazing productivity boost. But that's not a full accounting of the situation. A perspective that values teamwork and sustainability over army-of-one heroism and instant gratification can easily lead to the opposite conclusion.
- mattchamb 2y ago
  Ugh I am dealing with an amazingly productive platform team who churn out so much stuff that the product teams just can’t keep on top of all the changes to tools and tech. That one team is super productive at the cost of everyone else grinding to a halt.
  - bunderbunder 2y ago
    Some of the coolest demos from my Lean Six Sigma training were all about demonstrating that oftentimes the easiest way to increase the end-to-end throughput of your value chain is to find the team member with the greatest personal productivity and force them to slow down. You don't necessarily even have to do anything more than that. Just impose the rate limit on them and they'll automatically and unconsciously stop doing all sorts of little things - mostly various flavors of corner cutting - that make life harder for everyone around them.
    - mattchamb 2y ago
      Haha that’s a great idea and perspective on tackling it.
- 2y ago
UncleOxidant 2y ago
I was doing a short term contract over the summer porting some Verilog code to a newer FPGA. The Verilog code wasn't well documented and had been written over a period of about 20 years. There were no testbenches. Since I was making some requested changes to the design I wanted to be able to simulate it which requires creating a testbench. Verilog isn't my "native" HDL, I've done a lot more VHDL so I figured I'd see if an LLM could generate some of the boilerplate code required for a testbench (hooking up wires to the component under test, setting up clocks, etc.) I was using DeepSeek Coder for this - to my surprise it generated a pretty decent first-cut testbench, it even collected repeated code into verilog tasks and it figured out that there were some registers in the design that could be written to and read from. The testbench compiled the first time which really surprised me. There were some issues - for example, you can't write to certain registers in the design until you've set a particular bit in a particular register and it hadn't done that, but I was able to debug this pretty quickly in simulation. I figure that it saved me most of a day's work getting the testbench setup and the design simulating. My expectations were definitely exceeded.
deleted 2y ago
[deleted]
torginus 2y ago
I love working with Cursor/GenAI code, but there's one claim that I find repeated all the time that I found simply to be false - that junior/bad programmers put out tons of garbage code using GenAI assistants, that seniors later have to clean up. In my experience (mostly with Claude) generated code tends to be clean, nice looking, but ranges from somewhat flawed to totally busted. It's not ugly, messy code that works, but clean, nice code that doesn't. An inexperienced programmer would probably have no hope of fixing it up.
- koliber 2y ago
  Yes! I was explaining it to my wife yesterday. The code is clean, but occasionally the changes are in a completely wrong file. Or sometimes it changes something that is completely irrelevant. It's as if I had a decent programmer who occasionally loses his mind and does something non-sensical. This is dangerous because it's easy to get lulled into a false sense of security. Most of the code looks good, but you have to be very vigilant. It's the same with when LLMs generate writing. It sounds good, is grammatically correct, but sometimes, it's plain false.
- thiht 2y ago
  > that junior/bad programmers put out tons of garbage code That's what code review and mentoring are for. Don't blame tools, as a senior it's your role to pull the rest the team up. And in terms of process, it's ok to reject a PR or to delay it, business deadlines are not EVERYTHING. As a senior you're also in charge of what gets merged and when. You're the one responsible for the health of the codebase and tech debt.
  - bunderbunder 2y ago
    I think that this actually gets to the crux of the problem. On a high functioning team working on a brownfield project, the bottleneck typically isn't churning out code, it's quality control activities such as code review. The larger masses of lower-quality code that people are able to produce with Copilot make that problem worse. If the seniors are doing their jobs well then AI-generated code actually slows things down in more mature codebases, because the AI-generated code tends to take longer to review, which means that the rate of code getting into production slows down even as the rate at which developers can write it in the first place increases. I would bet a lot of money that the people who claim huge productivity improvements with tools like Copilot are mostly self-reporting their own experience with solo and greenfield projects. That's certainly strongly implied by the bulk of positive anecdotes that include sufficient detail to be able to hazard a guess. It's also consistent with the studies we're seeing.

fwsgonzo 2y ago

I think I agree. It's maybe 2x for me after the project has been solidly established. However, it's very useful to have the AI during the beginning, when you're writing a lot of boilerplate or integrating with APIs that you don't know super well. The AI can surprise you and connect some dots.

handzhiev 2y ago

And it's not just the speed. The mental energy I save from having AI do some work vs. me doing it is allowing me to feel better, have mental energy for different things, be more productive, or just have some rest.

janalsncm 2y ago

Internet of Bugs did a decent side-by-side of four of them. https://youtu.be/-ONQvxqLXqE https://youtu.be/-ONQvxqLXqE They can get you 95% of the way there, the question is whether fixing the last 5% will take more time than doing all of it yourself. I presume this depends on your familiarity with the domain.

microtonal 2y ago

The lower bound is certainly lower than 3x. I do very domain-specific work (in Rust/Python) and usually the output is useless and ignoring the garbage suggestions can have a higher mental load than not using it. It does work well for boilerplate though and since boilerplate is the most annoying code to write, I'll tolerate it. I agree that the upper bound is pretty high, with more straightforward MVPs, it can generate large swabs of code for you from a description. Also for little DSLs that I haven't memorized, generating a pyproject.toml from a bunch of constraints is so much nicer/quicker than reading the docs again every few months.

bunderbunder 2y ago

You've got to consider second order effects, though. Where I'm working the star developers don't have a great culture of being conscientious about writing code their non-star colleagues can read and modify without too much trouble. (In fairness, I've worked exactly one place where that wasn't the case, and that was only because avoiding this problem was a personal vendetta for the CTO.) We gave them Copilot and it only compounded the problem. They started churning out code that's damn near unreadable at a furious pace. And that pushed the productivity of the people who are responsible for supporting and maintaining their code down toward zero. If you're only looking at how quickly people clear tickets, it looks like Copilot was an amazing productivity boost. But that's not a full accounting of the situation. A perspective that values teamwork and sustainability over army-of-one heroism and instant gratification can easily lead to the opposite conclusion.

mattchamb 2y ago
Ugh I am dealing with an amazingly productive platform team who churn out so much stuff that the product teams just can’t keep on top of all the changes to tools and tech. That one team is super productive at the cost of everyone else grinding to a halt.
- bunderbunder 2y ago
  Some of the coolest demos from my Lean Six Sigma training were all about demonstrating that oftentimes the easiest way to increase the end-to-end throughput of your value chain is to find the team member with the greatest personal productivity and force them to slow down. You don't necessarily even have to do anything more than that. Just impose the rate limit on them and they'll automatically and unconsciously stop doing all sorts of little things - mostly various flavors of corner cutting - that make life harder for everyone around them.
  - mattchamb 2y ago
    Haha that’s a great idea and perspective on tackling it.
2y ago

UncleOxidant 2y ago

I was doing a short term contract over the summer porting some Verilog code to a newer FPGA. The Verilog code wasn't well documented and had been written over a period of about 20 years. There were no testbenches. Since I was making some requested changes to the design I wanted to be able to simulate it which requires creating a testbench. Verilog isn't my "native" HDL, I've done a lot more VHDL so I figured I'd see if an LLM could generate some of the boilerplate code required for a testbench (hooking up wires to the component under test, setting up clocks, etc.) I was using DeepSeek Coder for this - to my surprise it generated a pretty decent first-cut testbench, it even collected repeated code into verilog tasks and it figured out that there were some registers in the design that could be written to and read from. The testbench compiled the first time which really surprised me. There were some issues - for example, you can't write to certain registers in the design until you've set a particular bit in a particular register and it hadn't done that, but I was able to debug this pretty quickly in simulation. I figure that it saved me most of a day's work getting the testbench setup and the design simulating. My expectations were definitely exceeded.

deleted 2y ago

[deleted]

torginus 2y ago

I love working with Cursor/GenAI code, but there's one claim that I find repeated all the time that I found simply to be false - that junior/bad programmers put out tons of garbage code using GenAI assistants, that seniors later have to clean up. In my experience (mostly with Claude) generated code tends to be clean, nice looking, but ranges from somewhat flawed to totally busted. It's not ugly, messy code that works, but clean, nice code that doesn't. An inexperienced programmer would probably have no hope of fixing it up.

koliber 2y ago
Yes! I was explaining it to my wife yesterday. The code is clean, but occasionally the changes are in a completely wrong file. Or sometimes it changes something that is completely irrelevant. It's as if I had a decent programmer who occasionally loses his mind and does something non-sensical. This is dangerous because it's easy to get lulled into a false sense of security. Most of the code looks good, but you have to be very vigilant. It's the same with when LLMs generate writing. It sounds good, is grammatically correct, but sometimes, it's plain false.
thiht 2y ago
> that junior/bad programmers put out tons of garbage code That's what code review and mentoring are for. Don't blame tools, as a senior it's your role to pull the rest the team up. And in terms of process, it's ok to reject a PR or to delay it, business deadlines are not EVERYTHING. As a senior you're also in charge of what gets merged and when. You're the one responsible for the health of the codebase and tech debt.
- bunderbunder 2y ago
  I think that this actually gets to the crux of the problem. On a high functioning team working on a brownfield project, the bottleneck typically isn't churning out code, it's quality control activities such as code review. The larger masses of lower-quality code that people are able to produce with Copilot make that problem worse. If the seniors are doing their jobs well then AI-generated code actually slows things down in more mature codebases, because the AI-generated code tends to take longer to review, which means that the rate of code getting into production slows down even as the rate at which developers can write it in the first place increases. I would bet a lot of money that the people who claim huge productivity improvements with tools like Copilot are mostly self-reporting their own experience with solo and greenfield projects. That's certainly strongly implied by the bulk of positive anecdotes that include sufficient detail to be able to hazard a guess. It's also consistent with the studies we're seeing.