4 ms·

After using Claude 3.7 Sonnet for a few weeks, my verdict is that its coding abilities are unimpressive both for unsupervised coding but also for problem solvin

After using Claude 3.7 Sonnet for a few weeks, my verdict is that its coding abilities are unimpressive both for unsupervised coding but also for problem solving/debugging if you are expecting accurate results and correct code. However, as a debugging companion, it's slightly better than a rubber duck, because at least there's some suspension of disbelief so I tend to explain things to it earnestly and because of that, process them better by myself. That said, it's remarkable and interesting how quickly these models are getting better. Can't say anything about version 4, not having tested it yet, but in a five years time, the things are not looking good for junior developers for sure, and a few years more, for everybody.

StefanBatory 1y ago
Things were already not looking good for junior devs. I graduated this year in Poland, many of my peers were looking for jobs in IT for like a year before they were able to find anything. And many internships were faked as they couldn't get anything (here it's required for you to do internship if you want to graduate).
- GolDDranks 1y ago
  I sincerely hope you'll manage to find a job! What I meant was purely from the capabilities perspective. There's no way a current AI model would outperform an average junior dev in job performance over... let's say, a year to be charitable. Even if they'd outperform junior devs during the first week, no way for a longer period. However, that doesn't mean that the business people won't try to pre-empt potential savings. Some think that AI is already good enough, and others don't, but they count it to be good enough in the future. Whether that happens remains to be seen, but the effects are already here.
- v3np 1y ago
  If I may ask: what university was this? Asking as I am the CTO of a YC startup and we are hiring junior engineers in Berlin!
BeetleB 1y ago
I've noticed an interesting trend: Most people who are happy with LLM coding say something like "Wow, it's awesome. I asked it to do X and it did it so fast with minimal bugs, and good code", and occasionally show the output. Many provide even more details. Most people who are not happy with LLM coding ... provide almost no details. As someone who's impressed by LLM coding, when I read a post like yours, I tend to have a lot of questions, and generally the post doesn't have the answers. 1. What type of problem did you try it out with? 2. Which model did you use (you get points for providing that one!) 3. Did you consider a better model (compare how Gemini 2.5 Pro compares to Sonnet 3.7 on the Aider leaderboard)? 4. What were its failings? Buggy code? Correct code but poorly architected? Correct code but used some obscure method to solve it rather than a canonical one? 5. Was it working on an existing codebase or was this new code? 6. Did you manage well how many tokens were sent? Did you use a tool that informs you of the number of tokens for each query? 7. Which tool did you use? It's not just a question of the model, but of how the tool handles the prompts/agents under it. Aider is different from Code which is different from Cursor which is different form Windsurf. 8. What strategy did you follow? Did you give it the broad spec and ask it to do anything? Did you work bottom up and work incrementally? I'm not saying LLM coding is the best or can replace a human. But for certain use cases (e.g. simple script, written from scratch), it's absolutely fantastic. I (mostly) don't use it on production code, but little peripheral scripts I need to write (at home or work), it's great. And that's why people like me wonder what people like you are doing differently. But such people aren't forthcoming with the details.
- jiggawatts 1y ago
  Reminds me of the early days of endless “ChatGPT can’t do X” comments where they were invariably using 3.5 Turbo instead of 4, which was available to paying users only. Humans are much lazier than AIs was my takeaway lesson from that.
- sponnath 1y ago
  I feel like the opposite is true but maybe the issue is that we both live in separate bubbles. Often times I see people on X and elsewhere making wild claims about the capabilities of AI and rarely do they link to the actual output. That said, I agree that AI has been amazing for fairly closed ended problems like writing a basic script or even writing scaffolding for tests (it's about 90% effective at producing tests I'd consider good assuming you give it enough context). Greenfield projects have been more of a miss than a hit for me. It starts out well but if you don't do a good job of directing architecture it can go off the rails pretty quickly. In a lot of cases I find it faster to write the code myself.
jjmarr 1y ago
As a junior developer it's much easier for me to jump into a new codebase or language and make an impact. I just shipped a new error message in LLVM because Cline found the 5 spots in 10k+ files where I needed to make the code changes. When I started an internship last year, it took me weeks to learn my way around my team's relatively smaller codebase. I consider this a skill and cost issue. If you are rich and able to read fast, you can start writing LLVM/Chrome/etc features before graduating university. If you cannot afford the hundreds of dollars a month Claude costs or cannot effectively review the code as it is being generated, you will not be employable in the workforce.
- BeefySwain 1y ago
  I'm curious what tooling you are using to accomplish this?
  - jjmarr 1y ago
    I used Cline+Claude 3.7 Sonnet for the initial draft of this LLVM PR. There's a lot of handholding and the final version was much different than the original. https://github.com/llvm/llvm-project/pull/130458 https://github.com/llvm/llvm-project/pull/130458 Right now I'm using Roo Code and Claude 4.0. Roo Code looks cooler and draws diagrams but I don't know if it's better.
- kweingar 1y ago
  > If you cannot afford the hundreds of dollars a month Claude costs Employers will buy AI tools for their employees, this isn't a problem. If you're saying that you need to buy and learn these tools yourself in order to get a job, I strongly disagree. Prompting is not exactly rocket science, and with every generation of models it gets easier. Soon you'll be able to pick it up in a few hours. It's not a differentiator.
  - jjmarr 1y ago
    I need side projects and OSS contributions to get hired as a new graduate or an intern. The bar for both of those will be much higher if everyone is using AI.

StefanBatory 1y ago

Things were already not looking good for junior devs. I graduated this year in Poland, many of my peers were looking for jobs in IT for like a year before they were able to find anything. And many internships were faked as they couldn't get anything (here it's required for you to do internship if you want to graduate).

GolDDranks 1y ago
I sincerely hope you'll manage to find a job! What I meant was purely from the capabilities perspective. There's no way a current AI model would outperform an average junior dev in job performance over... let's say, a year to be charitable. Even if they'd outperform junior devs during the first week, no way for a longer period. However, that doesn't mean that the business people won't try to pre-empt potential savings. Some think that AI is already good enough, and others don't, but they count it to be good enough in the future. Whether that happens remains to be seen, but the effects are already here.
v3np 1y ago
If I may ask: what university was this? Asking as I am the CTO of a YC startup and we are hiring junior engineers in Berlin!

BeetleB 1y ago

I've noticed an interesting trend: Most people who are happy with LLM coding say something like "Wow, it's awesome. I asked it to do X and it did it so fast with minimal bugs, and good code", and occasionally show the output. Many provide even more details. Most people who are not happy with LLM coding ... provide almost no details. As someone who's impressed by LLM coding, when I read a post like yours, I tend to have a lot of questions, and generally the post doesn't have the answers. 1. What type of problem did you try it out with? 2. Which model did you use (you get points for providing that one!) 3. Did you consider a better model (compare how Gemini 2.5 Pro compares to Sonnet 3.7 on the Aider leaderboard)? 4. What were its failings? Buggy code? Correct code but poorly architected? Correct code but used some obscure method to solve it rather than a canonical one? 5. Was it working on an existing codebase or was this new code? 6. Did you manage well how many tokens were sent? Did you use a tool that informs you of the number of tokens for each query? 7. Which tool did you use? It's not just a question of the model, but of how the tool handles the prompts/agents under it. Aider is different from Code which is different from Cursor which is different form Windsurf. 8. What strategy did you follow? Did you give it the broad spec and ask it to do anything? Did you work bottom up and work incrementally? I'm not saying LLM coding is the best or can replace a human. But for certain use cases (e.g. simple script, written from scratch), it's absolutely fantastic. I (mostly) don't use it on production code, but little peripheral scripts I need to write (at home or work), it's great. And that's why people like me wonder what people like you are doing differently. But such people aren't forthcoming with the details.

jiggawatts 1y ago
Reminds me of the early days of endless “ChatGPT can’t do X” comments where they were invariably using 3.5 Turbo instead of 4, which was available to paying users only. Humans are much lazier than AIs was my takeaway lesson from that.
sponnath 1y ago
I feel like the opposite is true but maybe the issue is that we both live in separate bubbles. Often times I see people on X and elsewhere making wild claims about the capabilities of AI and rarely do they link to the actual output. That said, I agree that AI has been amazing for fairly closed ended problems like writing a basic script or even writing scaffolding for tests (it's about 90% effective at producing tests I'd consider good assuming you give it enough context). Greenfield projects have been more of a miss than a hit for me. It starts out well but if you don't do a good job of directing architecture it can go off the rails pretty quickly. In a lot of cases I find it faster to write the code myself.

jjmarr 1y ago

As a junior developer it's much easier for me to jump into a new codebase or language and make an impact. I just shipped a new error message in LLVM because Cline found the 5 spots in 10k+ files where I needed to make the code changes. When I started an internship last year, it took me weeks to learn my way around my team's relatively smaller codebase. I consider this a skill and cost issue. If you are rich and able to read fast, you can start writing LLVM/Chrome/etc features before graduating university. If you cannot afford the hundreds of dollars a month Claude costs or cannot effectively review the code as it is being generated, you will not be employable in the workforce.

BeefySwain 1y ago
I'm curious what tooling you are using to accomplish this?
- jjmarr 1y ago
  I used Cline+Claude 3.7 Sonnet for the initial draft of this LLVM PR. There's a lot of handholding and the final version was much different than the original. https://github.com/llvm/llvm-project/pull/130458 https://github.com/llvm/llvm-project/pull/130458 Right now I'm using Roo Code and Claude 4.0. Roo Code looks cooler and draws diagrams but I don't know if it's better.
kweingar 1y ago
> If you cannot afford the hundreds of dollars a month Claude costs Employers will buy AI tools for their employees, this isn't a problem. If you're saying that you need to buy and learn these tools yourself in order to get a job, I strongly disagree. Prompting is not exactly rocket science, and with every generation of models it gets easier. Soon you'll be able to pick it up in a few hours. It's not a differentiator.
- jjmarr 1y ago
  I need side projects and OSS contributions to get hired as a new graduate or an intern. The bar for both of those will be much higher if everyone is using AI.