5 ms·

> Or just don't use AI to write code. Anecdata, but I'm still finding CC to be absolutely outstanding at writing code. It's regularly writing systems-level co

> Or just don't use AI to write code. Anecdata, but I'm still finding CC to be absolutely outstanding at writing code. It's regularly writing systems-level code that would take me months to write by hand in hours, with minimal babysitting, basically no "specs" - just giving it coherent sane direction: like to make sure it tests things in several different ways, for several different cases, including performance, comparing directly to similar implementations (and constantly triple-checking that it actually did what you asked after it said "done"). For $200/mo, I can still run 2-3 clients almost 24/7 pumping out features. I rarely clear my session. I haven't noticed quality declines. Though, I will say, one random day - I'm not sure if it was dumb luck - or if I was in a test group, CC was literally doing 10x the amount of work / speed that it typically does. I guess strange things are bound to happen if you use it enough? Related anecdata: IME, there has been a MASSIVE decline in the quality of claude.ai (the chatbot interface). It is so different recently. It feels like a wanna-be crapier version of ChatGPT, instead of what it used to be, which was something that tried to be factual and useful rather than conversational and addictive and sycophantic.

ghurtado 6mo ago
> basically no "specs" - just giving it coherent sane direction This is one variable I almost always see in this discussion: the more strict the rules that you give the LLM, the more likely it is to deeply disappoint you The earlier in the process you use it (ie: scaffolding) the more mileage you will get out of it It's about accepting fallability and working with it, rather than trying to polish it away with care
- phatskat 6mo ago
  To me this still feels like it would be a net negative. I can scaffold most any project with a language/stack specific CLI command or even just checking out a repo. And sure, AI could “scaffold” further into controllers and views and maybe even some models, and they probably work ok. It’s then when they don’t, or when I need something tweaked, that the worry becomes “do I really understand what’s going on under the hood? Is the time to understand that worth it? Am I going to run across a small thread that I end up pulling until my 80% done sweater is 95% loose yarn?” To me the trade-off hasn’t proven worth it yet. Maybe for a personal pet project, and even then I don’t like the idea of letting something else undeterministically touch my system. “But use a VM!” they say, but that’s more overhead than I care for. Just researching the safest way to bootstrap this feels like more effort than value to me. Lastly, I think that a big part of why I like programming is that I like the act of writing code, understanding how it works, and building something I _know_.
  - michaelmrose 6mo ago
    A lot of the benefit of scaffolding is building basic context which you can also build by feeding it the files produced by whatever CLI tool and talk about it forcing it to think for lack of a better word about your design. You can also force feed it design and api documentation. If you think that you have given it too much you are almost certainly wrong. Doing nonsensical things with a library feed it the documentation still busted make it read the source
mlinsey 6mo ago
My anecdata is that it heavily depends on how much of the relevant code and instructions it can fit in the context window. A small app, or a task that touches one clear smaller subsection of a larger codebase, or a refactor that applies the same pattern independently to many different spots in a large codebase - the coding agents do extremely well, better than the median engineer I think. Basically "do something really hard on this one section of code, whose contract of how it intereacts with other code is clear, documented, and respected" is an ideal case for these tools. As soon as the codebase is large and there are gotchas, edge cases where one area of the code affects the other, or old requirements - things get treacherous. It will forget something was implemented somewhere else and write a duplicate version, it will hallucinate what the API shapes are, it will assume how a data field is used downstream based on its name and write something incorrect. IMO you can still work around this and move net-faster, especially with good test coverage, but you certainly have to pay attention. Larger codebases also work better when you started them with CC from the beginning, because it's older code is more likely to actually work how it exepects/hallucinates.
- onlyrealcuzzo 6mo ago
  > My anecdata is that it heavily depends on how much of the relevant code and instructions it can fit in the context window. Agreed, but I'm working on something >100k lines of code total (a new language and a runtime). It helps when you can implement new things as if they're green-field-ish AND THEN implement and plumb them later.
  - antonvs 6mo ago
    In a well-designed system, you can point an agent at a module of that system and it's perfectly capable of dealing with it. Humans also have a limited context window, and divide and conquer is always how we've dealt with it. The same approach works for agents.
    - onlyrealcuzzo 6mo ago
      > In a well-designed system, you can point an agent at a module of that system and it's perfectly capable of dealing with it. Yes, but the problem is that LLMs don't default to well-designed systems... So, you need to aggressively stay on top of them.
prmph 6mo ago
But, how do you know the code is good? If you do spot checks, that is woefully inadequate. I have lost count of the number of times when, poring over code a SOTA LLM has produced, I notice a lot of subtle but major issues (and many glaring ones as well), issues a cursory look is unlikely to pick up on. And if you are spending more time going over the code, how is that a massive speed improvement like you make it seem? And, what do you even mean by 10x the amount of work? I keep saying anybody that starts to spout these sort of anecdotes absolutely does NOT understand real world production level serious software engineering. Is the model doing 10x the amount of simplification, refactoring, and code pruning an effective senior level software engineer and architect would do? Is it doing 10x the detailed and agonizing architectural (re)work that a strong developer with honed architectural instincts would do? And if you tell me it's all about accepting the LLM being in the driver's seat and embracing vibe coding, it absolutely does NOT work for anything exceeding a moderate level of complexity. I used to try that several times. Up to now no model is able to write a simple markdown viewer with certain specific features I have wanted for a long time. I really doubt the stories people tell about creating whole compilers with vide coding. If all you see is and appreciate that it is pumping out 10x features, 10x more code, you are missing the whole point. In my experience you are actually producing a ton of sh*t, sorry.
- hirvi74 6mo ago
  > But, how do you know the code is good? Honestly, this more of a question about scope of the application and the potential threat vectors. If the GP is creating software that will never leave their machine(s) and is for personal usage only, I'd argue the code quality likely doesn't matter. If it's some enterprise production software that hundreds to millions of users depend on, software that manages sensitive data, etc., then I would argue code quality should asymptotically approach perfection. However, I have many moons of programming under my belt. I would honestly say that I am not sure what good code even is. Good to who? Good for what? Good how? I truly believe that most competent developers (however one defines competent) would be utterly appalled at the quality of the human-written code on some of the services they frequently use. I apply the Herbie Hancock philosophy when defining good code. When once asked what is Jazz music, Herbie responded with, "I can't describe it in words, but I know it when I hear it."
janalsncm 6mo ago
How can a person reconcile this comment with the one at the root of this thread? One person says Claude struggles to even meet the strict requirements of a spec sheet, another says Claude is doing a great job and doesn’t even need specific specs? I have my own anecdata but my comment is more about the dissonance here.
- sarchertech 6mo ago
  One person is rigorously checking to see if Claude is actually following the spec and one person isn’t?
  - flyinglizard 6mo ago
    ... or one person has a very strong mental model of what he expects to do, but the LLM has other ideas. FWIW I'm very happy with CC and Opus, but I don't treat it as a subordinate but as a peer; I leave it enough room to express what it thinks is best and guide later as needed. This may not work for all cases.
    - sarchertech 6mo ago
      If you don’t have a very strong mental model for what you are working on Claude can very easily guide in you into building the wrong thing. For example I’m working on a huge data migration right now. The data has to be migrated correctly. If there are any issues I want to fail fast and loud. Claude hates that philosophy. No matter how many different ways I add my reasons and instructions to stop it to the context, it will constantly push me towards removing crashes and replacing them with “graceful error handling”. If I didn’t have a strong idea about what I wanted, I would have let it talk me into building the wrong thing. Claude has no taste and its opinions are mostly those of the most prolific bloggers. Treating Claude like a peer is a terrible idea unless you are very inexperienced. And even then I don’t know if that’s a good idea.
      - oops 6mo ago
        That’s interesting to hear as for me Claude has been quite good about writing code that fails fast and loud and has specifically called it out more than once. It has also called out code that does not fail early in reviews.
hirvi74 6mo ago
Not sure about ChatGPT, but Claude was (is still?) an absolute ripper at cracking some software if one has even a little bit of experience/low level knowledge. At least, that's what my friend told me... I would personally never ever violate any software ToA.
kobe_bryant 6mo ago
months you say? how incredible. it beggars belief in fact
sameerds 6mo ago
> I can still run 2-3 clients almost 24/7 pumping out features. Honest question. How does one do that? My workflow is to create one git worktree per feature and start one session per worktree. And then I spent two hours in a worktree talking to Opus and reviewing what it is doing.
Planktonne 6mo ago
> It's regularly writing systems-level code that would take me months to write by hand in hours, with minimal babysitting Has your output kept pace with the code? Because months in hours means, even pushing those ratios quite far, to be years in days. Has your roadmap accelerated multiple years in the last few months in terms of verifiable results?
pighive 6mo ago
Curious to know what you are using $200/mo CC for? New applications? Business? What kind of application needs to run 2-3 clients run 24/7? How are you coming up with features. Just trying to understand the workflow.

ghurtado 6mo ago

> basically no "specs" - just giving it coherent sane direction This is one variable I almost always see in this discussion: the more strict the rules that you give the LLM, the more likely it is to deeply disappoint you The earlier in the process you use it (ie: scaffolding) the more mileage you will get out of it It's about accepting fallability and working with it, rather than trying to polish it away with care

phatskat 6mo ago
To me this still feels like it would be a net negative. I can scaffold most any project with a language/stack specific CLI command or even just checking out a repo. And sure, AI could “scaffold” further into controllers and views and maybe even some models, and they probably work ok. It’s then when they don’t, or when I need something tweaked, that the worry becomes “do I really understand what’s going on under the hood? Is the time to understand that worth it? Am I going to run across a small thread that I end up pulling until my 80% done sweater is 95% loose yarn?” To me the trade-off hasn’t proven worth it yet. Maybe for a personal pet project, and even then I don’t like the idea of letting something else undeterministically touch my system. “But use a VM!” they say, but that’s more overhead than I care for. Just researching the safest way to bootstrap this feels like more effort than value to me. Lastly, I think that a big part of why I like programming is that I like the act of writing code, understanding how it works, and building something I _know_.
- michaelmrose 6mo ago
  A lot of the benefit of scaffolding is building basic context which you can also build by feeding it the files produced by whatever CLI tool and talk about it forcing it to think for lack of a better word about your design. You can also force feed it design and api documentation. If you think that you have given it too much you are almost certainly wrong. Doing nonsensical things with a library feed it the documentation still busted make it read the source

mlinsey 6mo ago

My anecdata is that it heavily depends on how much of the relevant code and instructions it can fit in the context window. A small app, or a task that touches one clear smaller subsection of a larger codebase, or a refactor that applies the same pattern independently to many different spots in a large codebase - the coding agents do extremely well, better than the median engineer I think. Basically "do something really hard on this one section of code, whose contract of how it intereacts with other code is clear, documented, and respected" is an ideal case for these tools. As soon as the codebase is large and there are gotchas, edge cases where one area of the code affects the other, or old requirements - things get treacherous. It will forget something was implemented somewhere else and write a duplicate version, it will hallucinate what the API shapes are, it will assume how a data field is used downstream based on its name and write something incorrect. IMO you can still work around this and move net-faster, especially with good test coverage, but you certainly have to pay attention. Larger codebases also work better when you started them with CC from the beginning, because it's older code is more likely to actually work how it exepects/hallucinates.

onlyrealcuzzo 6mo ago
> My anecdata is that it heavily depends on how much of the relevant code and instructions it can fit in the context window. Agreed, but I'm working on something >100k lines of code total (a new language and a runtime). It helps when you can implement new things as if they're green-field-ish AND THEN implement and plumb them later.
- antonvs 6mo ago
  In a well-designed system, you can point an agent at a module of that system and it's perfectly capable of dealing with it. Humans also have a limited context window, and divide and conquer is always how we've dealt with it. The same approach works for agents.
  - onlyrealcuzzo 6mo ago
    > In a well-designed system, you can point an agent at a module of that system and it's perfectly capable of dealing with it. Yes, but the problem is that LLMs don't default to well-designed systems... So, you need to aggressively stay on top of them.

prmph 6mo ago

But, how do you know the code is good? If you do spot checks, that is woefully inadequate. I have lost count of the number of times when, poring over code a SOTA LLM has produced, I notice a lot of subtle but major issues (and many glaring ones as well), issues a cursory look is unlikely to pick up on. And if you are spending more time going over the code, how is that a massive speed improvement like you make it seem? And, what do you even mean by 10x the amount of work? I keep saying anybody that starts to spout these sort of anecdotes absolutely does NOT understand real world production level serious software engineering. Is the model doing 10x the amount of simplification, refactoring, and code pruning an effective senior level software engineer and architect would do? Is it doing 10x the detailed and agonizing architectural (re)work that a strong developer with honed architectural instincts would do? And if you tell me it's all about accepting the LLM being in the driver's seat and embracing vibe coding, it absolutely does NOT work for anything exceeding a moderate level of complexity. I used to try that several times. Up to now no model is able to write a simple markdown viewer with certain specific features I have wanted for a long time. I really doubt the stories people tell about creating whole compilers with vide coding. If all you see is and appreciate that it is pumping out 10x features, 10x more code, you are missing the whole point. In my experience you are actually producing a ton of sh*t, sorry.

hirvi74 6mo ago
> But, how do you know the code is good? Honestly, this more of a question about scope of the application and the potential threat vectors. If the GP is creating software that will never leave their machine(s) and is for personal usage only, I'd argue the code quality likely doesn't matter. If it's some enterprise production software that hundreds to millions of users depend on, software that manages sensitive data, etc., then I would argue code quality should asymptotically approach perfection. However, I have many moons of programming under my belt. I would honestly say that I am not sure what good code even is. Good to who? Good for what? Good how? I truly believe that most competent developers (however one defines competent) would be utterly appalled at the quality of the human-written code on some of the services they frequently use. I apply the Herbie Hancock philosophy when defining good code. When once asked what is Jazz music, Herbie responded with, "I can't describe it in words, but I know it when I hear it."

janalsncm 6mo ago

How can a person reconcile this comment with the one at the root of this thread? One person says Claude struggles to even meet the strict requirements of a spec sheet, another says Claude is doing a great job and doesn’t even need specific specs? I have my own anecdata but my comment is more about the dissonance here.

sarchertech 6mo ago
One person is rigorously checking to see if Claude is actually following the spec and one person isn’t?
- flyinglizard 6mo ago
  ... or one person has a very strong mental model of what he expects to do, but the LLM has other ideas. FWIW I'm very happy with CC and Opus, but I don't treat it as a subordinate but as a peer; I leave it enough room to express what it thinks is best and guide later as needed. This may not work for all cases.
  - sarchertech 6mo ago
    If you don’t have a very strong mental model for what you are working on Claude can very easily guide in you into building the wrong thing. For example I’m working on a huge data migration right now. The data has to be migrated correctly. If there are any issues I want to fail fast and loud. Claude hates that philosophy. No matter how many different ways I add my reasons and instructions to stop it to the context, it will constantly push me towards removing crashes and replacing them with “graceful error handling”. If I didn’t have a strong idea about what I wanted, I would have let it talk me into building the wrong thing. Claude has no taste and its opinions are mostly those of the most prolific bloggers. Treating Claude like a peer is a terrible idea unless you are very inexperienced. And even then I don’t know if that’s a good idea.
    - oops 6mo ago
      That’s interesting to hear as for me Claude has been quite good about writing code that fails fast and loud and has specifically called it out more than once. It has also called out code that does not fail early in reviews.

hirvi74 6mo ago

Not sure about ChatGPT, but Claude was (is still?) an absolute ripper at cracking some software if one has even a little bit of experience/low level knowledge. At least, that's what my friend told me... I would personally never ever violate any software ToA.

kobe_bryant 6mo ago

months you say? how incredible. it beggars belief in fact

sameerds 6mo ago

> I can still run 2-3 clients almost 24/7 pumping out features. Honest question. How does one do that? My workflow is to create one git worktree per feature and start one session per worktree. And then I spent two hours in a worktree talking to Opus and reviewing what it is doing.

Planktonne 6mo ago

> It's regularly writing systems-level code that would take me months to write by hand in hours, with minimal babysitting Has your output kept pace with the code? Because months in hours means, even pushing those ratios quite far, to be years in days. Has your roadmap accelerated multiple years in the last few months in terms of verifiable results?

pighive 6mo ago

Curious to know what you are using $200/mo CC for? New applications? Business? What kind of application needs to run 2-3 clients run 24/7? How are you coming up with features. Just trying to understand the workflow.