6 ms·

I often feel these types of blogposts would be more helpful if they demonstrated someone using the tools to build something non-trivial. Is Claude really "lear

I often feel these types of blogposts would be more helpful if they demonstrated someone using the tools to build something non-trivial. Is Claude really "learning new skills" when you feed it a book, or does it present it like that because you're prompting encourages that sort of response-behavior. I feel like it has to demo Claude with the new skills and Claude without. Maybe I'm a curmudgeon but most of these types of blogs feel like marketing pieces with the important bit is that so much is left unsaid and not shown, that it comes off like a kid trying to hype up their own work without the benefit of nuance or depth.

khaledh 1y ago
Agreed. The methodology needed here is something like an A/B test, with quantifiable metrics that demonstrate the effectiveness of the tool. And to do it not just once, but many times under different scenarios so that it demonstrates statistical significance. The most challenging part when working with coding agents is that they seem to do well initially on a small code base with low complexity. Once the codebase gets bigger with lots of non-trivial connections and patterns, they almost always experience tunnel vision when asked to do anything non-trivial, leading to increased tech debt.
- mwigdahl 1y ago
  The problem is that you're talking about a multistep process where each step beyond the first depends on the particular path the agent starts down, along with human input that's going to vary at each step. I made a crude first stab at an approach that at least uses similar steps and structure to compare the effectiveness of AI agents. My approach was used on a small toy problem, but one that was complex enough the agents couldn't one-shot and required error correction. It was enough to show significant differences, but scaling this to larger projects and multiple runs would be pretty difficult. https://mattwigdahl.substack.com/p/claude-code-vs-codex-cli-head-to https://mattwigdahl.substack.com/p/claude-code-vs-codex-cli-...
  - potatolicious 1y ago
    What you're getting at is the heart of the problem with the LLM hype train though, isn't it? "We should have rigorous evaluations of whether or not [thing] works." seems like an incredibly obvious thought. But in the realm of LLM-enabled use cases they're also expensive. You'd need to recruit dozens, perhaps even hundreds of developers to do this, with extensive observation and rating of the results. So rather than actually try to measure the efficacy, we just get blog posts with cherry-picked example of "LLM does something cool". Everything is just anecdata. This is also the biggest barrier to actual LLM adoption for many, many applications. The gap between "it does something REALLY IMPRESSIVE 40% of the time and shits the bed otherwise" and "production system" is a yawning chasm.
    - marcosdumay 1y ago
      It's the heart of the problem with all software engineer research. That's why we have so little reliable knowledge. It applies to using LLMs too. I guess the one largest difference here is that LLM has few enough companies with abundant enough money pushing it to make it trivial for them to run a test like this. So the fact that they aren't doing that also says a lot.
    - oblio 1y ago
      > What you're getting at is the heart of the problem with the LLM hype train though, isn't it? > "We should have rigorous evaluations of whether or not [thing] works." seems like an incredibly obvious thought. Heh, I'd rephrase the first part to: > What you're getting at is the heart of the problem with software development though, isn't it?
    - simonw 1y ago
      The UK government ran a study with thousands of developers quite recently: https://www.gov.uk/government/publications/ai-coding-assistant-trial/ai-coding-assistant-trial-uk-public-sector-findings-report https://www.gov.uk/government/publications/ai-coding-assista...
      - b_e_n_t_o_n 1y ago
- claytongulick 1y ago
  > The methodology needed here is something like an A/B test, with quantifiable metrics that demonstrate the effectiveness of the tool. And to do it not just once, but many times under different scenarios so that it demonstrates statistical significance. If that's what we need to do, don't we already have the answer to the question?
coolKid721 1y ago
Yeah I was reading this seeing if there was something he'd actually show that would be useful, what pain point he is solving, but it's just slop.
simonw 1y ago
Here's one from today: https://mitchellh.com/writing/non-trivial-vibing https://mitchellh.com/writing/non-trivial-vibing
- j_bum 1y ago
  This was a fun read. I’ve similarly been using spec.md and running to-do.md files that capture detailed descriptions of the problems and their scoped history. I mark each of my to-do’s with informational tags: [BUG], [FEAT], etc. I point the LLM to the exact to-do (or section of to-do’s) with the spec.md in memory and let it work. This has been working very well for me.
  - lcnPylGDnU4H9OF 1y ago
    Do you mind linking to example spec/to-do files?
    - SteveJS 1y ago
      Here is a (3 month old) repo where i did something like that and all the tasks are checked into the linear git history — https://github.com/KnowSeams/KnowSeams https://github.com/KnowSeams/KnowSeams
    - j_bum 1y ago
      Sure thing. Here is an example set of the agent/spec/to-do files for a hobby project I'm actively working on. https://gist.github.com/JacobBumgarner/d29b660cb81a227885acc0a053663dca https://gist.github.com/JacobBumgarner/d29b660cb81a227885acc...
      - lcnPylGDnU4H9OF 1y ago
        Thanks!
        j_bum 1y ago
        No problem! I’d love to hear any approach you’ve taken as well.
spankibalt 1y ago
> "Maybe I'm a curmudgeon but most of these types of blogs feel like marketing pieces with the important bit is that so much is left unsaid and not shown, that it comes off like a kid trying to hype up their own work without the benefit of nuance or depth." C'mon, such self-congratulatory "Look at My Potency: How I'm using Nicknack.exe" fluffies always were and always will be a staple of the IT industry.
- lcnPylGDnU4H9OF 1y ago
  Still, the best such pieces are detailed and explanatory.
causal 1y ago
Using LLMs for coding complex projects at scale over a long time is really challenging! This is partly because defining requirements alone is much more challenging than most people want to believe. LLMs accelerate any move in the wrong direction.
- dexwiz 1y ago
  My analogy is LLMs are a gas pedal. Makes you go fast, but you still have to know when to turn.
  - sreekanth850 1y ago
    True
- sreekanth850 1y ago
  One should know theend to end design and architecture. Should stop llm when adding complex fancy things.
- SteveJS 1y ago
  Having the llm write the spec/workunit from a conversation works well. Exploring a problem space with a (good) coding agent is fantastic. However for complex projects IMO one must read what was written by the llm … every actual word. When it ‘got away’ from me, in each case I left something in the llm written markdown that I should have removed. 99% “I can ask for that later” and 1% “that’s a good idea i hadn’t considered” might be the right ratio when reading an llm generated plan/spec/workunit. Breaking work into single context passes … 50-60k tokens in sonnet 4.5 has had typically fantastic results for me. My side project is using lean 4 and a carelessly left in ‘validate’ rather than ‘verify’ lead down a hilariously complicated path equivalent to matching an output against a known string. I recovered, but it wasn’t obvious to me that was happening. I however would not be able to write lean proofs myself, so diagnosing the problem and fixing it is a small price to be able to mechanically verify part of my software is correct.
danielmarkbruce 1y ago
Why not just use claude code and come to your own conclusion?

khaledh 1y ago

Agreed. The methodology needed here is something like an A/B test, with quantifiable metrics that demonstrate the effectiveness of the tool. And to do it not just once, but many times under different scenarios so that it demonstrates statistical significance. The most challenging part when working with coding agents is that they seem to do well initially on a small code base with low complexity. Once the codebase gets bigger with lots of non-trivial connections and patterns, they almost always experience tunnel vision when asked to do anything non-trivial, leading to increased tech debt.

mwigdahl 1y ago
The problem is that you're talking about a multistep process where each step beyond the first depends on the particular path the agent starts down, along with human input that's going to vary at each step. I made a crude first stab at an approach that at least uses similar steps and structure to compare the effectiveness of AI agents. My approach was used on a small toy problem, but one that was complex enough the agents couldn't one-shot and required error correction. It was enough to show significant differences, but scaling this to larger projects and multiple runs would be pretty difficult. https://mattwigdahl.substack.com/p/claude-code-vs-codex-cli-head-to https://mattwigdahl.substack.com/p/claude-code-vs-codex-cli-...
- potatolicious 1y ago
  What you're getting at is the heart of the problem with the LLM hype train though, isn't it? "We should have rigorous evaluations of whether or not [thing] works." seems like an incredibly obvious thought. But in the realm of LLM-enabled use cases they're also expensive. You'd need to recruit dozens, perhaps even hundreds of developers to do this, with extensive observation and rating of the results. So rather than actually try to measure the efficacy, we just get blog posts with cherry-picked example of "LLM does something cool". Everything is just anecdata. This is also the biggest barrier to actual LLM adoption for many, many applications. The gap between "it does something REALLY IMPRESSIVE 40% of the time and shits the bed otherwise" and "production system" is a yawning chasm.
  - marcosdumay 1y ago
    It's the heart of the problem with all software engineer research. That's why we have so little reliable knowledge. It applies to using LLMs too. I guess the one largest difference here is that LLM has few enough companies with abundant enough money pushing it to make it trivial for them to run a test like this. So the fact that they aren't doing that also says a lot.
  - oblio 1y ago
    > What you're getting at is the heart of the problem with the LLM hype train though, isn't it? > "We should have rigorous evaluations of whether or not [thing] works." seems like an incredibly obvious thought. Heh, I'd rephrase the first part to: > What you're getting at is the heart of the problem with software development though, isn't it?
  - simonw 1y ago
    The UK government ran a study with thousands of developers quite recently: https://www.gov.uk/government/publications/ai-coding-assistant-trial/ai-coding-assistant-trial-uk-public-sector-findings-report https://www.gov.uk/government/publications/ai-coding-assista...
    - b_e_n_t_o_n 1y ago
claytongulick 1y ago
> The methodology needed here is something like an A/B test, with quantifiable metrics that demonstrate the effectiveness of the tool. And to do it not just once, but many times under different scenarios so that it demonstrates statistical significance. If that's what we need to do, don't we already have the answer to the question?

coolKid721 1y ago

Yeah I was reading this seeing if there was something he'd actually show that would be useful, what pain point he is solving, but it's just slop.

simonw 1y ago

Here's one from today: https://mitchellh.com/writing/non-trivial-vibing https://mitchellh.com/writing/non-trivial-vibing

j_bum 1y ago
This was a fun read. I’ve similarly been using spec.md and running to-do.md files that capture detailed descriptions of the problems and their scoped history. I mark each of my to-do’s with informational tags: [BUG], [FEAT], etc. I point the LLM to the exact to-do (or section of to-do’s) with the spec.md in memory and let it work. This has been working very well for me.
- lcnPylGDnU4H9OF 1y ago
  Do you mind linking to example spec/to-do files?
  - SteveJS 1y ago
    Here is a (3 month old) repo where i did something like that and all the tasks are checked into the linear git history — https://github.com/KnowSeams/KnowSeams https://github.com/KnowSeams/KnowSeams
  - j_bum 1y ago
    Sure thing. Here is an example set of the agent/spec/to-do files for a hobby project I'm actively working on. https://gist.github.com/JacobBumgarner/d29b660cb81a227885acc0a053663dca https://gist.github.com/JacobBumgarner/d29b660cb81a227885acc...
    - lcnPylGDnU4H9OF 1y ago
      Thanks!
      - j_bum 1y ago
        No problem! I’d love to hear any approach you’ve taken as well.

spankibalt 1y ago

> "Maybe I'm a curmudgeon but most of these types of blogs feel like marketing pieces with the important bit is that so much is left unsaid and not shown, that it comes off like a kid trying to hype up their own work without the benefit of nuance or depth." C'mon, such self-congratulatory "Look at My Potency: How I'm using Nicknack.exe" fluffies always were and always will be a staple of the IT industry.

lcnPylGDnU4H9OF 1y ago
Still, the best such pieces are detailed and explanatory.

causal 1y ago

Using LLMs for coding complex projects at scale over a long time is really challenging! This is partly because defining requirements alone is much more challenging than most people want to believe. LLMs accelerate any move in the wrong direction.

dexwiz 1y ago
My analogy is LLMs are a gas pedal. Makes you go fast, but you still have to know when to turn.
- sreekanth850 1y ago
  True
sreekanth850 1y ago
One should know theend to end design and architecture. Should stop llm when adding complex fancy things.
SteveJS 1y ago
Having the llm write the spec/workunit from a conversation works well. Exploring a problem space with a (good) coding agent is fantastic. However for complex projects IMO one must read what was written by the llm … every actual word. When it ‘got away’ from me, in each case I left something in the llm written markdown that I should have removed. 99% “I can ask for that later” and 1% “that’s a good idea i hadn’t considered” might be the right ratio when reading an llm generated plan/spec/workunit. Breaking work into single context passes … 50-60k tokens in sonnet 4.5 has had typically fantastic results for me. My side project is using lean 4 and a carelessly left in ‘validate’ rather than ‘verify’ lead down a hilariously complicated path equivalent to matching an output against a known string. I recovered, but it wasn’t obvious to me that was happening. I however would not be able to write lean proofs myself, so diagnosing the problem and fixing it is a small price to be able to mechanically verify part of my software is correct.

danielmarkbruce 1y ago

Why not just use claude code and come to your own conclusion?