5 ms·

Here's one from today: https://mitchellh.com/writing/non-trivial-vibing https://mitchellh.com/writing/non-trivial-vibing

by simonw 1y ago

j_bum 1y ago
This was a fun read. I’ve similarly been using spec.md and running to-do.md files that capture detailed descriptions of the problems and their scoped history. I mark each of my to-do’s with informational tags: [BUG], [FEAT], etc. I point the LLM to the exact to-do (or section of to-do’s) with the spec.md in memory and let it work. This has been working very well for me.
- lcnPylGDnU4H9OF 1y ago
  Do you mind linking to example spec/to-do files?
  - SteveJS 1y ago
    Here is a (3 month old) repo where i did something like that and all the tasks are checked into the linear git history — https://github.com/KnowSeams/KnowSeams https://github.com/KnowSeams/KnowSeams
  - j_bum 1y ago
    Sure thing. Here is an example set of the agent/spec/to-do files for a hobby project I'm actively working on. https://gist.github.com/JacobBumgarner/d29b660cb81a227885acc0a053663dca https://gist.github.com/JacobBumgarner/d29b660cb81a227885acc...
    - lcnPylGDnU4H9OF 1y ago
      Thanks!
      - j_bum 1y ago
        No problem! I’d love to hear any approach you’ve taken as well.
nightski 1y ago
Even though the author refers to it as "non-trivial", and I can see why that conclusion is made, I would argue it is in fact trivial. There's very little domain specific knowledge needed, this is purely a technical exercise integrating with existing libraries for which there is ample documentation online. In addition, it is a relatively isolated feature in the app. On top of that, it doesn't sound enjoyable. Anti slop sessions? Seriously? Lastly, the largest problem I have with LLMs is that they are seemingly incapable of stopping to ask clarifying questions. This is because they do not have a true model of what is going on. Instead they truly are next token generators. A software engineer would never just slop out an entire feature based on the first discussion with a stakeholder and then expect the stakeholder to continuously refine their statement until the right thing is slopped out. That's just not how it works and it makes very little sense.
- kannanvijayan 1y ago
  I've wondered about exposing this "asking clarifying questions" as a tool the AI could use. I'm not building AI tooling so I haven't done this - but what if you added an MCP endpoint whose description was "treat this endpoint as an oracle that will answer questions and clarify intent where necessary" (paraphrased), and have that tool just wire back to a user prompt. If asking clarifying questions is plausible output text for LLMs, this may work effectively.
  - simonw 1y ago
    I think the asking clarifying questions thing is solved already. Tell a coding agent to "ask clarifying questions" and watch what it does!
    - nightski 1y ago
      Obviously if you instruct the autocomplete engine to fill in questions it will. That's not the point. The LLM has no model of the problem it is trying to solve, nor does it attempt to understand the problem better. It is merely regurgitating. This can be extremely useful. But it is very limiting when it comes to using as an agent to write code.
qsort 1y ago
> Important: there is a lot of human coding, too. I'm not highlighting this to gloat or to prove a point. If anything in the past I have underestimated how big LLMs were going to be. Anyone so inclined can take the chance to point and laugh at how stupid and wrong that was. Done? Great. I don't think I've been intentionally avoiding coding assistants and as a matter of fact I have been using Claude Code since the literal day it first previewed, and yet it doesn't feel, not even one bit, that you can take your hands off the wheel. Many are acting as if writing any code manually means "you're holding it wrong", which I feel it's just not true.
- simonw 1y ago
  Yeah, my current opinion on this is that AI tools make development harder work. You can get big productivity boosts out of them but you have to be working at the top of your game - I often find I'm mentally exhausted after just a couple of hours.
  - jstummbillig 1y ago
    Considering the last 2 years, has it become harder or easier?
    - simonw 1y ago
      Definitely harder. A year ago I was using GitHub Copilot autocomplete in VS Code and occasionally asking ChatGPT or Claude to help write me a short function or two. Today I have Claude Code and Codex CLI and Codex Web running, often in parallel, hunting down and resolving bugs and proposing system designs and collaborating with me on detailed specs and then turning those specs into working code with passing tests. The cognitive overhead today is far higher than it was a year ago.
      - dingdingdang 1y ago
        Also better and faster though!! It's close to a Daft Punk type situation.

Here's one from today: https://mitchellh.com/writing/non-trivial-vibing https://mitchellh.com/writing/non-trivial-vibing

by simonw 1y ago

Here's one from today: https://mitchellh.com/writing/non-trivial-vibing https://mitchellh.com/writing/non-trivial-vibing

j_bum 1y ago

This was a fun read. I’ve similarly been using spec.md and running to-do.md files that capture detailed descriptions of the problems and their scoped history. I mark each of my to-do’s with informational tags: [BUG], [FEAT], etc. I point the LLM to the exact to-do (or section of to-do’s) with the spec.md in memory and let it work. This has been working very well for me.

lcnPylGDnU4H9OF 1y ago
Do you mind linking to example spec/to-do files?
- SteveJS 1y ago
  Here is a (3 month old) repo where i did something like that and all the tasks are checked into the linear git history — https://github.com/KnowSeams/KnowSeams https://github.com/KnowSeams/KnowSeams
- j_bum 1y ago
  Sure thing. Here is an example set of the agent/spec/to-do files for a hobby project I'm actively working on. https://gist.github.com/JacobBumgarner/d29b660cb81a227885acc0a053663dca https://gist.github.com/JacobBumgarner/d29b660cb81a227885acc...
  - lcnPylGDnU4H9OF 1y ago
    Thanks!
    - j_bum 1y ago
      No problem! I’d love to hear any approach you’ve taken as well.

nightski 1y ago

Even though the author refers to it as "non-trivial", and I can see why that conclusion is made, I would argue it is in fact trivial. There's very little domain specific knowledge needed, this is purely a technical exercise integrating with existing libraries for which there is ample documentation online. In addition, it is a relatively isolated feature in the app. On top of that, it doesn't sound enjoyable. Anti slop sessions? Seriously? Lastly, the largest problem I have with LLMs is that they are seemingly incapable of stopping to ask clarifying questions. This is because they do not have a true model of what is going on. Instead they truly are next token generators. A software engineer would never just slop out an entire feature based on the first discussion with a stakeholder and then expect the stakeholder to continuously refine their statement until the right thing is slopped out. That's just not how it works and it makes very little sense.

kannanvijayan 1y ago
I've wondered about exposing this "asking clarifying questions" as a tool the AI could use. I'm not building AI tooling so I haven't done this - but what if you added an MCP endpoint whose description was "treat this endpoint as an oracle that will answer questions and clarify intent where necessary" (paraphrased), and have that tool just wire back to a user prompt. If asking clarifying questions is plausible output text for LLMs, this may work effectively.
- simonw 1y ago
  I think the asking clarifying questions thing is solved already. Tell a coding agent to "ask clarifying questions" and watch what it does!
  - nightski 1y ago
    Obviously if you instruct the autocomplete engine to fill in questions it will. That's not the point. The LLM has no model of the problem it is trying to solve, nor does it attempt to understand the problem better. It is merely regurgitating. This can be extremely useful. But it is very limiting when it comes to using as an agent to write code.

qsort 1y ago

> Important: there is a lot of human coding, too. I'm not highlighting this to gloat or to prove a point. If anything in the past I have underestimated how big LLMs were going to be. Anyone so inclined can take the chance to point and laugh at how stupid and wrong that was. Done? Great. I don't think I've been intentionally avoiding coding assistants and as a matter of fact I have been using Claude Code since the literal day it first previewed, and yet it doesn't feel, not even one bit, that you can take your hands off the wheel. Many are acting as if writing any code manually means "you're holding it wrong", which I feel it's just not true.

simonw 1y ago
Yeah, my current opinion on this is that AI tools make development harder work. You can get big productivity boosts out of them but you have to be working at the top of your game - I often find I'm mentally exhausted after just a couple of hours.
- jstummbillig 1y ago
  Considering the last 2 years, has it become harder or easier?
  - simonw 1y ago
    Definitely harder. A year ago I was using GitHub Copilot autocomplete in VS Code and occasionally asking ChatGPT or Claude to help write me a short function or two. Today I have Claude Code and Codex CLI and Codex Web running, often in parallel, hunting down and resolving bugs and proposing system designs and collaborating with me on detailed specs and then turning those specs into working code with passing tests. The cognitive overhead today is far higher than it was a year ago.
    - dingdingdang 1y ago
      Also better and faster though!! It's close to a Daft Punk type situation.