4 ms·

> I can do things while the machine works I find this workflow makes my life extremely tedious, because reviewing and fixing whatever the machine produces is a

> I can do things while the machine works I find this workflow makes my life extremely tedious, because reviewing and fixing whatever the machine produces is a slog for me I suppose it would be exhilarating if I just trusted the output but somehow I just can't bring myself to do that How do you reconcile this with your own work? Maybe you just skim the output? Or do you run it and test that way? Please don't tell me you just trust the AI to write automated tests or something...

the_mitsuhiko 1y ago
> I find this workflow makes my life extremely tedious, because reviewing and fixing whatever the machine produces is a slog for me I don't actually mind the code review process, it's what I'm used to with open source contributions. Not infrequently did I look at a PR that people left me and decide to rewrite it entirely anyways because I did not quite appreciate the method they chose. That however did not mean the PR was useless, it might have provided the necessary input that made me go down that path ultimately. I also think that what can make code review enjoyable is not necessarily because the person on the other side learns something from it. I actually tend to think that this something that does not necessarily happen in PR review, but in 1:1 conversations in person or though more synchronous collaboration. So really my answer is that I do not mind code review. That said there is an important part here: > How do you reconcile this with your own work? Maybe you just skim the output? Or do you run it and test that way? What makes agentic workflows work is good tooling and fast iteration cyclces. That is also exactly what a human needs for a PR review to be actually thorough. I cannot accept a change when I do not know that it exhibits the intended behavior, and to validate that behavior I usually need a test case. Do I need to run the test case myself? Not necessarily because the agent / CI already does for me. However I do sometimes run it with verbose output to also see the intermediate outputs that might otherwise be hidden from me to give me more confidence. It really is hard to describe, but when you put the necessary effort in to make an agentic workflow work, it's really enjoyable. But it does require some upfront investment for it to work. > Please don't tell me you just trust the AI to write automated tests or something... I do in fact trust the AI a lot to write tests. They have become incredibly good at taking certain requirements definitions and derive something that can be mostly trusted. It does not always work, but you get a pretty good feeling for if it will or not. At the end of the day many of the tasks we do are basic CRUD or data transformation tasks.
- starkparker 1y ago
  > It really is hard to describe, but when you put the necessary effort in to make an agentic workflow work, it's really enjoyable. I wish it wasn't hard to describe because every attempt I've made to reach that point with agents/tools-in-a-loop has ended with more frustration, more errors, and more broken code than when I started, even with common and small, tedious tasks. I'd very much like to understand what proponents are doing and seeing that works.
  - NoOn3 1y ago
    It would be interesting to watch a video with an example of effective work. To see why are we so unlucky with results:)
    - solarwindy 1y ago
      All too often with AI proponents I get the you’re holding it wrong vibe. I can’t help but develop the suspicion these people are putting out bad work and just never notice. Or to be a little less scathing, fine, there’s a very narrow range of tasks where these models can usefully alleviate drudgery—quite often that comes at the cost of code bloat, since a generative model inherently makes it easy to puke out code with total indifference to its burden over time.
      - ookblah 1y ago
        there was literally a post a few days ago where someone from cloudflare or fly (can't remember) released a new feature built with AI (albeit heavily supervised) and answered q's along the way about it. it's funny because what i'd also like to see is people who are skeptics make a video as well since sometimes i also have the opposite suspicion. i get a lot of the criticism, but i don't get the "it produces pure garbage" type ones.
        solarwindy 1y ago
        My comment was admittedly a little inflammatory, so I’ll try to elaborate on what I find so exasperating about the whole experience of using LLMs for coding (in any form, chat, agent, etc.): at root, it’s their total lack of a model of computation. This leads to mistakes that are difficult to catch, because while I know the kinds of mistakes another human might make, having been through the process of learning not to make them myself, LLMs produce whole classes of bizarre mistakes that I have no interest in learning to catch. There is no discernible flawed mental model behind these errors—which with a human could be discussed and corrected—just an opaque stochastic process which I can tediously try to set on a better course with ‘incantations’, attempting to dial in to a better part of the training data that avoids the relevant class of error. It’s honestly amusing how much of ‘prompt engineering’ (if it can be dignified with that term) amounts to a modern-day kind of mysticism. What better can we really do though when these models’ structure and operation is utterly opaque, on the one hand through deliberate, commercially-oriented obfuscation, and on the other because we still just cannot explain how a multi-billion parameter model works. It’s rewarding to work with human juniors because people actually learn and improve. My learning how to coax these models into producing better than trash just is not, especially on anything either remotely novel, or in a legacy codebase that requires genuine understanding of how an existing system functions at runtime. Once those two ends of the spectrum are ruled out, I find there’s little left that an LLM can accomplish, without necessitating a loop of prompt refinement that leaves me feeling like a worse developer for not just having thought through the problem myself, and resentful of the time wasted. Edit: this entire thread about coercing Gemini into behaving is exactly the kind of crap I have zero interest in: https://news.ycombinator.com/item?id=44194061 https://news.ycombinator.com/item?id=44194061
tuckerman 1y ago
Not GP but > I find this workflow makes my life extremely tedious, because reviewing and fixing whatever the machine produces is a slog for me This is a matter of personal preference, but reviewing code (from humans) was already a huge chunk of my job and one that I enjoy. Now I can have a similar workflow for side projects, especially for things that I find less enjoyment in coding myself. > Please don't tell me you just trust the AI to write automated tests or something... Like most tools, the more I use LLMs for coding the more I get a feel for what its good at, what its okay at, and what it will just make a mess of. I find for changes like extending existing test coverage before refactoring, I can often just skim the output. For writing tests from whole cloth it requires a lot more attention (depending on the area). For non-test code, it very much depends on how novel the work is and how much context/many examples it has to go on.
CuriouslyC 1y ago
Create stubs to ensure code structure roughly aligns with expectations, then tell the AI to generate code and paired tests and look at code test coverage and manually inspect the tests rather than inspecting the code directly.
osigurdson 1y ago
In my experience, the machine works for 10-30s. I think the only productive use of time during those gaps is meditation. Still however, flow state cannot be achieved. Vibe coding, for me is pretty horrible for anything that might approximate production code. It is useful for quick prototyping or used as a learning tool. I assume it will get better but a far more satisfying experience would be for it to work offline, creating PRs for incremental improvements that are easy to review. I suppose this is something like the "Devin" model but I haven't used this due to the price and have heard bad things about it.
- nojs 1y ago
  > I think the only productive use of time during those gaps is meditation Check out several copies of the repo and work on different branches concurrently.
  - osigurdson 1y ago
    I suspect the context switches would kill productivity. This is not where the puck is going in my opinion.
- osigurdson 1y ago
  Just like magic, OpenAI codex is now available in my ChatGPT+ subscription. I greatly prefer this async style workflow. For me, getting a useful PR is so much better than banging my head against the wall with synchronous tools like Windsurf.

the_mitsuhiko 1y ago

> I find this workflow makes my life extremely tedious, because reviewing and fixing whatever the machine produces is a slog for me I don't actually mind the code review process, it's what I'm used to with open source contributions. Not infrequently did I look at a PR that people left me and decide to rewrite it entirely anyways because I did not quite appreciate the method they chose. That however did not mean the PR was useless, it might have provided the necessary input that made me go down that path ultimately. I also think that what can make code review enjoyable is not necessarily because the person on the other side learns something from it. I actually tend to think that this something that does not necessarily happen in PR review, but in 1:1 conversations in person or though more synchronous collaboration. So really my answer is that I do not mind code review. That said there is an important part here: > How do you reconcile this with your own work? Maybe you just skim the output? Or do you run it and test that way? What makes agentic workflows work is good tooling and fast iteration cyclces. That is also exactly what a human needs for a PR review to be actually thorough. I cannot accept a change when I do not know that it exhibits the intended behavior, and to validate that behavior I usually need a test case. Do I need to run the test case myself? Not necessarily because the agent / CI already does for me. However I do sometimes run it with verbose output to also see the intermediate outputs that might otherwise be hidden from me to give me more confidence. It really is hard to describe, but when you put the necessary effort in to make an agentic workflow work, it's really enjoyable. But it does require some upfront investment for it to work. > Please don't tell me you just trust the AI to write automated tests or something... I do in fact trust the AI a lot to write tests. They have become incredibly good at taking certain requirements definitions and derive something that can be mostly trusted. It does not always work, but you get a pretty good feeling for if it will or not. At the end of the day many of the tasks we do are basic CRUD or data transformation tasks.

starkparker 1y ago
> It really is hard to describe, but when you put the necessary effort in to make an agentic workflow work, it's really enjoyable. I wish it wasn't hard to describe because every attempt I've made to reach that point with agents/tools-in-a-loop has ended with more frustration, more errors, and more broken code than when I started, even with common and small, tedious tasks. I'd very much like to understand what proponents are doing and seeing that works.
- NoOn3 1y ago
  It would be interesting to watch a video with an example of effective work. To see why are we so unlucky with results:)
  - solarwindy 1y ago
    All too often with AI proponents I get the you’re holding it wrong vibe. I can’t help but develop the suspicion these people are putting out bad work and just never notice. Or to be a little less scathing, fine, there’s a very narrow range of tasks where these models can usefully alleviate drudgery—quite often that comes at the cost of code bloat, since a generative model inherently makes it easy to puke out code with total indifference to its burden over time.
    - ookblah 1y ago
      there was literally a post a few days ago where someone from cloudflare or fly (can't remember) released a new feature built with AI (albeit heavily supervised) and answered q's along the way about it. it's funny because what i'd also like to see is people who are skeptics make a video as well since sometimes i also have the opposite suspicion. i get a lot of the criticism, but i don't get the "it produces pure garbage" type ones.
      - solarwindy 1y ago
        My comment was admittedly a little inflammatory, so I’ll try to elaborate on what I find so exasperating about the whole experience of using LLMs for coding (in any form, chat, agent, etc.): at root, it’s their total lack of a model of computation. This leads to mistakes that are difficult to catch, because while I know the kinds of mistakes another human might make, having been through the process of learning not to make them myself, LLMs produce whole classes of bizarre mistakes that I have no interest in learning to catch. There is no discernible flawed mental model behind these errors—which with a human could be discussed and corrected—just an opaque stochastic process which I can tediously try to set on a better course with ‘incantations’, attempting to dial in to a better part of the training data that avoids the relevant class of error. It’s honestly amusing how much of ‘prompt engineering’ (if it can be dignified with that term) amounts to a modern-day kind of mysticism. What better can we really do though when these models’ structure and operation is utterly opaque, on the one hand through deliberate, commercially-oriented obfuscation, and on the other because we still just cannot explain how a multi-billion parameter model works. It’s rewarding to work with human juniors because people actually learn and improve. My learning how to coax these models into producing better than trash just is not, especially on anything either remotely novel, or in a legacy codebase that requires genuine understanding of how an existing system functions at runtime. Once those two ends of the spectrum are ruled out, I find there’s little left that an LLM can accomplish, without necessitating a loop of prompt refinement that leaves me feeling like a worse developer for not just having thought through the problem myself, and resentful of the time wasted. Edit: this entire thread about coercing Gemini into behaving is exactly the kind of crap I have zero interest in: https://news.ycombinator.com/item?id=44194061 https://news.ycombinator.com/item?id=44194061

tuckerman 1y ago

Not GP but > I find this workflow makes my life extremely tedious, because reviewing and fixing whatever the machine produces is a slog for me This is a matter of personal preference, but reviewing code (from humans) was already a huge chunk of my job and one that I enjoy. Now I can have a similar workflow for side projects, especially for things that I find less enjoyment in coding myself. > Please don't tell me you just trust the AI to write automated tests or something... Like most tools, the more I use LLMs for coding the more I get a feel for what its good at, what its okay at, and what it will just make a mess of. I find for changes like extending existing test coverage before refactoring, I can often just skim the output. For writing tests from whole cloth it requires a lot more attention (depending on the area). For non-test code, it very much depends on how novel the work is and how much context/many examples it has to go on.

CuriouslyC 1y ago

Create stubs to ensure code structure roughly aligns with expectations, then tell the AI to generate code and paired tests and look at code test coverage and manually inspect the tests rather than inspecting the code directly.

osigurdson 1y ago

In my experience, the machine works for 10-30s. I think the only productive use of time during those gaps is meditation. Still however, flow state cannot be achieved. Vibe coding, for me is pretty horrible for anything that might approximate production code. It is useful for quick prototyping or used as a learning tool. I assume it will get better but a far more satisfying experience would be for it to work offline, creating PRs for incremental improvements that are easy to review. I suppose this is something like the "Devin" model but I haven't used this due to the price and have heard bad things about it.

nojs 1y ago
> I think the only productive use of time during those gaps is meditation Check out several copies of the repo and work on different branches concurrently.
- osigurdson 1y ago
  I suspect the context switches would kill productivity. This is not where the puck is going in my opinion.
osigurdson 1y ago
Just like magic, OpenAI codex is now available in my ChatGPT+ subscription. I greatly prefer this async style workflow. For me, getting a useful PR is so much better than banging my head against the wall with synchronous tools like Windsurf.