4 ms·

I don't really get this. At this point, my limiting factor is not how quickly Claude can self-trudge through code. It's whether Claude is going to do the task c

I don't really get this. At this point, my limiting factor is not how quickly Claude can self-trudge through code. It's whether Claude is going to do the task correctly or not. I need more mechanisms for controlling long-running sessions and dynamically injecting my thoughts, correction, and nudges rather than faster ways to burn through my tokens without knowing if the results are going to be correct.

jascha_eng 5mo ago
yes I agree with this, more granular going back, letting me interrupt where it went off the rails, or even editing file reads myself etc would be lovely. Ingesting parts of other conversations would also be cool!
dude250711 5mo ago
I have heard of "token-maxxing" but I have not heard of "correctness-maxxing" or "quality-maxxing".
- mirashii 5mo ago
  Not with those exact terms, but it is certainly being discussed. Wes McKinney said in a recent talk that with current coding agents there’s no longer an excuse for shipping suboptimal code that takes on tech debt. Writing tests has never been cheaper, writing custom fuzzers, linters, and other harnesses that serve as guardrails has never been cheaper. His take is that “we didn’t have enough engineering time to do it right” is no longer an excuse, and the only excuses left are that you don’t know any better or you have bad taste.
wrs 5mo ago
I think the theoretical answer here is this: "Agents address the problem from independent angles, other agents try to refute what they found, and the run keeps iterating until the answers converge." So you will be supplying the "ground truth" (test suite, detailed spec, whatever) and empower an agent to use it to guide the other agents. Currently a lot of people do this sequentially in the form of multiple code-review passes by fresh agent sessions looking at the work of previous sessions. Adversarial models are a longstanding technique in ML so it makes sense they would try to go this way.
- rendaw 5mo ago
  So, theoretically, in order to get claude to perform properly you just have to have a perfect spec up front, that anticipates all issues and contains no ambiguity?
- tsunamifury 5mo ago
  Ground truth is not consensus, it has to be graded against what actually works for the original goal. Plenty of scenarios with AI and Humans can result in consensus around incorrectness.
  - adamtaylor_13 5mo ago
    While pedantically correct, I think the comment above assumed that you've correctly specified the work. If you can't correctly specify your work, AI agents are just going to help you get a non-solution faster.
    - tsunamifury 5mo ago
      Isn't coding the act of specificying the work to a processor? And AI agents are supposed to bridge the gap with intelligence from less specificed to more specified or possibly even more intelligent and alternate implementations?
      - adamtaylor_13 5mo ago
        Yep. And yet, there's still some level of specification you have to do.
      - wrs 5mo ago
        What I meant by "ground truth" is that it is not fuzzy, not AI-evaluated, and not a consensus. The test suite passes or it doesn't. The codebase lints or it doesn't. The performance improved or it didn't. An agent can help you create the specification, but it's up to you to know whether it's correctly testing that you got the result you wanted.
- 5mo ago
root-parent 5mo ago
When this is all finished and done, these coding models will allow you to rewrite the linux kernel in rust, recode Kubernetes in assembly, and create your own web framework in 10 min. But each prompt will cost your company, 10 to 15 million dollars. An extra 20 million if you ask them to review the code and improve the comments.
Jarred 5mo ago
Dynamic workflows, in my experience, make Claude more effective at complex long-running tasks. They help precisely with getting Claude to do the task correctly. It feels more like a bespoke build system for the specific task/project than prompting a freeform chat.
- aloknnikhil 5mo ago
  As long as agents are fuzzy (which they will continue to be with the Transformers architecture), the need to validate will continue to exist. I cannot imagine merging code without at least 1 human review.
  - MeetingsBrowser 5mo ago
    I've used agents quite a bit and I agree. The current baseline workflow is something like agent output -> human review -> agent refinement -> human review -> agent refinement -> ... But agents are capable of making meaningful improvements to their own output. I'm hoping dynamic workflows move towards something like: agent output -> agent review -> agent refinement -> (cycle to fixed point) -> final human review
    - deleted 5mo ago
      [deleted]
- deleted 5mo ago
  [deleted]
chinathrow 5mo ago
I think for now it's better to convert tokens into code/library code and then work with that for deterministic results rather than relying on Claude being correct or not.
eggplantemoji69 5mo ago
This is my experience. Quantity of output is not the issue right now. Quality is. But I’m not sure if this will ever be solved for, given LLMs are non-deterministic sophisticated autocomplete at their core. Sure, ‘human in the loop’ and all that jazz, but I feel like my knowledge suffers even with this approach. I have to use llms w pinpoint focus to get decent results. The original copilot completions behavior might be peak llm performance for coding, sans having an agent write boilerplate and such.
encoderer 5mo ago
The answer for me has been actually more tokens, and create even more layers of automated verification
sfourdrinier 5mo ago
Yes, that, accuracy, speed, and single computer-use. I find those to be the limiting factors to speed. I have extensive rules, I do extensive planning. Yet at implementation, the rules are not respected, errors are introduced, etc... I spend more time fixing than writing code. Then speed... Because of the fixes and bad code quality even with frontiers model speed makes a very big difference. I (agents) spend hours daily doing reviews and fixes. 5x speed boost would make me much more productive. And when working super fast with agents, having only one computer is limiting. Even worktrees don't solve problems because I use things like convex, chrome use, etc... and it conflicts with each others all the time. Still many problems to solve. It's already evolved so much in the last two years.
clickety_clack 5mo ago
A more interactive Claude code would be great instead of 50 “here’s a tiny snapshot of a change shorn of the context you need to understand it. Yes or no?”s
brammertottens 5mo ago
[dead]
stefanhorne 5mo ago
[flagged]
Phoenixhq 4mo ago
[flagged]

jascha_eng 5mo ago

yes I agree with this, more granular going back, letting me interrupt where it went off the rails, or even editing file reads myself etc would be lovely. Ingesting parts of other conversations would also be cool!

dude250711 5mo ago

I have heard of "token-maxxing" but I have not heard of "correctness-maxxing" or "quality-maxxing".

mirashii 5mo ago
Not with those exact terms, but it is certainly being discussed. Wes McKinney said in a recent talk that with current coding agents there’s no longer an excuse for shipping suboptimal code that takes on tech debt. Writing tests has never been cheaper, writing custom fuzzers, linters, and other harnesses that serve as guardrails has never been cheaper. His take is that “we didn’t have enough engineering time to do it right” is no longer an excuse, and the only excuses left are that you don’t know any better or you have bad taste.

wrs 5mo ago

I think the theoretical answer here is this: "Agents address the problem from independent angles, other agents try to refute what they found, and the run keeps iterating until the answers converge." So you will be supplying the "ground truth" (test suite, detailed spec, whatever) and empower an agent to use it to guide the other agents. Currently a lot of people do this sequentially in the form of multiple code-review passes by fresh agent sessions looking at the work of previous sessions. Adversarial models are a longstanding technique in ML so it makes sense they would try to go this way.

rendaw 5mo ago
So, theoretically, in order to get claude to perform properly you just have to have a perfect spec up front, that anticipates all issues and contains no ambiguity?
tsunamifury 5mo ago
Ground truth is not consensus, it has to be graded against what actually works for the original goal. Plenty of scenarios with AI and Humans can result in consensus around incorrectness.
- adamtaylor_13 5mo ago
  While pedantically correct, I think the comment above assumed that you've correctly specified the work. If you can't correctly specify your work, AI agents are just going to help you get a non-solution faster.
  - tsunamifury 5mo ago
    Isn't coding the act of specificying the work to a processor? And AI agents are supposed to bridge the gap with intelligence from less specificed to more specified or possibly even more intelligent and alternate implementations?
    - adamtaylor_13 5mo ago
      Yep. And yet, there's still some level of specification you have to do.
    - wrs 5mo ago
      What I meant by "ground truth" is that it is not fuzzy, not AI-evaluated, and not a consensus. The test suite passes or it doesn't. The codebase lints or it doesn't. The performance improved or it didn't. An agent can help you create the specification, but it's up to you to know whether it's correctly testing that you got the result you wanted.
5mo ago

root-parent 5mo ago

When this is all finished and done, these coding models will allow you to rewrite the linux kernel in rust, recode Kubernetes in assembly, and create your own web framework in 10 min. But each prompt will cost your company, 10 to 15 million dollars. An extra 20 million if you ask them to review the code and improve the comments.

Jarred 5mo ago

Dynamic workflows, in my experience, make Claude more effective at complex long-running tasks. They help precisely with getting Claude to do the task correctly. It feels more like a bespoke build system for the specific task/project than prompting a freeform chat.

aloknnikhil 5mo ago
As long as agents are fuzzy (which they will continue to be with the Transformers architecture), the need to validate will continue to exist. I cannot imagine merging code without at least 1 human review.
- MeetingsBrowser 5mo ago
  I've used agents quite a bit and I agree. The current baseline workflow is something like agent output -> human review -> agent refinement -> human review -> agent refinement -> ... But agents are capable of making meaningful improvements to their own output. I'm hoping dynamic workflows move towards something like: agent output -> agent review -> agent refinement -> (cycle to fixed point) -> final human review
  - deleted 5mo ago
    [deleted]
deleted 5mo ago
[deleted]

chinathrow 5mo ago

I think for now it's better to convert tokens into code/library code and then work with that for deterministic results rather than relying on Claude being correct or not.

eggplantemoji69 5mo ago

This is my experience. Quantity of output is not the issue right now. Quality is. But I’m not sure if this will ever be solved for, given LLMs are non-deterministic sophisticated autocomplete at their core. Sure, ‘human in the loop’ and all that jazz, but I feel like my knowledge suffers even with this approach. I have to use llms w pinpoint focus to get decent results. The original copilot completions behavior might be peak llm performance for coding, sans having an agent write boilerplate and such.

encoderer 5mo ago

The answer for me has been actually more tokens, and create even more layers of automated verification

sfourdrinier 5mo ago

Yes, that, accuracy, speed, and single computer-use. I find those to be the limiting factors to speed. I have extensive rules, I do extensive planning. Yet at implementation, the rules are not respected, errors are introduced, etc... I spend more time fixing than writing code. Then speed... Because of the fixes and bad code quality even with frontiers model speed makes a very big difference. I (agents) spend hours daily doing reviews and fixes. 5x speed boost would make me much more productive. And when working super fast with agents, having only one computer is limiting. Even worktrees don't solve problems because I use things like convex, chrome use, etc... and it conflicts with each others all the time. Still many problems to solve. It's already evolved so much in the last two years.

clickety_clack 5mo ago

A more interactive Claude code would be great instead of 50 “here’s a tiny snapshot of a change shorn of the context you need to understand it. Yes or no?”s

brammertottens 5mo ago

[dead]

stefanhorne 5mo ago

[flagged]

Phoenixhq 4mo ago

[flagged]