5 ms·
I've found that an effective tactic for larger, more complex tasks is to tell it "Don't write any code now. I'm going to describe each of the steps of the prob
by nostrademons 1y ago
I've found that an effective tactic for larger, more complex tasks is to tell it "Don't write any code now. I'm going to describe each of the steps of the problem in more detail. The rough outline is going to be 1) Read this input 2) Generate these candidates 3) apply heuristics to score candidates 4) prioritize and rank candidates 5) come up with this data structure reflecting the output 6) write the output back to the DB in this schema". Claude will then go and write a TODO list in the code (and possibly claude.md if you've run /init), and prompt you for the details of each stage. I've even done this for an hour, told Claude "I have to stop now. Generate code for the finished stages and write out comments so you can pick up where you left off next time" and then been able to pick up next time with minimal fuss.
- hex4def6 1y agoFYI: You can force "Plan mode" by pressing shift-tab. That will prevent it from eagerly implementing stuff.
- jaggederest 1y ago> That will prevent it from eagerly implementing stuff. In theory. In practice, it's not a very secure sandbox and Claude will happily go around updating files if you insist / the prompt is bad / it goes off on a tangent. I really should just set up a completely sandboxed VM for it so that I don't care if it goes rm-rf happy.
- adastra22 1y agoPlan mode disabled the tools, so I don’t see how it would do that. A sandboxed devcontainer is worth setting up though. Lets me run it with —dangerously-skip-permissions
- jaggederest 1y agoI don't know either but I've seen it write to files in plan mode. Very confusing.
- EnPissant 1y agoThat's not possible. You are misremembering.
- sshine 1y agoI've seen it run commands that are naively assumed to be reading files or searching directories. I.e. not its own tools, but command-line executables. Its assumptions about these commands, and specifically the way it ran them, were correct. But I have seen it run commands in plan mode.
- nomoreofthat 1y agoIt’s entirely possible. Claude’s security model for subagents/tasks is incoherent and buggy, far below the standard they set elsewhere in their product, and planning mode can use subagent/tasks for research. Permission limitations on the root agent have, in many cases, not been propagated to child agents, and they’ve been able to execute different commands. The documentation is incomplete and unclear, and even to the extent that it is clear it has a different syntax with different limitations than are used to configure permissions for the root agent. When you ask Claude itself to generate agent configurations, as is recommended, it will generate permissions that do not exist anywhere in the documentation and may or may not be valid, but there’s no error admitted if an invalid permission is set. If you ask it to explain, it gets confused by their own documentation and tells you it doesn’t know why it did that. I’m not sure if it’s hallucinating or if the agent-generating-agent has access to internal detail details that are not documented anywhere in which the normal agent can’t see. Anthropic is pretty consistently the best in this space in terms of security and product quality. They seem to actually care about doing software engineering properly. (I’ve personally discovered security bugs in several competing products that are more severe and exploitable than what I’m talking about here.) I have a ton of respect for Anthropic. Unfortunately, when it comes to sub agents in Claude code, they are not living up to standard they have set.
- laborcontract 1y agoNo, it is possible. I just got it to write files both using Bash and its Write tools while in plan mode right now.
- faangguyindia 1y agohow can it plan if it does not have access to file read, search, bash tools to investigate things? If it has access to bash tools then it's going to write code, via echo or sed.
- adastra22 1y agoIt has file read, search, but not bash AFAIK.
- theshrike79 1y agoI've had it do it in plan mode. Nothing dangerous, but the limits are more like suggestions, as the Pirate code says.
- yahoozoo 1y agoHow does a token predictor “apply heuristics to score candidates”? Is it running a tool, such as a Python script it writes for scoring candidates? If not, isn’t it just pulling some statistically-likely “score” out of its weights rather than actually calculating one?
- astrange 1y agoToken prediction is the interface. The implementation is a universal function approximator communicating through the token weights.
- imtringued 1y agoYou can think of the K(=key) matrix in attention as a neural network where each token is turned into a tiny classifier network with multiple inputs and a single output. The softmax activation function picks the most promising activations for a given output token. The V(=value) matrix forms another neural network where each token is turned into a tiny regressor neural network that accepts the activation as an input and produces multiple outputs that are summed up to produce an intermediate token which is then fed into the MLP layer. From this perspective the transformer architecture is building neural networks at runtime. But there are some pretty obvious limitations here: The LLM operates on tokens, which means it can only operate on what is in the KV-cache/context window. If the candidates are not in the context window, it can't score them.
- yahoozoo 1y agoI’m not sure if I’m just misunderstanding or we are talking about two different things. I know at a high level how transformers/LLMs decide its next token in the response it is generating. My question to the post I replied to was basically: given a coding problem, and a list of possible solutions (candidates), how can a LLM generate a meaningful numerical score for each candidate to then say this one is a better solution than that one?