3 ms·

To avoid stuff like this I've tried to prompt for only the necessary changes. I haven't found a good prompt that does this repeatably. It might be worth some ex

by all2 3y ago

To avoid stuff like this I've tried to prompt for only the necessary changes. I haven't found a good prompt that does this repeatably. It might be worth some experimentation.

anigbrowl 3y ago
One can usually get around it by modifying the prompt, but one really shouldn't have to include things like 'do not remove comments' over and over. My IDE (shoutout to Jetbrains) lets me make custom prompts, but the real problems here are the transformer model's inability to maintain context or ask questions of its own, more readily, and a lack of visual grammar for 'what are we both talking about, and how'. Pair programming has gone some way toward improving that in recent years, but realistically transformers are getting to where they can spit out code faster than most people can compose or type it (not you of course). Looking at live diffs is not ideal, especially late in a coding session when fatigue is setting in. Some sort of shared context map for functions and variables might be a better approach. As things are I find myself using a clumsy mix of save-as and commits and often throwing away hours of work when an assistant is unable to backtrack.
CGamesPlay 3y ago
I've used Aider <https://aider.chat https://aider.chat> a fair amount, and I have read about Paul's experiments with different "diff producing" methods. I think a good avenue to experiment is two-pass. First, ask the LLM to output only the changed code, then (in a separate conversion) as the LLM just to update the relevant lines with the change the first LLM provided. I suspect this would clear up a lot of the "LLM failed to transcribe old code" and "diff isn't well-formed" errors that this tool has.
- all2 3y ago
  This seems ripe for using "cores" or specialist pipelines/agents to handle stuff like that. I may see what I can scrape together.

by all2 3y ago

To avoid stuff like this I've tried to prompt for only the necessary changes. I haven't found a good prompt that does this repeatably. It might be worth some experimentation.

anigbrowl 3y ago

One can usually get around it by modifying the prompt, but one really shouldn't have to include things like 'do not remove comments' over and over. My IDE (shoutout to Jetbrains) lets me make custom prompts, but the real problems here are the transformer model's inability to maintain context or ask questions of its own, more readily, and a lack of visual grammar for 'what are we both talking about, and how'. Pair programming has gone some way toward improving that in recent years, but realistically transformers are getting to where they can spit out code faster than most people can compose or type it (not you of course). Looking at live diffs is not ideal, especially late in a coding session when fatigue is setting in. Some sort of shared context map for functions and variables might be a better approach. As things are I find myself using a clumsy mix of save-as and commits and often throwing away hours of work when an assistant is unable to backtrack.

CGamesPlay 3y ago

I've used Aider <https://aider.chat https://aider.chat> a fair amount, and I have read about Paul's experiments with different "diff producing" methods. I think a good avenue to experiment is two-pass. First, ask the LLM to output only the changed code, then (in a separate conversion) as the LLM just to update the relevant lines with the change the first LLM provided. I suspect this would clear up a lot of the "LLM failed to transcribe old code" and "diff isn't well-formed" errors that this tool has.

all2 3y ago
This seems ripe for using "cores" or specialist pipelines/agents to handle stuff like that. I may see what I can scrape together.