3 ms·

I feel like the copy and paste thing is overdue a solution. I find this one particularly frustrating when working directly with ChatGPT and Claude via their ch

by simonw 1y ago

I feel like the copy and paste thing is overdue a solution. I find this one particularly frustrating when working directly with ChatGPT and Claude via their chat interfaces. I frequently find myself watching them retype 100+ lines of code that I pasted in just to make a one line change. I expect there are reasons this is difficult, but difficult problems usually end up solved in the end.

rhetocj23 1y ago
Whats the time horizon for said problems to be solved? Because guess what - time is running and people will not continue to aimlessly provide money at this stuff.
- simonw 1y ago
  I don't see this one as an existential crisis for AI tooling, more of a persistent irritation. AI labs already shipped changes related to this problem - most notable speculative decoding, which lets you provide the text you expect to see come out again and speeds it up: https://simonwillison.net/2024/Nov/4/predicted-outputs/ https://simonwillison.net/2024/Nov/4/predicted-outputs/ They've also been iterating on better tools for editing code a lot as part of the competition between Claude Code and Codex CLI and other coding agents. Hopefully they'll figure out a copy/paste mechanism as part of that work.
danenania 1y ago
Yeah, I’ve always wondered if the models could be trained to output special reference tokens that just copy verbatim slices from the input, perhaps based on unique prefix/suffix pairs. Would be a dramatic improvement for all kinds of tasks (coding especially).
keeda 1y ago
> I frequently find myself watching them retype 100+ lines of code that I pasted in just to make a one line change. In such cases, I specifically instruct LLMs to "only show the lines you would change" and they are very good at doing just that and eliding the rest. However, I usually do this after going through a couple of rounds of what you just described :-) I partly do this to save time and partly to avoid using up more tokens. But I wonder if it is actually saving tokens given that hidden "thinking tokens" are a thing these days. That is, even if they do elide the unchanged code, I'm pretty sure they are "reasoning" about it before identifying only the relevant tokens to spit out. As such, that does seem different from copy-and-paste tool use, which I believe is also solved. LLMs can already identify when code changes can be made programmatically... and then do so! I have actually seen ChatGPT write Python code to refactor other Python code: https://www.linkedin.com/posts/kunalkandekar_metaprogramming-codingwithai-chatgpt-activity-7308939193638936577-g43C?utm_source=share&utm_medium=member_desktop&rcm=ACoAAADgRhUBgrrol-p4RsQqKWjmaQAW44IaF8k https://www.linkedin.com/posts/kunalkandekar_metaprogramming... I had to fix a minor bug in its Python script to make it work, but it worked and was a bit of a <head-explode> moment for me. I still wonder if this is part of its system prompt or an emergent tool-use behavior. In either case, copy-and-paste seems like a much simpler problem that could be solved with specific prompting.

by simonw 1y ago

rhetocj23 1y ago

Whats the time horizon for said problems to be solved? Because guess what - time is running and people will not continue to aimlessly provide money at this stuff.

simonw 1y ago
I don't see this one as an existential crisis for AI tooling, more of a persistent irritation. AI labs already shipped changes related to this problem - most notable speculative decoding, which lets you provide the text you expect to see come out again and speeds it up: https://simonwillison.net/2024/Nov/4/predicted-outputs/ https://simonwillison.net/2024/Nov/4/predicted-outputs/ They've also been iterating on better tools for editing code a lot as part of the competition between Claude Code and Codex CLI and other coding agents. Hopefully they'll figure out a copy/paste mechanism as part of that work.

danenania 1y ago

Yeah, I’ve always wondered if the models could be trained to output special reference tokens that just copy verbatim slices from the input, perhaps based on unique prefix/suffix pairs. Would be a dramatic improvement for all kinds of tasks (coding especially).

keeda 1y ago

> I frequently find myself watching them retype 100+ lines of code that I pasted in just to make a one line change. In such cases, I specifically instruct LLMs to "only show the lines you would change" and they are very good at doing just that and eliding the rest. However, I usually do this after going through a couple of rounds of what you just described :-) I partly do this to save time and partly to avoid using up more tokens. But I wonder if it is actually saving tokens given that hidden "thinking tokens" are a thing these days. That is, even if they do elide the unchanged code, I'm pretty sure they are "reasoning" about it before identifying only the relevant tokens to spit out. As such, that does seem different from copy-and-paste tool use, which I believe is also solved. LLMs can already identify when code changes can be made programmatically... and then do so! I have actually seen ChatGPT write Python code to refactor other Python code: https://www.linkedin.com/posts/kunalkandekar_metaprogramming-codingwithai-chatgpt-activity-7308939193638936577-g43C?utm_source=share&utm_medium=member_desktop&rcm=ACoAAADgRhUBgrrol-p4RsQqKWjmaQAW44IaF8k https://www.linkedin.com/posts/kunalkandekar_metaprogramming... I had to fix a minor bug in its Python script to make it work, but it worked and was a bit of a <head-explode> moment for me. I still wonder if this is part of its system prompt or an emergent tool-use behavior. In either case, copy-and-paste seems like a much simpler problem that could be solved with specific prompting.