4 ms·

I've been curious about using LLMs for large-scale refactoring. Prompts like anywhere you find `FooBarBaz(blip, kap)` replace it with `new newThing(blip).bump(

I've been curious about using LLMs for large-scale refactoring. Prompts like anywhere you find `FooBarBaz(blip, kap)` replace it with `new newThing(blip).bump(kap)` I don't know how reliable it is, but it seems like if you can easily run this on commodity hardware it could totally replace most IDE refactoring tools, although obviously the IDE refactoring is more reliable, it seems like this could be made simple and flexible, and possibly just as reliable as IDEs. But also it could enable some interesting things that you could never do with an IDE refactoring tool.

potatoman22 3y ago
LLMs aren't very good at string operations. They struggle with things like character counts, extracting substrings, and replacements.
- simonw 3y ago
  Yeah, for this kind of thing I would get the LLM to generate the regular expression.
Ductapemaster 3y ago
I think that an LLM could be an amazing interface or translation layer for this sort of thing, but I would argue that the underlying operations of refactoring or something similar should remain very much like a function with discrete inputs and outputs.
sharkjacobs 3y ago
I already do things like "write a script to replace `FooBarBaz(blip, kap)` with `new newThing(blip).bump(kap)` in a project folder" I'm more comfortable with that because I find it usually takes two or three prompts to get it right e.g. A couple hours ago I prompted it to help me do a diff of two commits ignoring all white space, just to check if there were any other changes. The first response didn't ignore new newlines, the second one was a multiline script, the third response gave me what I actually wanted. diff -w <(git show 0bb2c8579efe775de883e0182db48989bfa324f2:"path/to/file"|tr -d '\n') <(git show 6c71efc17497ad7c90b9c7b690075ec031c13c69:"path/to/file"|tr -d '\n')

4 ms·

I've been curious about using LLMs for large-scale refactoring. Prompts like anywhere you find `FooBarBaz(blip, kap)` replace it with `new newThing(blip).bump(

by lukeschlather 3y ago

potatoman22 3y ago
LLMs aren't very good at string operations. They struggle with things like character counts, extracting substrings, and replacements.
- simonw 3y ago
  Yeah, for this kind of thing I would get the LLM to generate the regular expression.
Ductapemaster 3y ago
I think that an LLM could be an amazing interface or translation layer for this sort of thing, but I would argue that the underlying operations of refactoring or something similar should remain very much like a function with discrete inputs and outputs.
sharkjacobs 3y ago
I already do things like "write a script to replace `FooBarBaz(blip, kap)` with `new newThing(blip).bump(kap)` in a project folder" I'm more comfortable with that because I find it usually takes two or three prompts to get it right e.g. A couple hours ago I prompted it to help me do a diff of two commits ignoring all white space, just to check if there were any other changes. The first response didn't ignore new newlines, the second one was a multiline script, the third response gave me what I actually wanted. diff -w <(git show 0bb2c8579efe775de883e0182db48989bfa324f2:"path/to/file"|tr -d '\n') <(git show 6c71efc17497ad7c90b9c7b690075ec031c13c69:"path/to/file"|tr -d '\n')