3 ms·

At work i only had early copilot which was hysterically bad at everything. As i wanted it to do the same task repeatedly and could spot wrong results instantly

At work i only had early copilot which was hysterically bad at everything. As i wanted it to do the same task repeatedly and could spot wrong results instantly i kept evolving a prompt that attempted to correct all ways it found to do it wrong. It kept inventing new ways to get it wrong until it eventually got it right 90% of the time. My theory is that an avanced model that has no issues with a task could do the prompt enginering much better than i ever could. You could for example run x different queries that all do the same thing and compare the results y times. If there are >1 correct results and the wrong versions are all unique you should be able to drill down to a valid result with even a truly shit model running on a potato. Basically what humans do.

miki123211 1mo ago
Yes! As long as you have some criteria to judge the final answer, you can do a kind of "prompt-side RLVR", where you have the model generate prompt changes, try a bunch of different prompts and see which ones improve the results. You don't necessarily need a bigger model to do this.
smartbit 1mo ago
Trying until you find something that doesn’t fail is the model used throughout evolution & engineering, think of bridges collapsed, airplanes falling from the sky or exploded steam locomotives. In engineering you learn from these mistakes and try never making them again. Do we want to go through this evolution every time we solve a software issue? Just because we can with an unlimited number of cheap tokens? I think not, I’d rather use the knowledge build up that also knows about the edge cases forgotten to test. Or better, use multiple models that evaluate each other, as Entropic describes it in their recent report https://news.ycombinator.com/item?id=49316271 https://news.ycombinator.com/item?id=49316271 > We expect that agents coordinating in the wild will act in higher variance ways than we see here, because they’ll have different backgrounds and therefore different contexts. They also, presumably, won’t all be Claudes.
59nadir 1mo ago
Early Copilot was tab-complete in editors and was honestly the best version of LLM-assisted development I've used, because it was intentionally small in scope, trivial to verify output from at a glance and easy to opt in and opt out of. I have yet to see anything more useful in terms of code generation; very small-scale code generation (think function-level) comes close, but is a lot more tedious. For analysis, bug hunting, overview and some porting work to popular languages and so on I think the current SotA is fantastic, but they're still very disappointing for code generation past function or small module level.
- jodrellblank 1mo ago
  > "I have yet to see anything more useful in terms of code generation" Recently my employer hooked Claude into a bunch of SaaS services like a ticketing system and an asset inventory system, and I feel a big productivity boost is that I don't have to dig for their open browser tabs, remember their product name/URL/where I put the bookmark, find that my sessions have logged me out "for my protection" since I last looked, find my MFA code, approve sign-in on my phone, then use their mismashed/sluggish/poor-UX interfaces and slow searches to loko for things, and can instead ask Claude "search these systems for anything about <issue>" and it just does. It correlates between systems, summarizes things, and gives me references which system and where to look to check what it found. Internal search on a mess of acquired and disjoint systems. This feeling reminds me of two classic Joel Spolsky blog posts below, and your comment here about the surprisingly useful tab-complete gives me a similar feeling. I wonder how many smaller candidates there are where LLMs are more helpful and less hyped, against the big vibe-coding solve-everything hype which are currently less helpful than their hype? References [1]: "A lot of us thought in the 1990s that the big battle would be between procedural and object oriented programming, and we thought that object oriented programming would provide a big boost in programmer productivity. I thought that, too. Some people still think that. It turns out we were wrong. Object oriented programming is handy dandy, but it’s not really the productivity booster that was promised. The real significant productivity advance we’ve had in programming has been from languages which manage memory for you automatically. [...] Whenever you hear someone bragging about how productive their language is, they’re probably getting most of that productivity from the automated memory management, even if they misattribute it. Sidebar: Why does automatic memory management make you so much more productive? 1) Because you can write f(g(x)) without worrying about how to free the return value from g, which means you can use functions which return interesting complex data types and functions which transform interesting complex data types, in turn allowing you to work at a higher level of abstraction" and [2]: "I have seen many language and programming fads come and go. But there’s only ONE, that’s right, ONE language feature I’ve ever seen that actually improves your productivity significantly. No, it’s not object oriented programming; no, it’s not intentional programming or assertions or programming by example or CASE or UML or XML or Java. The only thing that improves your programming productivity is using managed code – that is, using a language in which memory management is automatic." [1] https://www.joelonsoftware.com/2004/06/13/how-microsoft-lost-the-api-war/ https://www.joelonsoftware.com/2004/06/13/how-microsoft-lost... [2] https://www.joelonsoftware.com/2001/10/17/working-on-citydesk-part-three/ https://www.joelonsoftware.com/2001/10/17/working-on-citydes...
parasti 1mo ago
I've done this. Ask Claude to permutate a prompt and run Claude Code or a subagent to observe effects until success. Claude will iterate on the prompt faster than I could and likely catch more edge cases, too. It's one of those tasks where the end goal is very clear and the agent just needs to iterate on permutations. Arguably this is a perfect match, too, because prompt engineering is really just text generation, so why not have a text generation machine do it.
- killix 1mo ago
  [flagged]

miki123211 1mo ago

Yes! As long as you have some criteria to judge the final answer, you can do a kind of "prompt-side RLVR", where you have the model generate prompt changes, try a bunch of different prompts and see which ones improve the results. You don't necessarily need a bigger model to do this.

smartbit 1mo ago

Trying until you find something that doesn’t fail is the model used throughout evolution & engineering, think of bridges collapsed, airplanes falling from the sky or exploded steam locomotives. In engineering you learn from these mistakes and try never making them again. Do we want to go through this evolution every time we solve a software issue? Just because we can with an unlimited number of cheap tokens? I think not, I’d rather use the knowledge build up that also knows about the edge cases forgotten to test. Or better, use multiple models that evaluate each other, as Entropic describes it in their recent report https://news.ycombinator.com/item?id=49316271 https://news.ycombinator.com/item?id=49316271 > We expect that agents coordinating in the wild will act in higher variance ways than we see here, because they’ll have different backgrounds and therefore different contexts. They also, presumably, won’t all be Claudes.

59nadir 1mo ago

Early Copilot was tab-complete in editors and was honestly the best version of LLM-assisted development I've used, because it was intentionally small in scope, trivial to verify output from at a glance and easy to opt in and opt out of. I have yet to see anything more useful in terms of code generation; very small-scale code generation (think function-level) comes close, but is a lot more tedious. For analysis, bug hunting, overview and some porting work to popular languages and so on I think the current SotA is fantastic, but they're still very disappointing for code generation past function or small module level.

jodrellblank 1mo ago
> "I have yet to see anything more useful in terms of code generation" Recently my employer hooked Claude into a bunch of SaaS services like a ticketing system and an asset inventory system, and I feel a big productivity boost is that I don't have to dig for their open browser tabs, remember their product name/URL/where I put the bookmark, find that my sessions have logged me out "for my protection" since I last looked, find my MFA code, approve sign-in on my phone, then use their mismashed/sluggish/poor-UX interfaces and slow searches to loko for things, and can instead ask Claude "search these systems for anything about <issue>" and it just does. It correlates between systems, summarizes things, and gives me references which system and where to look to check what it found. Internal search on a mess of acquired and disjoint systems. This feeling reminds me of two classic Joel Spolsky blog posts below, and your comment here about the surprisingly useful tab-complete gives me a similar feeling. I wonder how many smaller candidates there are where LLMs are more helpful and less hyped, against the big vibe-coding solve-everything hype which are currently less helpful than their hype? References [1]: "A lot of us thought in the 1990s that the big battle would be between procedural and object oriented programming, and we thought that object oriented programming would provide a big boost in programmer productivity. I thought that, too. Some people still think that. It turns out we were wrong. Object oriented programming is handy dandy, but it’s not really the productivity booster that was promised. The real significant productivity advance we’ve had in programming has been from languages which manage memory for you automatically. [...] Whenever you hear someone bragging about how productive their language is, they’re probably getting most of that productivity from the automated memory management, even if they misattribute it. Sidebar: Why does automatic memory management make you so much more productive? 1) Because you can write f(g(x)) without worrying about how to free the return value from g, which means you can use functions which return interesting complex data types and functions which transform interesting complex data types, in turn allowing you to work at a higher level of abstraction" and [2]: "I have seen many language and programming fads come and go. But there’s only ONE, that’s right, ONE language feature I’ve ever seen that actually improves your productivity significantly. No, it’s not object oriented programming; no, it’s not intentional programming or assertions or programming by example or CASE or UML or XML or Java. The only thing that improves your programming productivity is using managed code – that is, using a language in which memory management is automatic." [1] https://www.joelonsoftware.com/2004/06/13/how-microsoft-lost-the-api-war/ https://www.joelonsoftware.com/2004/06/13/how-microsoft-lost... [2] https://www.joelonsoftware.com/2001/10/17/working-on-citydesk-part-three/ https://www.joelonsoftware.com/2001/10/17/working-on-citydes...

parasti 1mo ago

I've done this. Ask Claude to permutate a prompt and run Claude Code or a subagent to observe effects until success. Claude will iterate on the prompt faster than I could and likely catch more edge cases, too. It's one of those tasks where the end goal is very clear and the agent just needs to iterate on permutations. Arguably this is a perfect match, too, because prompt engineering is really just text generation, so why not have a text generation machine do it.

killix 1mo ago
[flagged]