3 ms·
Has it been publicly benchmarked yet, if this approach: Hello LLM, please solve this task: <task> Can be improved by performing this afterwards? for
by mg 2y ago
Has it been publicly benchmarked yet, if this approach:
Hello LLM, please solve this task: <task>
Can be improved by performing this afterwards?
for iteration in range(10):
Hello LLM, please solve this task: <task>
Here is a possible solution: <last_reply>
Please look at it and see if you can improve it.
Then tell me your improved solution.
- Kiro 2y agoIsn't that the whole reason that o1 works?
- ben_w 2y agoI think o1 is more like "pretend you're doing a job interview, think step and show your working". I tried something similar to the suggested iterative loop on a blog post I'd authored but wanted help copy editing; first few were good enough, but then it got very confused and decided the blog post wasn't actually a blog post to be edited and instead that what I really wanted to know was the implications of Florida something something Republican Party. Benchmark would be neat, because all I have is an anecdote.
- bachback 2y agofor coding tasks see https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/ the question is how would you define "improve" and "solve". RLHF in a way delegates this to humans.
- eykrehbein 2y agoBruteforceLLM
- lorepieri 2y agoNot sure if it has been benchmarked, but I've called this technique the "for-loop of thought". :)