3 ms·

> This process repeats over time, providing feedback to the LLM To what extent do you know this to be true? Can you describe the mechanism that is used? I wou

> This process repeats over time, providing feedback to the LLM To what extent do you know this to be true? Can you describe the mechanism that is used? I would contrast your statement with cases where chat gpt generated something, I read it and note various incorrect things and then walk away. Further, there are cases where the human does not realize there are errors. In both cases I'm not aware of any kind of feedback loop that would even be really possible - i never told the LLM it was wrong. Nor should the LLM assume it was wrong because I run more queries. Thus, there is no signal back that the answers were wrong. Hence, where do you see the feedback loop existing?

visarga 2y ago
> To what extent do you know this to be true? Can you describe the mechanism that is used? Like, for example, a developer working on a project, will iterate many times, some codes generated by AI might generate errors, they will discuss that with the model to fix the code. This way the model gets not just one round interactions, but multi-round with feedback. > I read it and note various incorrect things and then walk away. I think the general pattern will be people sticking with the task longer when it fails, trying to solve it with persistence. This is all aggregated over a huge number of sessions and millions of users. In order to protect privacy we could only train preference models from this feedback data, and then fine-tune the base model without using the sensitive interaction logs directly. The model would learn a preference for how to act in specific contexts, but not remember the specifics.
- seadan83 2y ago
  > I think the general pattern will be people sticking with the task longer when it fails, trying to solve it with persistence. This is where I quibble. Accurately detecting someone is actually still on the same task (indicating they are not satisfied) is perhaps as challenging as generating any answer to begin with. That is why I also mentioned when people don't know the result was incorrect. That'll potentially drive a strong "this answer was correct" signal. So, during development of a tool, I can envision that feedback loop. But something simply presented to millions, without a way to determine false negatives nor false positives-- how exactly does that feedback loop work? edit We might be talking past each other. I did not quite read that "discuss with AI" part carefully. I was picturing something like copilot or chat got, where it is pretty much: 'here is your answer's Even with an interactive AI, how to account for positive negatives (human accepts wrong answer), or when human simply gives up (which also looks like a success). If we told AI evertime it was wrong or right - can that scale to the extent it would actually train a model?
- HarHarVeryFunny 2y ago
  I think AGI is going to have to learn itself on-the-job, same as humans. Trying to collect human traces of on-the-job training/experience and pre-training an AGI on them seems doomed to failure since the human trace is grounded in the current state (of knowledge/experience) of the human's mind, but what the AGI needs is updates relative to it's own state. In any case, for a job like (human-level) software developer AGI is going to need human-level runtime reasoning and learning (>> in-context learn by example), even if it were possible to "book pre-train" it rather than on-the-job train it. Outside of repetitive genres like CRUD-apps, most software projects are significantly unique, even if they re-use learnt developer skills - it's like Chollet's ARC test on mega-steroids, with dozens/hundreds of partial-solution design techniques, and solutions that require a hierarchy of dozens/hundreds of these partial-solutions (cf Chollet core skills, applied to software) to be arranged into a solution in a process of iterative refinement. There's a reason senior software developers are paid a lot - it's not just economically valuable, it's also one of the more challenging cognitive skills that humans are capable of.

visarga 2y ago

> To what extent do you know this to be true? Can you describe the mechanism that is used? Like, for example, a developer working on a project, will iterate many times, some codes generated by AI might generate errors, they will discuss that with the model to fix the code. This way the model gets not just one round interactions, but multi-round with feedback. > I read it and note various incorrect things and then walk away. I think the general pattern will be people sticking with the task longer when it fails, trying to solve it with persistence. This is all aggregated over a huge number of sessions and millions of users. In order to protect privacy we could only train preference models from this feedback data, and then fine-tune the base model without using the sensitive interaction logs directly. The model would learn a preference for how to act in specific contexts, but not remember the specifics.

seadan83 2y ago
> I think the general pattern will be people sticking with the task longer when it fails, trying to solve it with persistence. This is where I quibble. Accurately detecting someone is actually still on the same task (indicating they are not satisfied) is perhaps as challenging as generating any answer to begin with. That is why I also mentioned when people don't know the result was incorrect. That'll potentially drive a strong "this answer was correct" signal. So, during development of a tool, I can envision that feedback loop. But something simply presented to millions, without a way to determine false negatives nor false positives-- how exactly does that feedback loop work? edit We might be talking past each other. I did not quite read that "discuss with AI" part carefully. I was picturing something like copilot or chat got, where it is pretty much: 'here is your answer's Even with an interactive AI, how to account for positive negatives (human accepts wrong answer), or when human simply gives up (which also looks like a success). If we told AI evertime it was wrong or right - can that scale to the extent it would actually train a model?
HarHarVeryFunny 2y ago
I think AGI is going to have to learn itself on-the-job, same as humans. Trying to collect human traces of on-the-job training/experience and pre-training an AGI on them seems doomed to failure since the human trace is grounded in the current state (of knowledge/experience) of the human's mind, but what the AGI needs is updates relative to it's own state. In any case, for a job like (human-level) software developer AGI is going to need human-level runtime reasoning and learning (>> in-context learn by example), even if it were possible to "book pre-train" it rather than on-the-job train it. Outside of repetitive genres like CRUD-apps, most software projects are significantly unique, even if they re-use learnt developer skills - it's like Chollet's ARC test on mega-steroids, with dozens/hundreds of partial-solution design techniques, and solutions that require a hierarchy of dozens/hundreds of these partial-solutions (cf Chollet core skills, applied to software) to be arranged into a solution in a process of iterative refinement. There's a reason senior software developers are paid a lot - it's not just economically valuable, it's also one of the more challenging cognitive skills that humans are capable of.