3 ms·

Self-evaluation might be good enough in some domains? Then the AI is doing repeated self-evaluation, trying things out to find a response that scores higher ac

Self-evaluation might be good enough in some domains? Then the AI is doing repeated self-evaluation, trying things out to find a response that scores higher according to its self metric.

dullcrisp 2y ago
Sorry but I have to ask: what makes you think this would be a good idea?
- skirmish 2y ago
  This will just lead to the evaluatee finding anomalies in evaluator and exploiting them for maximum gains. It happened many times already where a ML model controled an object in a physical world simulator, and all it learned was to exploit simulator bugs [1] [1] https://boingboing.net/2018/11/12/local-optima-r-us.html https://boingboing.net/2018/11/12/local-optima-r-us.html
  - CooCooCaCha 2y ago
    Thats a natural tendency for optimization algorithms
Jensson 2y ago
Being able to fix your errors and improve over time until there are basically no errors is what humans do, so far all AI models just corrupt knowledge they don't purify knowledge like humanity did except when scripted with a good value function from a human like AlphaGo where the value function is winning games. This is why you need to constantly babysit todays AI and tell it to do steps and correct itself all the time, because you are much better at getting to pure knowledge than the AI is, it would quickly veer away into nonsense otherwise.
- visarga 2y ago
  > all AI models just corrupt knowledge they don't purify knowledge like humanity You got to take a step back and look at LLMs like ChatGPT. With 180 million users and assuming 10,000 tokens per user per month, that's 1.8 trillion interactive tokens. LLMs are given tasks, generate responses, and humans use those responses to achieve their goals. This process repeats over time, providing feedback to the LLM. This can scale to billions of iterations per month. The fascinating part is that LLMs encounter a vast diversity of people and tasks, receiving supporting materials, private documents, and both implicit and explicit feedback. Occasionally, they even get real-world feedback when users return to iterate on previous interactions. Taking a role of assistant LLMs are primed to learn from the outcomes of their actions, scaling across many people. Thus they can learn from our collective feedback signals over time. Yes, that uses a lot of human in the loop, not just real world in the loop, but humans are also dependent on culture and society, I see no need for AI to be able to do it without society. I actually think that AGI will be a collective/network of humans and AI agents, this perspective fits right in. AI will be the knowledge and experience flywheel of humanity.
  - seadan83 2y ago
    > This process repeats over time, providing feedback to the LLM To what extent do you know this to be true? Can you describe the mechanism that is used? I would contrast your statement with cases where chat gpt generated something, I read it and note various incorrect things and then walk away. Further, there are cases where the human does not realize there are errors. In both cases I'm not aware of any kind of feedback loop that would even be really possible - i never told the LLM it was wrong. Nor should the LLM assume it was wrong because I run more queries. Thus, there is no signal back that the answers were wrong. Hence, where do you see the feedback loop existing?
    - visarga 2y ago
      > To what extent do you know this to be true? Can you describe the mechanism that is used? Like, for example, a developer working on a project, will iterate many times, some codes generated by AI might generate errors, they will discuss that with the model to fix the code. This way the model gets not just one round interactions, but multi-round with feedback. > I read it and note various incorrect things and then walk away. I think the general pattern will be people sticking with the task longer when it fails, trying to solve it with persistence. This is all aggregated over a huge number of sessions and millions of users. In order to protect privacy we could only train preference models from this feedback data, and then fine-tune the base model without using the sensitive interaction logs directly. The model would learn a preference for how to act in specific contexts, but not remember the specifics.
- ben_w 2y ago
  Kinda, but also not entirely. One of the things OpenAI did to improve performance was to train an AI to determine how a human would rate an output, and use that to train the LLM itself. (Kinda like a GAN, now I think about it). https://forum.effectivealtruism.org/posts/5mADSy8tNwtsmT3KG/the-true-story-of-how-gpt-2-became-maximally-lewd-1 https://forum.effectivealtruism.org/posts/5mADSy8tNwtsmT3KG/... But this process has probably gone as far as it can go, at least with current architectures for the parts, as per Amdahl's law.
jgalt212 2y ago
> Self-evaluation might be good enough in some domains? This works perfectly in games. e.g. Alpha Zero. In other domains, not so much.
- coffeebeqn 2y ago
  Games are closed systems. There’s no unknowns in the rule set or world state because the game wouldn’t work if there were. No unknown unknowns. Compare to physics or biology where we have no idea if we know 1% or 90% of the rules at this point.
  - jgalt212 2y ago
    self-evaluation would still work great even where there are probabilistic and changing rule sets. The linchpin of the whole operation is automated loss function evaluation, not a set of known and deterministic rules. Once you have to pay and employ humans to compute loss functions, the scale falls apart.

Self-evaluation might be good enough in some domains? Then the AI is doing repeated self-evaluation, trying things out to find a response that scores higher according to its self metric.

dullcrisp 2y ago

Sorry but I have to ask: what makes you think this would be a good idea?

skirmish 2y ago
This will just lead to the evaluatee finding anomalies in evaluator and exploiting them for maximum gains. It happened many times already where a ML model controled an object in a physical world simulator, and all it learned was to exploit simulator bugs [1] [1] https://boingboing.net/2018/11/12/local-optima-r-us.html https://boingboing.net/2018/11/12/local-optima-r-us.html
- CooCooCaCha 2y ago
  Thats a natural tendency for optimization algorithms

Jensson 2y ago

Being able to fix your errors and improve over time until there are basically no errors is what humans do, so far all AI models just corrupt knowledge they don't purify knowledge like humanity did except when scripted with a good value function from a human like AlphaGo where the value function is winning games. This is why you need to constantly babysit todays AI and tell it to do steps and correct itself all the time, because you are much better at getting to pure knowledge than the AI is, it would quickly veer away into nonsense otherwise.

visarga 2y ago
> all AI models just corrupt knowledge they don't purify knowledge like humanity You got to take a step back and look at LLMs like ChatGPT. With 180 million users and assuming 10,000 tokens per user per month, that's 1.8 trillion interactive tokens. LLMs are given tasks, generate responses, and humans use those responses to achieve their goals. This process repeats over time, providing feedback to the LLM. This can scale to billions of iterations per month. The fascinating part is that LLMs encounter a vast diversity of people and tasks, receiving supporting materials, private documents, and both implicit and explicit feedback. Occasionally, they even get real-world feedback when users return to iterate on previous interactions. Taking a role of assistant LLMs are primed to learn from the outcomes of their actions, scaling across many people. Thus they can learn from our collective feedback signals over time. Yes, that uses a lot of human in the loop, not just real world in the loop, but humans are also dependent on culture and society, I see no need for AI to be able to do it without society. I actually think that AGI will be a collective/network of humans and AI agents, this perspective fits right in. AI will be the knowledge and experience flywheel of humanity.
- seadan83 2y ago
  > This process repeats over time, providing feedback to the LLM To what extent do you know this to be true? Can you describe the mechanism that is used? I would contrast your statement with cases where chat gpt generated something, I read it and note various incorrect things and then walk away. Further, there are cases where the human does not realize there are errors. In both cases I'm not aware of any kind of feedback loop that would even be really possible - i never told the LLM it was wrong. Nor should the LLM assume it was wrong because I run more queries. Thus, there is no signal back that the answers were wrong. Hence, where do you see the feedback loop existing?
  - visarga 2y ago
    > To what extent do you know this to be true? Can you describe the mechanism that is used? Like, for example, a developer working on a project, will iterate many times, some codes generated by AI might generate errors, they will discuss that with the model to fix the code. This way the model gets not just one round interactions, but multi-round with feedback. > I read it and note various incorrect things and then walk away. I think the general pattern will be people sticking with the task longer when it fails, trying to solve it with persistence. This is all aggregated over a huge number of sessions and millions of users. In order to protect privacy we could only train preference models from this feedback data, and then fine-tune the base model without using the sensitive interaction logs directly. The model would learn a preference for how to act in specific contexts, but not remember the specifics.
ben_w 2y ago
Kinda, but also not entirely. One of the things OpenAI did to improve performance was to train an AI to determine how a human would rate an output, and use that to train the LLM itself. (Kinda like a GAN, now I think about it). https://forum.effectivealtruism.org/posts/5mADSy8tNwtsmT3KG/the-true-story-of-how-gpt-2-became-maximally-lewd-1 https://forum.effectivealtruism.org/posts/5mADSy8tNwtsmT3KG/... But this process has probably gone as far as it can go, at least with current architectures for the parts, as per Amdahl's law.

jgalt212 2y ago

> Self-evaluation might be good enough in some domains? This works perfectly in games. e.g. Alpha Zero. In other domains, not so much.

coffeebeqn 2y ago
Games are closed systems. There’s no unknowns in the rule set or world state because the game wouldn’t work if there were. No unknown unknowns. Compare to physics or biology where we have no idea if we know 1% or 90% of the rules at this point.
- jgalt212 2y ago
  self-evaluation would still work great even where there are probabilistic and changing rule sets. The linchpin of the whole operation is automated loss function evaluation, not a set of known and deterministic rules. Once you have to pay and employ humans to compute loss functions, the scale falls apart.