4 ms·

I assume RL stands for reinforcement learning, which just means that the weights are adjusted by evaluating some loss function. If the loss function uses the tr

I assume RL stands for reinforcement learning, which just means that the weights are adjusted by evaluating some loss function. If the loss function uses the training data (i.e. supervised learning) then the original assumption still holds. So I fail to see how this is a misconception.

atherton94027 3mo ago
I think OP might be referring to synthetic data, which is now used to provide more data for training?
- runarberg 3mo ago
  That would makes sense. This is how AlphaGo became better then the training data, and was the basis for Alpha Zero which used no human data and became even better. However the loss function (or the success criteria more broadly) for the game of go is extremely simple. We do not know whether such a success criteria even exists for a generic task like coding, and it is a mistake to assume that the AI labs have found one.
tulio_ribeiro 3mo ago
Supervised learning and reinforcement learning are mathematically distinct. Supervised pre-training as you said maximizes the likelihood of a token sequence. RL optimizes a policy against an external reward signal or execution environment. For example, RL evaluates code against runtime execution/unit tests/proof checkers (e.g lean). Models learn new strategies and are able to produce novel code if trained in an RL environment.
- runarberg 3mo ago
  I don‘t understand what you mean. Supervised learning is reinforcement learning, while not all reinforcement learning is supervised learning (e.g unsupervised learning is also reinforcement learning; i.e. reinforcement learning with unlabelled data). Like you said, you can have reinforcement learning which doesn’t use training data. But that is not what my parent said. What they said is: since RL is used heavily in the training. And since reinforcement learning is a broad category which includes supervised learning, nothing in their logic disproves the strawman they created from an AI skeptic.
  - xdavidliu 3mo ago
    that's not how the terminology is typical used, even if it makes literal sense

atherton94027 3mo ago

I think OP might be referring to synthetic data, which is now used to provide more data for training?

runarberg 3mo ago
That would makes sense. This is how AlphaGo became better then the training data, and was the basis for Alpha Zero which used no human data and became even better. However the loss function (or the success criteria more broadly) for the game of go is extremely simple. We do not know whether such a success criteria even exists for a generic task like coding, and it is a mistake to assume that the AI labs have found one.

tulio_ribeiro 3mo ago

Supervised learning and reinforcement learning are mathematically distinct. Supervised pre-training as you said maximizes the likelihood of a token sequence. RL optimizes a policy against an external reward signal or execution environment. For example, RL evaluates code against runtime execution/unit tests/proof checkers (e.g lean). Models learn new strategies and are able to produce novel code if trained in an RL environment.

runarberg 3mo ago
I don‘t understand what you mean. Supervised learning is reinforcement learning, while not all reinforcement learning is supervised learning (e.g unsupervised learning is also reinforcement learning; i.e. reinforcement learning with unlabelled data). Like you said, you can have reinforcement learning which doesn’t use training data. But that is not what my parent said. What they said is: since RL is used heavily in the training. And since reinforcement learning is a broad category which includes supervised learning, nothing in their logic disproves the strawman they created from an AI skeptic.
- xdavidliu 3mo ago
  that's not how the terminology is typical used, even if it makes literal sense