3 ms·
There are multiple ways to model reward. You can have it be fine-grained, such that every token gets its own reward, but by far the most common is to feed in th
by maleldil 2y ago
There are multiple ways to model reward. You can have it be fine-grained, such that every token gets its own reward, but by far the most common is to feed in the whole sequence and generate a single reward at the end.
- 1024core 2y agoI guess I'm not sure how the "feed in the whole sequence" works, if there's a single reward at the end.
- maleldil 2y agoIt depends on the model and the problem. As an example, BERT-based models have a special [CLS] token that was pre-trained to encode information about the whole sequence. A reward model based on BERT would take the output embedding of that token from the last layer and feed it through a classification head, which would depend on your problem. You could then train this classification head on your alignment dataset like a classification problem. You can check the examples from the TRL library for more information.
- 1024core 2y ago> You can check the examples from the TRL library for more information. What library is that? Thanks!