6 ms·
from my understanding that is what they do, see the paper: > We use a pre-trained GPT-2 (Radford et al., 2019) as the base model for all experiments. I agree t
by fabmilo 2y ago
from my understanding that is what they do, see the paper:
> We use a pre-trained GPT-2 (Radford et al., 2019) as the base model for all experiments.
I agree the feedback is necessary, and the mechanism simple and cheap, but I don't think is optimal.
- zxexz 2y agoYes, they use a pre-trained model, but they do further training (please correct me if I mis-read, and also I realize my above comment could be interpreted as saying they train a new model entirely from scratch). > We use a pre-trained GPT-2 (Radford et al., 2019) as the base model for all experiments. The learning rate is set to 1 × 10−4 while the effective batch size is 128. Following Deng et al. (2024), we also reset the optimizer when the training stages switch.