4 ms·
Thank you so much for your comprehensive answer, this helps a lot. If I understand nshepperd's code correctly, it uses a constant and small learning rate. Do y
by leod 7y ago
Thank you so much for your comprehensive answer, this helps a lot.
If I understand nshepperd's code correctly, it uses a constant and small learning rate. Do you know if this works better than the learning rate schedule that is usually used for Transformer models (https://www.tensorflow.org/alpha/tutorials/text/transformer_files/output_80_1.png https://www.tensorflow.org/alpha/tutorials/text/transformer_...)?
- gwern 7y agoIt's a constant, yes. We haven't tried any other learning rate schedules (for my poetry GPT-2s, I simply drop the LR 10x each day or so). I have no idea if this is optimal for transfer learning or not.