> I used the AdamW optimizer and selected a learning rate of 5e-5. I’ve seen learning rates of 5e-6 for pretraining and 5e-5 for finetuning. I would consider th

> I used the AdamW optimizer and selected a learning rate of 5e-5. I’ve seen learning rates of 5e-6 for pretraining and 5e-5 for finetuning. I would consider this closer to the latter - I don’t want to totally destroy the knowledge Qwen already had, I just want to add to it a bit. Is this a typo? Maybe 5e-4 for pretraining? Otherwise this goes against all the intuition I have around learning rates and catastrophic forgetting. (a smaller learning rate causing knowledge degredation)