These LLMs with cosine learning rate are usually decayed just to 1/10th the peak LR. Even if you used exponential decay, your LR on your final step could still

These LLMs with cosine learning rate are usually decayed just to 1/10th the peak LR. Even if you used exponential decay, your LR on your final step could still be arbitrarily large depending on how fast you configure it to decay.