3 ms·

<"I'll add it to my list of optimizers to try for new deep learning projects. " Can you share the list of your go to optimizers outside of the Adam family?

sdenton4 3y ago
I think there's Adam and "Nothing is obviously substantially better than Adam so why bother?"
- neggles 3y ago
  I've had a lot of luck with CAME https://arxiv.org/abs/2307.02047 https://arxiv.org/abs/2307.02047
johndough 3y ago
> Can you share the list of your go to optimizers outside of the Adam family? Sure! It depends a bit on what I'm doing. If I want to optimize someone else's model, I start with Adam, because that's most-likely what the hyperparameters have been optimized for. Once I've verified that Adam works, I'll try other optimizers. If I have very few parameters and don't care about overfitting, I try LBFGS, which usually gets to the local optimum the fastest. Note that this will likely find a sharp local optimum. For better generalization performance, you often prefer a wide optimum, so the model still works if there is a bit of drift in the data. If I do not want to mess around with learning rates, I use Adafactor, which is a bit slower, but usually works okay without any tuning. If I had very little memory available, I'd use SGD, but in my opinion it's not worth the hassle of tuning learning rate, momentum, dampening and weight decay. I'd rather use a smaller model if possible. I usually do not train with extremely large batch sizes, but if I did, I'd try the optimizers which claim to work well for large batch sizes. All in all, it probably does not matter too much which optimizer you are using, as long as you tuned it a little bit. Same goes for the model, loss functions, activation functions and all that other fluff. What /is/ important is that you design your problem in such a way that it is as easy as possible to solve. For example, it is very difficult to read arbitrary hand-written text from an image. If you have control over where the data comes from, it would be better to write the text character by character into a printed grid with additional optical markers for image registration. Or even better, replace it with a multiple choice list. If there are not too many exceptional cases, an "other" option for manual review could be added. Often, automating 99 % of the work is more than good enough and it is better to keep a human in the loop to handle edge cases. Secondly, control the data capture as strictly as possible. For example, use uniform lightning, place the object to recognize at exactly the same position, exclude disruptive elements, etc. Lastly, data is king. If your training data does not match the test data, you can train all you want and still get garbage results. Either collect enough training data to cover all test cases or, if that is not possible from the start, retrain with new data regularly. Data augmentation might help to some degree, but it is impossible to predict everything.

sdenton4 3y ago

I think there's Adam and "Nothing is obviously substantially better than Adam so why bother?"

neggles 3y ago
I've had a lot of luck with CAME https://arxiv.org/abs/2307.02047 https://arxiv.org/abs/2307.02047

johndough 3y ago

> Can you share the list of your go to optimizers outside of the Adam family? Sure! It depends a bit on what I'm doing. If I want to optimize someone else's model, I start with Adam, because that's most-likely what the hyperparameters have been optimized for. Once I've verified that Adam works, I'll try other optimizers. If I have very few parameters and don't care about overfitting, I try LBFGS, which usually gets to the local optimum the fastest. Note that this will likely find a sharp local optimum. For better generalization performance, you often prefer a wide optimum, so the model still works if there is a bit of drift in the data. If I do not want to mess around with learning rates, I use Adafactor, which is a bit slower, but usually works okay without any tuning. If I had very little memory available, I'd use SGD, but in my opinion it's not worth the hassle of tuning learning rate, momentum, dampening and weight decay. I'd rather use a smaller model if possible. I usually do not train with extremely large batch sizes, but if I did, I'd try the optimizers which claim to work well for large batch sizes. All in all, it probably does not matter too much which optimizer you are using, as long as you tuned it a little bit. Same goes for the model, loss functions, activation functions and all that other fluff. What /is/ important is that you design your problem in such a way that it is as easy as possible to solve. For example, it is very difficult to read arbitrary hand-written text from an image. If you have control over where the data comes from, it would be better to write the text character by character into a printed grid with additional optical markers for image registration. Or even better, replace it with a multiple choice list. If there are not too many exceptional cases, an "other" option for manual review could be added. Often, automating 99 % of the work is more than good enough and it is better to keep a human in the loop to handle edge cases. Secondly, control the data capture as strictly as possible. For example, use uniform lightning, place the object to recognize at exactly the same position, exclude disruptive elements, etc. Lastly, data is king. If your training data does not match the test data, you can train all you want and still get garbage results. Either collect enough training data to cover all test cases or, if that is not possible from the start, retrain with new data regularly. Data augmentation might help to some degree, but it is impossible to predict everything.