It is important to remember that the optimizers used in mainstream deep learning models have all been developed and fine tuned to work well with classic NN arch

It is important to remember that the optimizers used in mainstream deep learning models have all been developed and fine tuned to work well with classic NN architectures. There is no such thing as a generic optimization algorithm.