A lot of adaptive GD algos work by kind of approximating the inverse Hessian, which is usually too expensive to solve at every step. Adagrad (my go-to) essentia

A lot of adaptive GD algos work by kind of approximating the inverse Hessian, which is usually too expensive to solve at every step. Adagrad (my go-to) essentially finds a diagonal approximation to the Hessian, which often works well in practice. There's fancier techniques to take low-rank approximations and that sort of thing, but I feel like they're usually more trouble than their worth.