I'm not too familiar with SHARP but I've implemented SophiaH before, it is effectively mSGD but with the momentum being scaled by inverse of a separate hessian

I'm not too familiar with SHARP but I've implemented SophiaH before, it is effectively mSGD but with the momentum being scaled by inverse of a separate hessian moment with some additional clipping. It works surprisingly well if you can afford the additional memory/compute. It seems like SHARP directly incorporates the hessian directly into the SGD momentum, and increases the 'alpha' or momentum contribution over time.