4 ms·
>Likely, there is a method that can use the orthogonalization machinery of Muon while keeping the signal-to-noise estimation of Adam, and this optimizer will be
by big-chungus4 11mo ago
>Likely, there is a method that can use the orthogonalization machinery of Muon while keeping the signal-to-noise estimation of Adam, and this optimizer will be great.
if you take SOAP and change all betas to 0, it still works well, so SOAP is that already