Optimization algorithm they arrived at was (quoting from the paper) def train(weight, gradient, momentum, lr): update = interp(gradient, momentum, β1)

Optimization algorithm they arrived at was (quoting from the paper) def train(weight, gradient, momentum, lr): update = interp(gradient, momentum, β1) update = sign(update) momentum = interp(gradient, momentum, β2) weight_decay = weight * λ update = update + weight_decay update = update * lr return update, momentum The key thing they do, sending the update through sign, means all weight updates turn into either 1 or -1.