Per the paper, 37.5% of the regularization regimes were Batch Norm + Weight Decay. According to other findings, Batch Norm and Weight decay have an interesting

Per the paper, 37.5% of the regularization regimes were Batch Norm + Weight Decay. According to other findings, Batch Norm and Weight decay have an interesting interaction that results in a learning rate adjustment, not regularization: https://blog.janestreet.com/l2-regularization-and-batch-norm/ https://blog.janestreet.com/l2-regularization-and-batch-norm...