Reminds of some earlier openai work https://openai.com/research/nonlinear-computation-in-deep-linear-networks https://openai.com/research/nonlinear-computation-

Reminds of some earlier openai work https://openai.com/research/nonlinear-computation-in-deep-linear-networks https://openai.com/research/nonlinear-computation-in-deep-li... "Neural networks consist of stacks of a linear layer followed by a nonlinearity like tanh or rectified linear unit. Without the nonlinearity, consecutive linear layers would be in theory mathematically equivalent to a single linear layer. So it’s a surprise that floating point arithmetic is nonlinear enough to yield trainable deep networks."