> So any N-layer network with the relu _/ activation function The paper talks only about models with additive operations among activations. It doesn't say anyt

> So any N-layer network with the relu _/ activation function The paper talks only about models with additive operations among activations. It doesn't say anything about more complex networks like transformers. In transformers there are multiplicative interactions between activations inside the attention matrix, it is unclear if they can be approximated with just a 3 layer ReLU network, or if such conversion would be practical at all.