> That's very ignorant to say that attention or convolution is anywhere close to the expressivity of RNNs(LSTM). But that's exactly the point of the Transforme

> That's very ignorant to say that attention or convolution is anywhere close to the expressivity of RNNs(LSTM). But that's exactly the point of the Transformer model, with a paper aptly titled "Attention is all you need" [1]. And the Bert architecture, based in this idea, seems to be doing well. And they claim to be bery flexible, too[2]. Maybe that's what you meant with "unless you brutely search over hundreds of hyperparameters configs", but then again, isn't that what NNs are about anyway? [1] https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1706.03762 [2] https://arxiv.org/abs/1810.04805 https://arxiv.org/abs/1810.04805