I think there are two big things: 1. Attention mechanism (transformers). Really moved forward NLP. It allows long distance dependencies without the need for se

I think there are two big things: 1. Attention mechanism (transformers). Really moved forward NLP. It allows long distance dependencies without the need for sequential processing (like in RNNs). Generally scales much better (see BERT/GPT-2/XL-Net/etc) 2. Separable convolutions. Significantly sped-up the standard CV networks. (mobile net, but now everything, really).