I'll throw in a seventh... Neural architecture search is finding more efficient architectural building blocks which translate to better quality models with few

I'll throw in a seventh... Neural architecture search is finding more efficient architectural building blocks which translate to better quality models with fewer overall parameters. There's a lot of work going on right now to improve transformer architectures in particular; I think once the dust settles they will be massively more efficient than the current models. Efficient transformers often also introduce an inductive bias, which ends up improving model quality. (similar to replacing fully connected networks with convolutional networks; more efficient, but also carrying an inductive bias that improves model quality.)