> from a longer term perspective Yes. Longer term, new hardware will also make it practical to train large models in a fully parallelized, fully distributed m

> from a longer term perspective Yes. Longer term, new hardware will also make it practical to train large models in a fully parallelized, fully distributed manner -- i.e., without having to backpropagate gradients, which requires a lot of complex bookkeeping and plumbing for distributed training. Recent progress suggests this will happen. See, for example: https://arxiv.org/abs/2006.04182 https://arxiv.org/abs/2006.04182 https://arxiv.org/abs/2103.03725 https://arxiv.org/abs/2103.03725 https://arxiv.org/abs/2010.01047 https://arxiv.org/abs/2010.01047 I for one am excited to see what happens over the next decade as it becomes trivial to train/use models with 1K, 1M, or 1B times more dense connections than present state-of-the-art models.