I think you would be hard-pressed to find any experts who, even prior to 2017, hadn't settled on the mental model of neural networks as lossy compression machin

I think you would be hard-pressed to find any experts who, even prior to 2017, hadn't settled on the mental model of neural networks as lossy compression machines. Back when enthusiasts and researchers read mostly textbooks, papers, and Wikipedia (rather than blog posts, tweets, and READMEs) there was much more discussion around the 'InfoMax Criterion' -- quite elegantly demonstrated by Tishby et al. via his closely-related 'Information Bottleneck' studies -- which is just that idea: mutual information is maximized between input and output, subject to inherent limits of statistical processing of the realizations of such systems. What determines the asymptotic maximal value of the mutual information is the inductive bias of the system vis a vis the training set. This is all standard theory, perhaps so fundamental that it is obscured by all the application-oriented study and instruction.