> what’s wrong with us as a species We ask for faster horses, when automobiles would get us where we want to go. It is a lack of imagination for what is possib

> what’s wrong with us as a species We ask for faster horses, when automobiles would get us where we want to go. It is a lack of imagination for what is possible in the future that keeps us risk-averse and focused instead on a quest for incremental improvements to the status quo. This was the focus of the idealistic vision of the whole "disruption" meta-meme as it spread through Silicon Valley. Not to say that maintenance and refinement aren't important, but innovation doesn't look like just adding more parameters and ways to scale massive computations. The main concept that AI laypeople (even those who burn OpenAI credits on a hobbyist basis) are missing to really grasp the possibilities of these kinds of statistical analysis techniques is a deep understanding of the notion of inductive bias. For instance, Transformers are so powerful precisely because they have generalized that idea of inductive bias one level up by using multi-head attention to project onto many different linear spaces and attend to them differently in different contexts, rather than just filtering through effectively one linear perspective at a time as with your traditional dense NNs. The kinds of innovations we have witnessed most recently are actually pretty small steps in the space of possible architectures, not to mention training methods, etc. For instance, predictive coding approximates gradient descent[1] which opens up all sorts of new architectures (feedback arch, modular/federated/local compositions, etc.) that are intractable with traditional backprop-based techniques unless you can manage the infrastructure around periodic global parameter/state syncs. [1] https://openreview.net/pdf?id=PdauS7wZBfC https://openreview.net/pdf?id=PdauS7wZBfC