Interesting that we see the same things - everything you say is true - but see different implications. The long history of gradient descent is even funnier tha

Interesting that we see the same things - everything you say is true - but see different implications. The long history of gradient descent is even funnier than it seems now. A defining problem for neural networks, that famously put the early field into a winter, was the inability to find a way to train hidden layers. In 1969, Marvin Minsky and Seymour Papert released a book that demonstrated the limitations of single layers of neurons. They can perform linear separation, and little else, as you know. And without a good way to train middle/hidden layers that was a devastating conclusion. Most people abandoned the neural approach to AI research. Then more than a decade later, a few different people realized you could apply the long known calculus derivative chain rule & gradient descent! The field picked up some momentum again. You could say, duh! That isn’t even an advance, we should have already known that! On the other hand, suddenly multi-layer networks were trainable and the field came back from the dead. Similar to now. Advances might be very minor, but their impact still profound. Today with lots more computing power and data, a machine can write poetry about cheerleading, spoken with the voice of Arnold Schwarzenegger in a Shakespearean play. And then proceed to converse in that voice. Easy to make light of, not the most useful skill. Just another step in chat achieved mostly by scaling up what we already knew how to do. But on the impact side, these models are now flexible in integrating totally different types of knowledge, from vast numbers of subjects, and returning results in real time. And as you point out, this progress is being made despite most papers and efforts being very uninspired. The lesson may be, that the means by which we achieve general AI are likely to seem very mundane. That won’t stop their impact. (And not finding global minima isn’t an impediment of any kind. Our own minds are notoriously unreliable and non-optimal. Intelligence and mathematical perfection are very different things. Virtually unrelated problems.)