3 ms·
Oh, come on - this article is from three days ago, but it starts with "Transformers are a new development in machine learning". Transformers have been around fo
by kleiba 4y ago
Oh, come on - this article is from three days ago, but it starts with "Transformers are a new development in machine learning". Transformers have been around for six years now, that's an eternity when you consider how fast this field is moving.
- fulafel 4y agoDeep learning (multilayer perceptrons) was 1965... The timeline is pretty long.
- kleiba 4y agoMaybe on a theoretical level, but of course you will agree that what we mean by deep learning today has only become possible with the availability of sufficient computational power (and Hinton's work around the mid-2000's).
- fulafel 4y agoThere are lots of important moments I guess, I would credit the main break in practicality to the ~1990 Swiss/German stuff: https://people.idsia.ch/~juergen/deep-learning-miraculous-year-1990-1991.html#Sec.%201 https://people.idsia.ch/~juergen/deep-learning-miraculous-ye... But still even the early perceptrons stuff was applied research, I wouldn't call it purely theoretical by any means.
- kleiba 4y agoI agree about your notion of there being lots of important moments in history, and Schmidhuber's contributions are not small by any means. Yet, 1990 was not the year when Deep Learning took off.
- HarHarVeryFunny 4y agoTrue, yet we're still all here talking about them with plenty of questions and confusion, and people outside of ML are suddenly curious to know what's behind things like ChatGPT, unaware of the history. FWIW one of the founders of Cohere (where this article comes from) was Aidan Gomez who was one of the transformer paper authors.