5 ms·
This is an uncharitable and oddly dismissive take (i.e. perfect for HN, I suppose). Today's incredible state-of-the-art does not exist without the transformer
by a_wild_dandan 3y ago
This is an uncharitable and oddly dismissive take (i.e. perfect for HN, I suppose).
Today's incredible state-of-the-art does not exist without the transformer architecture. Transformers aren't merely some lucky passengers riding the coattails of compute scale. If they were, then the ChatGPT app which set the world ablaze would've instead been called ChatMLP, or ChatCNN. But it's not. And in 2024 we still have no competing NLP architecture. Because the transformer is a genuinely profound, remarkable idea with remarkable properties (e.g. training parallelism). It's easy to downplay GPTs as a mostly derivative idea with the benefit of hindsight. I'm sure we'll perform the same revisionist history with state-space models, or whatever architecture eventually supplants transformers. Do GPTs build on prior work? Do other approaches and ideas deserve recognition? Yeah, obviously. Like...welcome to science. But the transformer's architects earned their praise -- including via this article -- which isn't some slight against everyone else, as if accolades were a zero-sum game. These 8 people changed our world and genuinely deserve the love!
- hn_throwaway_99 3y agoQuestion for you, as someone relatively new to the world of AI (well, not exactly new - I took many courses in AI, including neural networks, but in the late 90s... the world is just a tad different now!) Is there any good summary of the history of AI/deep learning from, say, late 00s/2010 to the present? I think learning some of this history would really help be better understand how we ended up at the current state of the art.
- Swiffy0 3y agoI've been following a data science course called "The Data Science Course 2023: Complete Data Science Bootcamp" at Udemy. The course starts all the way back from basic statistics and goes through things like linear regression and supposedly will arrive at neural networks and machine learning at some point. So I don't know if something like this is exactly what you're looking for, but I think that, in general, if one wants to learn about (the history) AI, then it might be a good idea to start from statistics and learn about how we got from statistics to where we are now.
- Oioioioiio 3y agoNot sure if this is wayy to off of what you ask: https://www.youtube.com/watch?v=OFS90-FX6pg https://www.youtube.com/watch?v=OFS90-FX6pg
- felixleungsc 3y agoMaybe this? https://arxiv.org/abs/2212.11279 https://arxiv.org/abs/2212.11279
- mptest 3y agoThe worlds I see by fei fei li is exactly what you're looking for, intertwined with a beautiful story of a brilliant scientist's perseverance
- cooljoseph 3y ago> And in 2024 we still have no competing NLP architecture. No, we do. State space models are both faster and scale just as well. E.g., RWKV and Mamba. > Transformers aren't merely some lucky passengers riding the coattails of compute scale. Err... they are, though. They were just the type of model the right researchers were already using at the time, probably for translating between natural languages.
- goertzen 3y agoIt’s reply’s like this that make HN a good place.