10 ms·
Ugh. I just spent the past few days reading this exact paper and preparing a detailed presentation on it for my job as an AI researcher, and headlines like this
by mkaic 3y ago
Ugh. I just spent the past few days reading this exact paper and preparing a detailed presentation on it for my job as an AI researcher, and headlines like this make me roll my eyes very hard.
MEGABYTE is indeed a cool new architecture, but it is still very much just a proof of concept at the moment. The paper shows that the model can compete with (but not decimate) vanilla Transformers on the scale of ~1B parameters in long-sequence prediction tasks. They did not "unleash" anything, and scalability to very large parameter counts and datasets still has not been tested.
I'm certainly very excited to see where this architecture goes as the community gets ahold of it and starts developing it, but to call it "revolutionary" this early on is disingenuous. I personally have a few experiments I want to run with it, but I put the probability of it being a true GPT-killer at <20%. I would love to be wrong, though!
- nwoli 3y agoI wish you had a link to your blog or twitter in your bio, you have the kind of nuanced tone I’d like to hear more from
- riwsky 3y agothey should consider doing this for a living
- mkaic 3y agoI'll consider doing it for a living when it can pay my rent, which is unlikely to be any time soon :P
- mkaic 3y agoOh, thanks! Though to be honest, my blog does have a good bit of wild speculation on it, as I'm a bit of a hypocrite on that front :) I've added links to both in my bio.
- lordofgibbons 3y agoI'd love to watch a recording of this presentation or write up, if possible!
- mkaic 3y agoHmm, I may make a blog post using some of the graphics I made for the presentation. Can't make any guarantees though.
- jdthedisciple 3y agoWhen I started reading this and immediately came across the word "groundbreaking" I paused for a sec and thought to myself: Let me just pretend this word isn't there, it's probably an exaggeration. It takes a special kind of adaptation to filter out all the 10%-40% bluff usually found in these kinds of news articles.
- derefr 3y agoNow I kind of want to see a browser extension that deletes all the intensifier adjectives from runs of text, and is able to be enabled/disabled per domain.
- mkaic 3y agoSo a less unhinged version of [0]? [0] https://xkcd.com/1288/ https://xkcd.com/1288/
- mejutoco 3y agoBit random, but a while ago I created this extension to remove "actually", "literally" and other words (it is slighty configurable) that can be annoying. It is sort of a joke, but some days I think it is not a joke. https://chrome.google.com/webstore/detail/actually-no-remove-actual/dnjekdlckidinafaebkhcojebjjpphkb https://chrome.google.com/webstore/detail/actually-no-remove... Maybe I should extend it :)
- kjkjadksj 3y agoIts always better to just get at the underlying, usually far dryer scientific paper. Even if its not in your field try and read the intro and the conclusion, the authors will describe why this is generally relevant with much less flowery language than the press release. Even a word like “novel” is bold to use too many times.
- xp84 3y agoEvery 'news' site: "But how are we gonna get clicks if we don't vastly exaggerate and sensationalize everything?" Thanks for the actual sober analysis. As not-an-AI-researcher, I could have been easily bamboozled by an article like this.
- PoignardAzur 3y ago> The paper shows that the model can compete with (but not decimate) vanilla Transformers on the scale of ~1B parameters in long-sequence prediction tasks. They did not "unleash" anything, and scalability to very large parameter counts and datasets still has not been tested. One thing to keep in mind in particular is that the vast majority of transformer alternatives/improvements/optimizations that initially showed promise ultimately ended up scaling less well than the baseline transformer architecture. The transformer has been weirdly hard to improve/dethrone.
- mkaic 3y agoYeah, it's had remarkable staying power. I think it says a lot that I can see "Vaswani, et al." cited in a paper and know exactly what the author is referring to lol. I don't usually memorize researcher names but the original Attention Is All You Need paper is just freaking ubiquitous.