8 ms·
Is it just me or does anyone else cringe when they read "is/is not all you need" in the title of an AI-related paper? Also, what is "SOTA" for review? There is
by haldujai 4y ago
Is it just me or does anyone else cringe when they read "is/is not all you need" in the title of an AI-related paper?
Also, what is "SOTA" for review? There isn't exactly a benchmark to compare...
In terms of comprehensiveness they don't mention PaLM + variants, which probably should be mentioned as it is currently the largest LLM with SOTA on several benchmarks (e.g. MedQA-USMLE).
In terms of correctness, I admittedly skipped to the sections I'm familiar with (LLMs) but I don't understand why they are distinguishing 'text-science' from 'text-text', they're both text-text and there is no reason why you can't, for example, adapt GPT3.5 to a scientific domain domain (some people even argue this is a better approach). A lot of powerful language models in the biomedical domain were initialized from general language models and use out-of-domain tokenizers/vocabularies (e.g. BioBERT).
The authors also make this statement regarding Galactica:
"The main advantage of [Galactica] is the ability to train on it for multiple epochs without overfitting"
This is not a unique feature of Galactica and has been done before. You're allowed to train LLMs for more than 1 epoch and in fact it can be very beneficial (see BioBERT as an example of increasing training length).
People GENERALLY don't do this because the corpus used during self-supervised training is filled with garbage/noise, so the model starts to fit to that instead of what you desire. There is nothing special about Galactica's architecture that specifically allows/encourages longer training cycles but rather they curated the dataset to minimize garbage. As another example, my research involves radiology NLP and when doing domain adaptive pretraining on a highly curated dataset we have been going up to 8 epochs without overfitting.
- azinman2 4y ago> Is it just me or does anyone else cringe when they read "is/is not all you need" in the title of an AI-related paper? I believe it's a reference to 'attention is all you need,' which is a very famous paper now. I'm guessing you probably already know that -- did you cringe with it? It was a landmark paper so at least it's worth hyperbole in the title. Maybe I'm missing that its usage started before that. Reminds me of my ML prof who would complain about people using the word 'optimum' or 'optimized' in titles of papers... one could almost always optimize more, and with respect to what is not specified by the title.
- haldujai 4y agoYes, aware of the reference and hence the cringe. 'Attention is all you need' is a clever play on words for a seminal paper whose title directly related to its main scientific contribution. Not sure that every other AI-related paper needs to copy this format, this trend is the academic equivalent of click-baiting (seemingly trying to associate with the Vaswani et al paper) and in my anecdotal experience the usage of this play on words seems inversely correlated with the paper's quality.
- earleybird 4y agoI believe it has an analogue with Heinlein's "there are two kinds of jokes; funny always and funny one time". Subsequent uses are a bit cringe in the same way that a subsequent telling of a funny-one-time joke is.
- ryanmcbride 4y agoThose kinds of references/cliches in titles always bug me too. Always feels lazy. I chalk this one up there with: Dr. Strangelove Titles - "<product>: or how I learned to stop <doing something> and love <idea>" Dairy Farmers of America reference - "Got <Product>?" Breakin' 2 reference - "<Product> 2, Electric Boogaloo" Honorable Mention, titles that end in phrases like "And That's a Good Thing!", "Here's What That Means", "This is Why That Matters"
- hyperdimension 4y ago"Repeated Tropes Considered Harmful"
- nextaccountic 4y ago> 'Attention is all you need' is a clever play on words Can you elaborate? (I'm not a native speaker)
- jrvarela56 4y agoI think the clever play on words is the fact that the title is cute and references the main finding directly: transformers have a mechanism called 'self-attention' that distinguishes them from previous approaches.
- zone411 4y agoI agree. This paper also has issues in the areas I'm familiar with (e.g. Jukebox: being "limited to English songs" is the least of its issues). I don't know why it was randomly submitted to HN.
- sjsdaiuasgdia 4y agoI like that you ask what SOTA means when the acronym is expanded in the title you see as soon as you click the link. "ChatGPT is not all you need. A State of the Art Review of large Generative AI models"
- deleted 4y ago[deleted]
- zone411 4y agoYou missed the point. It's obvious what it means to anybody in ML. But that's not how the term is used.
- haldujai 4y agoI’m aware of what SOTA means, but how does that apply to a review? How do we or the authors assess that it is in fact the state of the art (best) without an objective measure? If they mean it’s the most current, every review is state of the art at time of publication by definition.
- tracerbulletx 4y agoSensationalist click bait titles have taken over more or less completely at this point. :(
- dpkirchner 4y agoIf this counts as sensationalist then I'd say the term click bait has lost all meaning.
- tracerbulletx 4y agoSensationalist might not have been the best word choice, I just kind of feel like the title is clout hunting or something by using the word choices it uses.
- thelittleone 4y agoWe need an AI that de-click baits titles. That would be a great service to the world.