6 ms·
I wish we could stop with the “X is all you need” papers! The first one was unintuitive and so are the rest.
by opnac 3y ago
I wish we could stop with the “X is all you need” papers! The first one was unintuitive and so are the rest.
- mottiden 3y agoI agree. The paper is really interesting, but the title not so much :)
- jillesvangurp 3y agoA bit click baity at least. And without opening it you have no chance to understand what this is about. I know HN has a policy against editorializing but in this case, a brief summary would have been helpful.
- mottiden 3y agoThe paper introduces a new method for text generation, named Copy-Over-Generate (COG), which differs from traditional approaches that generate words from a fixed vocabulary. Instead, COG progressively copies phrases from a massive text collection, aiming to generate coherent text continuations through multiple rounds of phrase retrieval. COG stands on the line of retrieval-augmented text generation research but takes a radical step forward. Unlike previous work that combines retrieval and generation, in COG, retrieval is generation. COG shares some ideas with previous work such as replacing the fixed vocabulary with a nonparametric phrase table. The paper presents experimental results showing the advantages of COG over strong baselines in three experimental settings: standard language modeling (using the WikiText-103 dataset), domain adaptation (using the Law-MT dataset), and an enlarged phrase index (using the En-Wiki dataset). Despite the promising results, the authors acknowledge that there are some flaws in the COG method. For example, COG may copy a phrase that is incoherent with the previously copied phrase, or it may only copy a part of a complete phrase, leading to inaccurate generation results.
- CamperBob2 3y agoBefore long, if these NN refinements continue at their current pace, it's going to become impossible to tell synthetic HN posts from organic ones. Going to get weird.
- flangola7 3y agoThis is already the case with LLaMA.
- hardware2win 3y ago"is all you need" is considered harmful just like "considered harmful" is considered harmful by HNers?
- MR4D 3y agoGiven that redundancy is considered harmful, you probably want to create a ConsideredHarmful class and then a ConsideredHarmfulFactory to make for a more enterprise-ready structure. :)
- hardware2win 3y agoThats so 2015, we need to move it to ConsideredHarmful Microservice
- ot 3y agoAt least in this case it's self-referential (not sure if intended)
- lolinder 3y agoI agree that the copycats are wearing thin, but the original paper's title seems fine to me. It's an accurate description of the breakthrough they made. The first few sentences of the Attention is All You Need abstract explain it pretty well: > The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf https://proceedings.neurips.cc/paper/2017/file/3f5ee243547de...
- macleginn 3y agoThe first sentence seems to imply that the main thing they want to do away with is the encoder–decoder arch, which they actually kept. (But BERT and GPT later did manage to simplify it.)
- _ea1k 3y agoIt took a lot of my attention to even begin to understand that, but in the end, I agree with you.
- KnobbleMcKnees 3y agoBut I'm so close to finishing "goto is all you need"!
- lotsofspots 3y agoBeat me to it, I was going to go for "X Is All You Need Papers Considered Harmful"
- anticensor 3y agoAnd the dual of it, "X is harmful papers are all you need".
- nonameiguess 3y agoDidn't somebody prove the mov instruction is Turing-complete all by itself? I believe some code obfuscators actually take advantage of this.
- rubslopes 3y agoThere's this guy in Twitter fighting against titles like that: https://twitter.com/SayWhatYouFound https://twitter.com/SayWhatYouFound
- furyofantares 3y agoI agree. X is all you need considered harmful
- butterisgood 3y agoX is all you need considered harmful is all you need!