5 ms·
Pretraining on the Test Set Is All You Need
- RecycledEle 3y agoIs this a joke? I guess there are cases where training on a tests set is reasonable, but why use a LLM in that case?
- SubiculumCode 3y agoThe Discussion lets on that it is satire. So does Table 1.
- naniwaduni 3y agoSo does the title. And the abstract, if you have enough of an understanding of the field to make it fruitful to be reading prepubs in the first place...
- YetAnotherNick 3y ago> Disclaimer: if you haven’t figured out by now that this manuscript is satire, this manuscript is satire.
- asah 3y agoLOL: https://www.google.com/search?q=site%3Aarxiv.org+%22is+all+you+need%22 https://www.google.com/search?q=site%3Aarxiv.org+%22is+all+y... (apparently, a lot of things are "all you need" !)
- sva_ 3y agoA significant part of the work in writing an ml paper seems to go into coming up with a catchy title.
- unflxw 3y agoPerhaps it's not great to set the expectation that any specific approach to AI "is all you need"? (or, in other words, "Is All You Need" Considered Harmful)
- deleted 3y ago[deleted]
- SubiculumCode 3y agoThere were certain language choices in the paper that led me to think that just maybe they weren't doing what they were doing, but the longer I read, the more certain I became that they were doing exactly what they shouldn't, wasting my time.
- maleldil 3y agoIt's satire. They're criticising "good" results obtained by trained on contaminated datasets.
- SubiculumCode 3y agowell yes, so said the discussion. I was just irritated that i read theough it that far. but not being a machine learning pro, I'm happy that I knew enough to be highly suspicious.
- marcosdumay 3y agoIt's a 3 pages paper with absurd ideas from the title on. Imagining it was not clear enough, the author decided to name the model something 'pronounced “fictional”', and shows how it can answer questions that weren't even asked yet with perfect accuracy right on the abstract. Accept that as an exercise of familiarizing yourself with the area's language. If you are interested on it, it's not lost time.
- naniwaduni 3y agoIt's a two-and-a-half-page paper with about a page and a half of total text with an obvious joke title, absurd and obviously joking claims in the abstract, and names the model "fictional". This is all before you even click through to the PDF! If it takes you reading the whole paper up to the point where the author explains the joke to realize that the paper tagged satire is satire, I mean, I'm not sure paper-reading is for you. There are unmarked papers out there that aren't even satirical, just wrong!
- alephxyz 3y agoThe thread that inspired the article: https://nitter.cz/suchenzang/status/1701615026648605095 https://nitter.cz/suchenzang/status/1701615026648605095
- civilized 3y agoContamination seems to be a very serious problem for LLM evaluation. These models typically seek to learn from, essentially, The Entire Internet. What's left, after you use, say, "half" of The Entire Internet for training? What's left that is definitely not the same as the half you used? How can you be sure, when you're working with terabytes or petabytes? Credible benchmarking is going to have to become a field in itself.
- tveita 3y agoHow do you even put together an open evaluation benchmark without it being slurped up into the training set the minute someone puts it on Github? Maybe it needs to be obfuscated, or all the questions should be obscene so they don't pass the content filtering.
- gwern 3y agoThere are 'canary tokens' that you can add to your dataset which are supposed to be respected by self-regulating scrapers & responsible ML practitioners to grep them out before doing any training. How well it works is unknown, of course...
- deleted 3y ago[deleted]
- EMM_386 3y ago> Disclaimer: if you haven’t figured out by now that this manuscript is satire, this manuscript is satire I find this annoying, and not funny, honestly. Why are satire papers mixed in with real ones on arxiv.org? Obviously, I'm not a research scholar and don't know how the whole process works, but as someone who likes to know whether they are dealing with fact or fiction from the start, this was a waste of time.
- waveBidder 3y agoIf I were you, my take away would be that if you can't evaluate a paper critically enough to immediately tell this is satire, maybe stick to sources that have been vetted by those who can (published papers, books, etc) until you're familiar enough with the field. Arxiv is for researchers communicating with other researchers, and the tendency of AI/crypto groups to use it for marketing has been a disservice to the public.
- gardnr 3y agoThere aren't any special critical evaluation skills required. The arxiv page says: > Comments: 3 pages, satire Tangentially, it's worth pointing out that you are gatekeeping read access to a resource because of a write access problem. If you don't want open access then go back to Elsevier.
- xboxnolifes 3y agoTechnically they're gatekeeping because of a read ability problem.
- gardnr 3y agoYou are correct! I must have had a read ability problem when they conflated the two issues.
- waveBidder 3y agoI was giving advice for how to learn a subject. The text is obviously satire even without the warnings if you understand the fundamentals of ML. I'm not saying they need a PhD or something. I have no problem with open access, and would rather journals were required to be open access. But reading brand new results that haven't had any peer review without familiarity with the field is putting the cart before the horse. I certainly don't learn a new subject by reading the hottest papers on arxiv.
- deleted 3y ago[deleted]
- gardnr 3y agoThis is funny because everyone uses the same tests to evaluate different models. One example is the Hugging Face OpenLLM Leaderboard. [1] The problem that this "paper" is making fun of is that any model can score very well on these tests when the content of the tests themselves is used in the training of the models. Here is one example of this happening with a 13B model that scored way too high. [2] In reality, when reviewing test scores, it's difficult to discern whether a model has been pre-trained on some specific content that is related to the test, or if the model has been trained in a way that produces novel emergent abilities not seen in other models. [3] This means that LeaderBoards and test scores for LLMs provide a low-confidence metric for what the propose to be testing. The paper's title is itself a continuation of the phenomenon where papers are named after the landmark paper "Attention Is All You Need". [4] A good example of this is the "Textbooks Are All You Need" [5] paper that propsed pre-training in an incremental way, similar to how humans learn, produces novel results. 1. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... 2. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard/discussions/213#64e8a3267ef2c72cbce22126 https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb... 3. https://arxiv.org/abs/2206.07682 https://arxiv.org/abs/2206.07682 4. https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1706.03762 5. https://arxiv.org/abs/2306.11644 https://arxiv.org/abs/2306.11644
- ianand 3y agoReminds me of “FedEx is all you need” from ICML https://x.com/quantumaephraim/status/1685358421359341568?s=46&t=x6aIMKxtOHuyj50nx-e74A https://x.com/quantumaephraim/status/1685358421359341568?s=4... Full disclosure: I have an AI side project that also uses “is all you need” in the title.
- gardnr 3y agoThis made my day: The implementation is just enough to run very small workloads: * Full GPT2 small (124M parameters) model including byte pair encoding, embeddings, multi-headed attention, and multi-layer perceptron stages * Inference/forward pass only (no training) * Context is limited to 10 tokens in length * 10 characters per word limit * Zero temperature output only This sheet is very big. Unfortunately, it is not unusual for Excel to lock up while using this spreadsheet. It is highly recommended to use the manual calculation mode in Excel. from: https://spreadsheets-are-all-you-need.ai/ https://spreadsheets-are-all-you-need.ai/