16 ms·
Textbooks are all you need
- chrgy 3y agoDoes using the synthetic data generated by GPT has the same effect of being RLHF-ly Aligned by GPT3.5, kind of like aligning the NNs to get similar performance as GPT.
- famouswaffles 3y ago8 passes so 50B tokens seen. Still, any other >50% HumanEval model is much much bigger https://twitter.com/SebastienBubeck/status/1671326369626853376 https://twitter.com/SebastienBubeck/status/16713263696268533... Now this wouldn't be possible without the high quality synthetic dataset produced by GPT(1B tokens) but this is more evidence in line with Tiny Stories (https://arxiv.org/abs/2305.07759 https://arxiv.org/abs/2305.07759). That is, LLMs only need to be so big (both data and parameters) to learn the total sum of human knowledge (and deal with trash data).
- skepticATX 3y agoThis result puts more emphasis on data quality, which might actually be problematic long term though? The vast majority of human knowledge is not represented in high quality textbooks. It’s not clear from these results but the paper seems to at least imply that the importance of the synthetic data is to “unlock” the pre-training data. My takeaway as a non-expert is that this is a good result for small and efficient models focused on well-defined domains, but a neutral or maybe even a bad result for a model that displays general intelligence.
- airgapstopgap 3y agoThe vast majority of human data can be refined into something approximating high quality textbooks, which is what happened here too.
- skepticATX 3y agoOnly 1/7 of the data they are using is synthetic though. The other 6/7 was just filtered from a larger dataset, not refined in any way.
- WJW 3y agoI wonder if that is true. Intuitively it seems to me that there are probably many areas of knowledge that cannot be reliably summarized into a textbook. Most subjective experiences, for example. Qualia have been notoriously difficult to describe too, even if the might be absolute. Even for fields that lend themselves well to being converted to textbook format there is often a tradeoff between accuracy and conciseness. The more you refine, the more nuance you throw away. It seems like a Very Hard Problem (tm) to know which data are superfluous and which are not, especially at scale.
- cjbprime 3y agoWell, used to be a Very Hard Problem. Now you can just test it: train a model on the textbook and ask it to solve many problem challenges that it hasn't seen before. After you get a score for the model trained on the whole textbook, try removing each sentence in the book in turn. If removing that particular sentence decreased the test scores then keep it in, else throw it away.
- WJW 3y agoI don't think that is sufficient: - Perhaps the model can be improved by adding sentences, but which ones? There is a potentially infinite amount of sentences to add and no good way to select them - Your proposed method handwaves the method of acquiring "many problem challenges that it hasn't seen before", which just moves the problem. Constructing a problem set containing the full range of potential problems to solve is again a Very Hard Problem. The second problem is not just theoretical either: see https://sitn.hms.harvard.edu/flash/2020/racial-discrimination-in-face-recognition-technology/ https://sitn.hms.harvard.edu/flash/2020/racial-discriminatio... for example, where a facial recognition algorithm performed much worse on non-white women because the training set didn't contain enough pictures of them. AI software is notorious for finding this kind of loophole and for overfitting itself for the training set rather than for the real world.
- IanCal 3y agoThe speed of training is really interesting to me too. Llambda labs rent out 8 A100s for ~$9-12/hour. Total training cost there would be ~$850-1200. Another ~$2000 for the GPT training data, some more given there would be input tokens too. Very approachable prices. > Now this wouldn't be possible without the high quality synthetic dataset produced by GPT(1B tokens) Just a note that this is GPT3.5 (I assume turbo?).
- eightysixfour 3y agoThe next step change for ML will be some kind of broad model (like the the LLM equivalent of JEPA) outlining the answer from a “big picture view” and then using an ensemble of small, precise models like this one filling in the details.
- tehsauce 3y agoOverall a very interesting result. I’m not sure why they compare to a list of results for the MBPP benchmark but don’t seem to include these results which are much better: https://paperswithcode.com/sota/code-generation-on-mbpp https://paperswithcode.com/sota/code-generation-on-mbpp
- onos 3y agoI vote the title meme “… is all you need” end now.
- behnamoh 3y agoThe whole NLP field is full of memes. Just look at the model names: Llama, Gorilla, Jarvis, Falcon, Vicuña, Alpaca, etc.
- littlestymaar 3y agoNot just NLP the whole IA field, see YOLO and NeRF.
- SheepHerdr 3y agoThen there's all the sesame street names for NLP models. https://i.redd.it/048rybz0yuv81.png https://i.redd.it/048rybz0yuv81.png
- sn41 3y agoSuggested title: "Is all you need considered harmful"
- ninepoints 3y agoMyths data scientists believe about "is all you need considered harmful"
- bongobingo1 3y ago> using a selection of ``textbook quality" data from the web ... and synthetically generated textbooks and exercises with GPT-3.5 Is this common, training a LLM from another LLMs generated output? How do you avoid "bad code" from GPT, if not out right hallucinations?
- kramerger 3y agoIf you think about it, how do humans learn things? A child gets alot of his knowledge by randomly copying what grownups do.
- famouswaffles 3y ago>Is this common, training a LLM from another LLMs generated output? It's not uncommon. And the level the SOTA LLMs are now, it'll only become more common. It happens because we're at the stage where GPT-3.5/4 can generate much better data (or hit more encompassing or general distributions with the right instructions) than what the majority of LLMs will be typically trained on. Also 4 can recheck output for bugs, mistakes, errors etc It may not be perfect but neither would "natural data" and depending on exactly what kind of data, it might be as close to perfect as you'll get - https://arxiv.org/abs/2305.07759 https://arxiv.org/abs/2305.07759
- zarzavat 3y agoWhen humans think this is what they are doing: retraining themselves on their own learned data. In other words, knowledge is not a conserved quantity. Of course the extent of the extra knowledge you can produce without access to the environment is limited.
- ChatGTP 3y agoDo you have any actual idea if that's the case or are you just saying stuff? I've read that people reorganize information when they sleep, I wouldn't say that's "thinking".
- 3y ago
- ilaksh 3y agoBut it's still only 50 something percent on HumanEval right? So to really prove this out you still need to try to make a much larger model. I wish someone would do something like that but not use OpenAI's models to create the training data because then supposedly it can't be used for commercial purposes. Not that it's easy to create a lot of high quality training data.
- light_hue_1 3y agoThis is a misleading title. GPT-3.5 is all you need! Their whole method relies on getting GPT-3.5 to produce examples and then training a network on those examples. This is a run-of-the-mill method called distillation. There's nothing new or special here.
- waynecochran 3y ago"displays surprising emergent properties" Section 3 states "includes managing intricate algorithmic tasks." I take this as meaning it can write new code (not regurgitation). It is impressive that is can write code as described in the prompt, but I didn't think this was considered "emergent" but "generalization." What is the difference?
- mjburgess 3y agoThere is no difference, "emergent" is another hype term used in place of "discontinuous generalisation". ie., with 1MB data you dont get generalisation, with 1TB you do. So they call that emergence. I'm almost at the point where I can let these things go; but each new bit of mystifying hype, another stone falls in my shoe.
- irthomasthomas 3y agoYeah, the way I read "emergent" is as "surprising" to our intuition, but technically expected. So I read this as surprisingly surprising performance.
- kirab 3y agoThe English explanations you find first for emergent are not really fitting here. Interestingly the Germany explanation for the same word seems to be much better. I'm pretty sure they mean emergent as in emergent gameplay. Here's a good explanation for that: > Emergent gameplay refers to complex situations in video games, board games, or table top role-playing games that emerge from the interaction of relatively simple game mechanics. So emergent is not inherently surprising. Games like Dwarf Fortress or Rimworld have a lot of emergent complexity but most of it was not surprising. So in the context of LLM it means that complex reasoning is emergent based on simple underlying mechanics. And since we thought humans’ complex reasoning skills were very unique.. We are surprised by the emergence of it in LLMs. PS: The explanation for "emergence" is much better: > In philosophy, systems theory, science, and art, emergence occurs when a complex entity has properties or behaviors that its parts do not have on their own, and emerge only when they interact in a wider whole.
- lisasays 3y agoSuriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, Yuanzhi Li Actually, it turns out that impact factor is all you need.
- behnamoh 3y agoTurns out, all you need is, indeed, all you need.
- elif 3y agoI bet it still gaslights invented facts.
- daniel-cussen 3y ago[dead]
- z7 3y agoSam Altman has said it will take 1-2 years to solve the hallucination issue.
- stephencanon 3y agoSo it’ll be solved a little after Tesla achieves full self-driving, and just before we get fusion power. Great.
- ototot 3y agowhy don't they use a real textbook?
- elefanten 3y agoBecause it’d either be expensive or illegal, and all content owners are paying attention now
- haliskerbas 3y agoIt’s interesting to me how a lot of these new models use GPT for the synthetic data generation to then train a better or more focused model
- gtsnexp 3y agoIs it possible to test or access this model through a web interface or any other means?
- rjb7731 3y agoIts interesting this paper & the Orca LLM paper from Microsoft are using GPT3/4 model outputs to train 'powerful' models. Big question is will they allow your average joes/businesses to do the same on their own data? Doubt it, considering it breaks OAIs terms of use at present. Will opensource lead the way on this? Bring on Llama-v2
- regularfry 3y agoLooking at the `Educational values deemed by the filter` table on page 5, I can't help wondering whether compressibility might not be a usable proxy for educational value. High compressibility would imply low information content. High marginal compressibility when added to a large dataset would imply the information is already in there. This is basically Normalised Compression Distance all over again - it's only a rough proxy, but I guess the question is whether it could work at all, rather than is it as good as this result. If that worked as a proxy value, you could sidestep needing GPT-4 at all.
- jackphilson 3y agoThe words "are all you need" are all you need
- eimrine 3y agoAll you need is weed, weed, weed is all you need.
- hcks 3y ago´all you need’ is going to become an anti-signal very quick
- nojvek 3y agoThe cool kids now use 'everything everywhere all at once' https://arxiv.org/search/?query=everything+everywhere+all+at+once&searchtype=all&source=header https://arxiv.org/search/?query=everything+everywhere+all+at...
- n0us 3y agoAll you need considered harmful
- ekaryotic 3y ago[flagged]
- CatWChainsaw 3y agoIs your company still using buzzwords? You're behind the times, friend, buzzphrases are all the rage now!
- q_andrew 3y agoIronically, if they are using attention methods, then dictionaries are not the only thing you'll need.
- cafaxo 3y agoFrom page 6, when they describe their synthetic textbook dataset: "Consider the matrix A = np.array([[1, 2], [2, 4]]). We can check if this matrix is singular or nonsingular using the determinant function. [...]" No. The determinant is not a suitable way to do that. A proper way to numerically measure singularity would be to compute the condition number of the matrix (the ratio of its largest to smallest singular value).
- aix1 3y agoAm I misreading this, or is it really about a 2x2 matrix? (In which case computing the determinant involves just two multiplications and one subtraction.)
- cafaxo 3y agoThey define a generic "is_singular" function and test it with a 2x2 matrix. The problem with the determinant is not about performance. It is just useless for determining if a matrix is singular. The thing that gives it away is that the determinant is influenced by a rescaling of the matrix: det(s A) = s^n det(A) where A is a n x n matrix As an example, would you say that [[1e-10, 0], [0, 1e-10]] is singular? It has condition number 1.
- renonce 3y agoIt does work in theory and for integer numbers. Such an algorithm might be used in practice when numerical stability is concerned, but condition number can certainly be defined somewhere in another textbook and you just need more context to tell it to use another method.
- cafaxo 3y agoEdit: Sorry, I completely messed up my original answer here. A better version: Let's say we are in a setting where we only work with integers. A matrix is invertible iff its determinant is invertible in the underlying ring. The only invertible elements in Z are -1 and 1. So, the code is also incorrect in the integer setting. Here, we should not check for 0, but for -1 or 1.
- SergeAx 3y ago> phi-1 attains pass@1 accuracy 50.6% on HumanEval So, a coin toss effectively, right?
- revertmean 3y agoNo, scoring 50% on a test is not the same as a coin toss, for very obvious reasons.
- dist-epoch 3y agoOnly if the questions are binary (yes/no). If multi-choice or free-text, random answering will be much below 50%.
- jug 3y agoWe have so much to learn and optimize. I wonder how far we might get. But it was evident from the start to me that e.g. harvesting Reddit and other sources and just crossing your fingers, hoping for the best may not be the most optimal method. Quality is everything and part of quality is also having it be structured and geared towards an AI which may not be identical to being geared for human reading.
- mguin 3y agogg
- opan 3y agoI was hoping this would be about learning on your own, but it's "AI" stuff.
- lovasoa 3y agoThe title is a reference to the famous "Attention is all you need" paper. https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1706.03762
- gorjusborg 3y agoApparently, when coming up with titles for papers 'All you need is all you need'.
- TeMPOraL 3y ago(Someone has to say the obvious thing.) "'All you need' considered harmful"
- gorjusborg 3y agoThe internet increased the speed of information propagation, but it also made 'cute' imitation annoying way faster. I guess it shouldn't be surprising. With reams of information piling up, it can be hard to get noticed. So you look for gimmicks to get attention (hah!). The trouble is that you are not unique and a thousand other people have the same idea... and your cute imitation is just tired.
- TeMPOraL 3y agoWell, of course, which is why I prefixed the imitation with a self-referential remark in parentheses, meant to turn a low-effort attempt at humor into a meta-joke, that the audience would consider clever. It used to work, but judging by your diatribe, maybe it got tired already and I didn't notice.
- 3y ago
- mkl 3y agoThe title seems rather speculative given that they didn't use any textbooks as training data, or even anything especially similar to textbooks. They used mostly StackOverflow data and source code from The Stack dataset, and also explanations, examples, and exercises generated by GPT3.5.
- gench 3y agoExactly.. Considering the model that generated their exercises was also trained on Stack Overflow data, one may say, not all but what you definitely need for LLM training is Stack Overflow data. People keep getting it granted but I dont think it will continue that way for a long time.
- atemerev 3y agoModel binary to Huggingface or it didn’t happen.
- lcrmorin 3y agoCan't believe I had to scroll that much to see this comment. Upvoted and Thanks.
- amai 3y agotldr; "Just as a comprehensive, well-crafted textbook can provide a student with the necessary knowledge to master a new subject, our work demonstrates the remarkable impact of high-quality data in honing a language model’s proficiency in code-generation tasks. By crafting “textbook quality” data we were able to train a model that surpasses almost all open-source models on coding benchmarks such as HumanEval and MBPP despite being 10x smaller in model size and 100x smaller in dataset size. We hypothesize that such high quality data dramatically improves the learning efficiency of language models for code as they provide clear, self-contained, instructive, and balanced examples of coding concepts and skills."
- reil_convnet 3y agoInteresting that two of the most powerful techniques which can essentially kill chat GPT : Orca and Textbooks, both come from Microsoft.