5 ms·
Is it that a lot of capacity is unused in those behemoth LLMs, or that the smaller language model just mimics the reasoning task? (Mimics the mimicking?)
by ziofill 3y ago
Is it that a lot of capacity is unused in those behemoth LLMs, or that the smaller language model just mimics the reasoning task? (Mimics the mimicking?)
- famouswaffles 3y agoThere is no actual distinction between the "real" thing and "mimicking". The datasets behemoth LLMs are trained on include a lot of noise that derail progress. They also just contain a lot of irrelevant knowledge that the LLM has to learn or memorize so an obscene amount of parameters is required. When you're not trying to teach a language model the sum total of human knowledge and you provide a high quality curated dataset, the scale barrier is much lower. https://arxiv.org/abs/2305.07759 https://arxiv.org/abs/2305.07759
- behnamoh 3y agoWhy don't people just train LLMs on pure Wikipedia, arXiv, and other scientific websites? That would reduce the noise and improve hallucinations, no?
- vczf 3y agoI don't think it would know how to have a conversation if it's never been exposed to one before.
- taneq 3y agoThat’s what the fine tuning is about. It learns the language, concepts etc. from the main dataset and is then tweaked by continuing to train on a smaller, high quality, hand curated dataset. That’s how it learns to generate conversational responses by default instead of needing a complicated prompt.
- brucethemoose2 3y agoThat is essentially Phi. The results are promising. But LLMs do gain useful "emergent" properties from training on massive lower quality datasets.
- deleted 3y ago[deleted]
- astrange 3y agoIt has to learn the meaning of words, including implicit associations, and to do that it needs to see approximately all the English text ever. We don't know how to balance this with only feeding it useful knowledge.
- eru 3y agoIt doesn't necessarily have to see approximately all the English text ever. Real people don't learn English like that, for example. It's just that given what we know about neural networks, it's often easier and simpler and more effective to increase the amount of training data than to change anything else.
- basch 3y agoReal people might not speak every dialect of English. They may not be well versed in local grammatical oddities.
- marshray 3y agoDoesn't seem to be much of a problem in practice? If someone knew all the math and science in Wikipedia, for example, I think they'd probably be forgiven for not knowing every regionalism.
- astrange 3y agoUnfortunately models aren't always good at knowing what they don't know ("out of distribution data") so it could lead to confidently wrong answers if you leave something out. And if you want it to be superhuman then you're by definition not capable of knowing what's important, I guess.
- eru 3y agoBtw, models like GPT 4 can express that they are not confident. But that looks like a 'smeared' out probability distribution on the next token. Not like the text produced by an unsure human.
- msp26 3y agoIt's been done before. "Textbooks are all you need". https://arxiv.org/abs/2306.11644 https://arxiv.org/abs/2306.11644
- fulafel 3y agoThat question seems synonymous with "are those [current] behemoth LLMs close to optimal?" and it seems obvious that they are not. I wonder what ideas there are about how one could estimate the optimal size.
- sp332 3y agoThe large models generalize better. The small models are easier to train for specific tasks.