8 ms·
Tiny Language Models Come of Age
- dmezzetti 3y agoI'm a big believer that smaller models will take hold. The throwing paint at the wall strategy of having 100B+ models and random data won't scale. It's possible it will take new model architectures to get there. But having more fine-tuned control over model parameters is a pattern that will emerge. Then smaller, domain specific models can be joined together to form larger models, if full generalization is needed. I have an article on building micromodels that discusses some of this: https://neuml.hashnode.dev/train-a-language-model-from-scratch https://neuml.hashnode.dev/train-a-language-model-from-scrat...
- pama 3y agoEmpirically, perhaps due to the lottery ticket hypothesis, one needs to wait a lot longer for small models to reach a useful loss, so if one is impatient it almost always pays off to train the largest model that a given compute configuration can support.
- benrutter 3y ago100%, I'd be amazed if we don't hit a time when the huge models we currently have are looked at the same way we see old fashioned room size computers now.
- bavell 3y agoVery apt analogy, stealing this :)
- whimsicalism 3y agoI would take the other side of that bet for sure. Modern computers don’t have less ram than the old fashioned room size computers. Amount of RAM seems much more analogous here than the size of the hardware running the software. Of course the latter will go down, but the former? I don’t see why.
- tensor 3y agoTechniques like distilling show that you can in fact get similar performance with 60% fewer parameters. Yes it depends on making the large model first, but that only implies that somehow the large model is perhaps more useful in initial learning. And if that's the case, then it would be rather surprising if we couldn't identify why and create new training techniques that go directly to the smaller equivalent model. So in this way you've already lost your bet, as we already know that it's possible to create vastly smaller models with similar performance.
- whimsicalism 3y ago60% is not vastly smaller in the way that I meant - we are talking about differences of three orders of magnitude in this article and that is what I am skeptical of. Nevertheless, I have not actually seen a model exhibiting high-level LM capabilities (of the GPT-4 level) that was distilled. Just because a model can match in perplexity with fewer parameters is not the same as matching the performance as subjectively measured by humanity interacting with the model.
- klyrs 3y agoEarly days, yet. If we could cut the size by 60% every few years, we'd be looking at a situation akin to Moore's law where a performant AI might fit in my grandkid's pocket.
- fnordpiglet 3y agoThat doesn’t mean that a much larger with high quality training data and new training techniques won’t be more powerful than a smaller model. We have incredible efficiencies in modern computers in addition to immense resources. They both are needed for the best performance. I suspect it’s the same for LLM. Superior data, superior techniques, and superior scale will yield superior models.
- yayitswei 3y agoInformation density will increase in models just as spatial density increased for hardware.
- dekhn 3y agoThe computers I use are basically warehouse-sized now.
- CuriouslyC 3y agoYou could always start with a big model then periodically prune low activation parameters as part of fine tuning until the desired size is reached.
- api 3y agoI’ve been wondering about variations on panel of experts designs that use a “mob” of small models or possibly a network of them. Sort of like crowdsourcing with baby AIs. Could we have libraries of small models that are all experts on niche subjects that basically collaborate? The advantage is that it’s easier to scale and distribute lots of small models on commodity hardware than it is to run giant models that require heaps of contiguous ram. I’d love to have some time to play with versions of this.
- nickpsecurity 3y agoThe best design we know is how God made humans. Our brains have specialized parts that work together with analog components. We’re raised on knowledge of increasing complexity as humans, given general-purpose heuristics, and then we specialize. From there, small teams of specialists interacting typically do the best jobs. So, I’d try to do stuff like that first since it’s proven in the only general intelligence that exists. If I do experiments, I’m making models specialized from foundational to programming to specific language to specific tasks. Each pass will have training data so it picks up general concepts. Then, it will be trained to focus more on what it needs. Eventually, what’s slushing around in its neurons should be just the right information. Or it won’t work. Anyway, my concepts that I published are at this link: https://gethisword.com/tech/exploringai/index.html https://gethisword.com/tech/exploringai/index.html (See alternative models section.)
- atorodius 3y agoAgreed with the overall point (small models will prove powerful) but I disagree with: > But having more fine-tuned control over model parameters is a pattern that will emerge This assumes parameters do something interpretable. IMO, it’s more that parameters themselves are meaningless/dumb and patterns evolve via interaction of many such units (like in ants)
- gwern 3y ago> I'm a big believer that smaller models will take hold. The throwing paint at the wall strategy of having 100B+ models and random data won't scale. But all these smaller models are based on those 100b+ models! Like OP on TinyStories relies on larger models twice, to generate and then evaluate. Or the recent wave of small models which are all based on extracting data from GPT-3/4 to borrow the capbilities+RLHFing for free. You can talk about how interesting it is and how much of an overhang neural nets have in terms of being overparameterized (which is an important AI safety/capabilities issue - the first AGI will be the largest, slowest, and worst one, by a long shot), but the one thing it doesn't tell you is that smaller models can replace big models entirely. Because they are parasitic on said big models, so you still need to train the big models to begin with. (The smaller models also seem to lose a lot of the qualitative capabilities of larger ones, like meta-learning. Their TinyStories model does only stories. I doubt you could get it to 'dax a blick'.)
- benxh 3y agoNot all of them per se, take a look at something like Mistral. It's a 7B model displaying incredible performance. IMO, we still haven't even scratched the surface of what is possible with small LLMs. Especially not with pre-filtered/classified pre-training data. (Interesting LLMs based on their data approach and relatively small size: Qwen, InternLM, Mistral, Phi)
- pama 3y agoIn the context of this article, the small/tiny models were 1--30 million parameters and the large model was 1.5 billion parameters. Efficient training of a 7-billion parameter model already requires algorithms for training across multiple GPU because the memory requirement for derivatives and optimizers will not fit on the typical 80G Ram of the current high end GPUs.
- brrrrrm 3y agomind boggling that 7B is now considered a small model. I think it's valid, given the preeminence of 70B+ sized models. But wow, the community really just leap frogged over single digit billion parameter sizes.
- whimsicalism 3y agoI disagree - larger models are significantly more capable, likely as a function of their size, and the cost to inference them is only going to go down. Maybe for niche use cases like handling customer support requests small models will do well, but GPT4 is not going to be condensed to a few billion anytime soon imo.
- logicchains 3y agoThere are hard limits to what a "brain" of a given size can do. Expecting ensembles of 7B models to match what properly trained 180B model can do at specialised tasks is like expecting a thousand monkeys trained on a specialised task to be able to match human performance.
- samatman 3y agoIf this were true, then GPT4 would be better at chess than Stockfish. Since it's not, it isn't.
- Terr_ 3y agoWhat if there was a large inefficient model just good enough that it could generate small specialized ones and orchestrate them? Now that I've said it, I wonder if there's any evidence human brains work that way.
- taneq 3y agoBy that logic, humans should be better at chess than Stockfish, but they aren’t.
- zooch 3y agoI would like to have a more modular approach with more specialized training of the models. Currently I can only really use smaller models for micro-tasks like sentiment analysis or classification, but any type of problem solving has to be left to GPT-4.
- blueboo 3y agoBelieving this is existential for a thousand AI seed-stage startups. I hope it’s true for their sakes. Hear the other timeline from Darius Amodei. https://a16z.com/improving-ai/ https://a16z.com/improving-ai/ We may be entering a regime of multi-billion training costs on custom hardware. Open-source AI will be like open-source CPUs. Cool but the real thing even at the smallest scale comes out of monopsonized (is that a word?) inaccessible infrastructure
- lyapunova 3y agoI appreciate the conciseness of this article. Thank you for sharing. I agree. The kitchen sink data strategy appears to be fairly inefficient with current model architectures.
- sheepscreek 3y agoTiny Stories trained models and alternative implementation based on llama 2: https://github.com/karpathy/llama2.c https://github.com/karpathy/llama2.c
- sunshine-o 3y agoCould we use this to create better models for very specific medical domains for example?
- spandextwins 3y agoI wonder how google runs AI on a phone doing imaging stuff and I can’t even get it to work on a 64GB RAM machine with an Nvidia 3070 card.
- stavros 3y agoWhy doesn't it work? Try Ollama, or llama.cpp, they should work fine out of the box. They do on my 3070 just fine. Also, try a model that fits your VRAM.
- jebarker 3y agoMost the AI doing image processing is architecturally very different and smaller than LLMs.
- woadwarrior01 3y agoRunning (quantised) LLMs (encoder-decoder and decoder only transformers) on phones isn't hard. Running CNNs and encoder-only transformers, the kind used in vision on phones is even easier.
- wseqyrku 3y agoCombined with MoE, this is the microservice architecture for AI. You know it's coming.
- amelius 3y agoImho we should move AI out of the realm of CS, very quickly. The information overload is getting too much. As an analogy, it would be strange if psychology was a part of physics. Physicists would (I think) not be amused if their work literature became flooded with papers about the human psyche. Even if the underlying "hardware" is physics.
- actionfromafar 3y agoOn the other extreme, what distinguishes CS from math?
- phkahler 3y ago>> On the other extreme, what distinguishes CS from math? Doesn't that support the argument even more? CS and math share a lot more then CS and AI, yet CS and math are different disciplines.
- ethbr1 3y ago> what distinguishes CS from math? The art of application
- orra 3y agoNo? Applied maths is maths, so applying maths does not stop maths from being maths. Of course, computing is closer to pure math than applied maths. Computing is applied pure maths, perhaps.
- datameta 3y agoIs it possible to use ML to improve on training returns? Similar to neural architecture search or circuit design optimization, can we use it to maximize performance and accuracy through training set selection/optimization?
- woadwarrior01 3y agoData subset selection is an age old problem / sub-field in ML. Feature selection is a special case of this broader problem.
- btbuildem 3y agoIntuitively, this seems like a good approach -- pare down the datasets to isolate the essential, if not rudimentary, elements. It feels like a good direction would be to "grow" these models from a linguistic kernel, then supply them with a more organized / reproducible "fact bank" / memory store. We try to mimic this right now with prompt engineering, but in the end it's always just word soup. Having layers to it, like a language synthesis layer, logic layer, memory layer -- where the logic is a constraint on the language, and the memory provides the working data -- it could lead to a model that hallucinates far less.
- ZeljkoS 3y agoFor all interested, TinyStories dataset they used to train "tiny" models is on HuggingFace: https://huggingface.co/datasets/roneneldan/TinyStories https://huggingface.co/datasets/roneneldan/TinyStories
- whimsicalism 3y agoIn my opinion, models that are trained based on highly selected, curated data are going to lose out to the larger more unsupervised approaches. This is just yet another instantiation of the bitter lesson.
- deleted 3y ago[deleted]
- zzzeek 3y agoall three of those "Katie and her cat" stories are horrible. besides that none of them sound plausible, there is no substance in these stories that would make them useful to even present to children. they're just random events strung together. there's actually reasons humans write and tell stories beyond just sounding "like a story a human wrote".
- hliyan 3y agoStill, > Eldan and Li presented the same challenge to OpenAI’s GPT-2, a 1.5-billion-parameter model released in 2019. It fared far worse — before the story’s abrupt ending, the man threatens to take the girl to court, jail, the hospital, the morgue and finally the crematorium.
- deleted 3y ago[deleted]
- hcks 3y agoI know that HN loves small models because they are more accessible and hacking-friendly. However, the returns over increased capabilities vastly overwhelm the costs of running larger models. It doesn’t matter that a model requires $100,000 worth of hardware if it can automate the work of several information worker paid $200k annually.
- ospider 3y agoAssuming you don't take privacy into consideration, yes.
- visarga 3y agoI am certain investments into synthetic data will proliferate exponentially. Synthetic data can be more diverse, less toxic, and of higher average quality. But the foremost rationale we require it is to improve models. You likely recall the reversal curse - if an LLM trains "A is B" it doesn't automatically deduce "B is A". You need to explicitly append the second part to the training set to make it complete. Fragmented or incomplete information in the training set persists fragmented in the trained model. Training is blind, only inference is intelligent. Similarly, many training examples contain apparent information that conceals implicit deductions, for instance a math problem - the statement is evident and apparent but the chain of thought is implicit, concealed. What I am getting at is a general principle - LLMs need to "study" the original training data and make it comprehensive, elicit those implicit deductions, and enable LLMs to traverse the conceptual space in all directions. Study, then train. In other words, to study is to digest the raw data, to reconnect fragmented information, to generate insight, to execute the instructions and see the result, generally to unfold what is hidden. It's a matter of utilizing LLMs in generative mode to prepare the dataset before training. Microsoft has generated a 150B token synthetic dataset for Phi-1.5 and it demonstrated 5x efficiency gains. They will likely increase 100x. The promise of faster, cheaper inference and fewer errors is very alluring.
- seu 3y agoSome of their findings are really interesting, as well as their approach with the children stories, but I find it worrying that at no point the article mentions the problems inherent to the feeding AI models with the output of other AI models. > The success of the TinyStories models also suggests a broader lesson. The standard approach to compiling training data sets involves vacuuming up text from across the internet and then filtering out the garbage. Synthetic text generated by large models could offer an alternative way to assemble high-quality data sets that wouldn’t have to be so large. And they completely ignore the fact that those models are building their "synthetic stories" based on the knowledge from all that vacuuming text from across the internet, as if now we had already solved the problem of sourcing human data without the need for further vacuuming.
- visarga 3y agoRelated questions: What is the status of synthetic data generated from a model trained on copyright infringing content? How about the text generated from an open source model, trained on open or licensed data, but having copyrighted material in the prompt for reference. Does going through an AI model wash copyrights away? Does any hint of copyrighted data in the training corpus or prompt invalidate the right to publish the results? Only when they are similar enough to the copyrighted content? How about when the content itself is pretty common and not unique at all, like a solution to bubblesort?
- peter_d_sherman 3y ago>"Eldan and Li observed hints that networks with fewer layers but more neurons per layer were better at answering questions that required factual knowledge; conversely, networks with more layers and fewer neurons per layer were better at keeping track of characters and plot points from earlier in the story." That is a very interesting result! This may point to the idea that the notion of time itself (as history/historical knowledge of the state of the system at an earlier point in its evaluation/evolution/computations) being more "present" (or pronounced, "relative to results", "apparent in results") in neural networks which are deep (many layers) rather than wide (more neurons per layer). Which, if true and widely confirmed to be so by other researchers -- would be an interesting discovery indeed! Also, if true (and it's a big 'if'!), this may help researchers in other areas better understand the relationship between time and information better (i.e., Leonard Susskind, ER=EPR, etc.: https://en.wikipedia.org/wiki/Leonard_Susskind https://en.wikipedia.org/wiki/Leonard_Susskind , https://en.wikipedia.org/wiki/ER_%3D_EPR https://en.wikipedia.org/wiki/ER_%3D_EPR , https://www.youtube.com/results?search_query=leonard+susskind https://www.youtube.com/results?search_query=leonard+susskin... )
- ilaksh 3y agoUsing GPT to train smaller models is against the OpenAI TOS. At least for commercial purposes. Is it not? Do we just ignore it because it's not fair or they can't prove it? How can we get from needing a million stories to being able to use just 10 stories?
- seydor 3y agoThere should be a law prohibiting them from claiming that. Someone should challenge it in court
- woadwarrior01 3y agoCompanies are free to put whatever they wish in their TOS. Whether it holds legal water is an entirely different question and for the courts to decide. The law of the land always takes precedence over any arbitrary clauses that companies put in their TOS.
- alpark3 3y agoI think the next big innovation in LLMs (sort of like the attention mechanism) will be some method of distributing work to much smaller, specialized, and capable units, rather than having one giant network. We already see hints of this with MoE, but something entirely new wouldn't surprise me.