3 ms·
How Small Can Language Models Be and Still Speak Coherent English?
- LoganDark 4y agoWhere is the code / model they trained?
- famouswaffles 4y agohttps://huggingface.co/papers/2305.07759 https://huggingface.co/papers/2305.07759
- LoganDark 4y agoThank you! <3
- famouswaffles 4y agoBasically Tiny Language Models (below 10m parameters or only one transformer block) can generate paragraphs of coherent text and reason....provided training is limited to stories that only contain words typical 3 to 4 year olds would understand. They asked 3.5 and 4 to create a synthetic dataset for this. How they did that is pretty interesting as well. >In order to address the problem of diversity, we collected a vocabulary consisting of about 1500 basic words, which try to mimic the vocabulary of a typical 3-4 year-old child, separated into nouns, verbs, and adjectives. In each generation, 3 words are chosen randomly (one verb, one noun, and one adjective). The model is instructed to generate a story that somehow combines these random words into the story. As we argue below, this greatly increases the diversity of the dataset, forcing the stories to span the entire vocabulary a child is familiar with, and to include a rich set of ways to combine different concepts. In addition, we constructed a list of possible features a story could have (such as a dialogue, a plot twist, a bad ending or a moral value). For each story we generated a random subset of those features and prompted the model with the extra requirement for the story to have these features
- ttpphd 4y agoUnless the model is hooked up to a text-to-speech module it doesn't speak any English at all. Signed, A frustrated auditory scientist.
- robinduckett 4y agoSimilar energy to when someone who practises Archery watches historical fiction depicting someone telling bowmen to “fire!”
- leobg 4y agoVery interesting! So what I’m wondering is this: Can you train a small model on this “child dataset“. And then, once it speaks fluently in this simple language, add new layers on top of it, and, freezing the rest of the model, train those on a corpus of more complex material.
- IIAOPSW 4y agoWhy use many training when few training will do? -the machine