Basically Tiny Language Models (below 10m parameters or only one transformer block) can generate paragraphs of coherent text and reason....provided training is

Basically Tiny Language Models (below 10m parameters or only one transformer block) can generate paragraphs of coherent text and reason....provided training is limited to stories that only contain words typical 3 to 4 year olds would understand. They asked 3.5 and 4 to create a synthetic dataset for this. How they did that is pretty interesting as well. >In order to address the problem of diversity, we collected a vocabulary consisting of about 1500 basic words, which try to mimic the vocabulary of a typical 3-4 year-old child, separated into nouns, verbs, and adjectives. In each generation, 3 words are chosen randomly (one verb, one noun, and one adjective). The model is instructed to generate a story that somehow combines these random words into the story. As we argue below, this greatly increases the diversity of the dataset, forcing the stories to span the entire vocabulary a child is familiar with, and to include a rich set of ways to combine different concepts. In addition, we constructed a list of possible features a story could have (such as a dialogue, a plot twist, a bad ending or a moral value). For each story we generated a random subset of those features and prompted the model with the extra requirement for the story to have these features