Is bigger always better? Surely "bigger is better" is at odds with "garbage in, garbage out". But I am not an LLM modeler. IOW can LLMs be improved by improving

Is bigger always better? Surely "bigger is better" is at odds with "garbage in, garbage out". But I am not an LLM modeler. IOW can LLMs be improved by improving the quality and/or even *limiting the size of their "corpus"? Could we instead use, as corpus, a select body of knowledge primarily based on Western sources such as those texts a student might read as part of an excellent liberal arts education? Further, would it not be possible to limit the corpus text (and consequently later, queries) to grammatically correct language? That is, must we include all the "errors" of common speech in a corpus, such as occur especially in fiction? Indeed, could we not exclude fictional works altogether, esp. if the model is to be used for scientific work? I'd be curious to see an LLM trained on "The Great Books" plus a college course plan of textbooks. But again, I am not an LLM modeler. Anyone experimenting with such "Small Language Models(SLM)"?