From the limited amount I understand training an LLM solely on Wikipedia would lead to a lower quality model as there's parts of language and intelligent commun

From the limited amount I understand training an LLM solely on Wikipedia would lead to a lower quality model as there's parts of language and intelligent communication missing (but that's my intuition). Instead a LORA could be trained on Wikipedia to have it's responses more match the style of wikipedia. Or you could feed a "regular" LLM the whole of Wikipedia and it'd answer questions.