It’s enough for pre training to later tackle specific NLP tasks. To get something interesting you would have to generate an instruct dataset from it. It would

It’s enough for pre training to later tackle specific NLP tasks. To get something interesting you would have to generate an instruct dataset from it. It would have to cover a diverse range of tasks. The completions themselves do not make LLMs manifest knowledge and reasoning, a large and diverse instruct dataset does.