The hybrid approach looks powerful, however I wonder how it would compare to taking the best ~7B size model and fine tuning it directly on Wikipedia? Maybe imp

The hybrid approach looks powerful, however I wonder how it would compare to taking the best ~7B size model and fine tuning it directly on Wikipedia? Maybe impractical, but seems like it could potentially be more seamlessly integrated. Keeping in mind: - Many models already include WP in their training, at least parts of it. - The parts that are missing or a subset of what’s missing could be supplemented via finetuning. - Data is commonly compressed about 100x during training (it’s lossy) so the total size might still run on a phone. You’d then have a fully capable LLM, but enhanced to be comprehensively encyclopedic.