Its complicated. Training on something huge like "the internet" is what gives rise to those amazing emergent properties missing in smaller models (including th

Its complicated. Training on something huge like "the internet" is what gives rise to those amazing emergent properties missing in smaller models (including the recent Pi model). And there are only so many datasets that huge. But its also indeed a waste, as Pi proves. There probably is some sweet spot (6B-40B?) for specialized, heavily focused models pre trained with high quality general data.