I think if you go check the n-gram (Markov chain) models which were trained by IBM and Google and others (eg. 'unreasonable effectiveness of data'), you'll find

I think if you go check the n-gram (Markov chain) models which were trained by IBM and Google and others (eg. 'unreasonable effectiveness of data'), you'll find them remarkably chonky sample-wise. (This was possible because processing data for fitting n-grams was cheaper than training any neural net on them, so even in the '80s or '90s you could fit n-grams to billions of tokens.) But their scaling was much, much worse. An example: https://gwern.net/doc/ai/scaling/2020-chrisdyer-aacl2020-machinetranslationscaling-ngramsvsrnns.jpg https://gwern.net/doc/ai/scaling/2020-chrisdyer-aacl2020-mac... Imagine how far to the right you would have to draw out that n-gram curve before it reached the seq2seq RNN trained on a fixed dataset size - and where that RNN is considered garbage by current standards. (Note the x-axis is in "millions of tokens".)