Yes, the problem with such long context lengths is sparsity - you still don't have enough data points to make representative many-gram distributions. This was e

Yes, the problem with such long context lengths is sparsity - you still don't have enough data points to make representative many-gram distributions. This was exactly the motivation for Bengio and others to propose neural language models in 2003: https://dl.acm.org/doi/10.5555/944919.944966 https://dl.acm.org/doi/10.5555/944919.944966 I recommend everyone interested in neural language models and/or wondering why we can't scale up Markov models by just using longer ngrams to read this paper. It's pretty accessible and explains how we got to where we are now in 2023.