Context length is limited by the length of articles in the training data. You couldn’t use infinite context lengths unless the LLM had been trained on infinity

Context length is limited by the length of articles in the training data. You couldn’t use infinite context lengths unless the LLM had been trained on infinity long articles. Context lengths aren’t arbitrary limits - they reflect the ability (and limits) of the model to actually reason about lengthy topics. If you want longer context lengths you’ll need more training data, larger models, and a lot more money.