LLMs silently accumulate errors with the length of the sequence predicted ("label bias" is the term that should be in that book, but not in the context of LLMs)

LLMs silently accumulate errors with the length of the sequence predicted ("label bias" is the term that should be in that book, but not in the context of LLMs). The problem is exacerbated by a very big token dictionary. Every new token you generate is independently sampled (from tokens outside of the context size). If you have any task that requires joint distribution modeling LLM will fail and HMMs/CRFs will succeed. Of course, it depends when the problem will manifest and I do not know that, but skipping the whole book is not recommended. For example, there were approaches for many language tasks in that book that used CNNs (instead of something more principled) and were extremely successful (despite the lack of joint modeling). Who knows when this accumulation of errors becomes measurable. As long as your generation of tokens fits inside the context size, you're modeling everything jointly. But I guess you need to be aware that when you drop the furthest tokens to generate next ones, the performance can drastically drop (multiplicative accumulation of errors).