I guess the text generation might be using some kind of beam search as a final layer (or some other enumerative search procedure). A trivial computational inten

I guess the text generation might be using some kind of beam search as a final layer (or some other enumerative search procedure). A trivial computational intensity reduction trick would be to reduce search size and thus reduce the quality of the output, without even changing the model. So I can imagine they reduced the compute per chat over time like this. I semi agree with the observation, though I'm not sure if that's just a hype recovery bias from my side. For example, in the early days I would just marvel at the output looking right. Later I started running the output and noting it is often quite wrong.