> OpenAI might be using speculative decoding on GPT-4's inference. (not sure 100%) The idea is to use a smaller faster model to decode several tokens in advance

> OpenAI might be using speculative decoding on GPT-4's inference. (not sure 100%) The idea is to use a smaller faster model to decode several tokens in advance, and then feeds them into a large oracle model as a single batch. If the small model was right about its predictions – the larger model agrees and we can decode several tokens in a single batch. But if the larger model rejects the tokens predicted by the draft model then the rest of the batch is discarded. And we continue with the larger model. The conspiracy theory that the new GPT-4 quality had been deteriorated might be simply because they are letting the oracle model accept lower probability sequences from the speculative decoding model. [0] Well if the recent leaks are true it's because they're running a smaller, likely heavily quantized model to do the bulk of the generation now, with the full float one only stepping in on occasion to save GPU time. The interesting thing is that it seems that 3.5-turbo has slightly improved in recent months, while performance of 4 has deteriorated, it's like they're converging them to a middle ground or something. [0] https://archive.ph/2RQ8X https://archive.ph/2RQ8X