There was a recent post here linking to the blog “The Secret Sauce behind 100K context window in LLMs: all tricks in one place” : https://blog.gopenai.com/how-t

There was a recent post here linking to the blog “The Secret Sauce behind 100K context window in LLMs: all tricks in one place” : https://blog.gopenai.com/how-to-speed-up-llms-and-use-100k-context-window-all-tricks-in-one-place-ffd40577b4c https://blog.gopenai.com/how-to-speed-up-llms-and-use-100k-c... My impression is that combining all of those, plus all of the post training quantisation and sparsity tricks into a new model with more training compute than GPT would yield an amazing improvement. Especially the price/performance would be expected to dramatically improve. The current models are very wasteful during inference, there’s easily a factor of ten improvement available there.