We use some of these techniques at Groq to make the world's fastest LLM appliance! You can see for yourself how fast it is at https://groq.com/ https://groq.com

We use some of these techniques at Groq to make the world's fastest LLM appliance! You can see for yourself how fast it is at https://groq.com/ https://groq.com/ (click "Launch demo" in the top right). The model is Llama 2 70B. Our hardware is an "LPU" (language processing unit). The silicon has a very regular/uniform/homogeneous design with compute located close to memory, which allows us to get a much lower latency for language models than a GPU can. A lot of the smarts is in the compiler toolchain, which can fit high-level model descriptions (say Torch, ONNX) to the hardware without the need for handwritten kernels. I work on the assembler part of the toolchain, which is written in Haskell. EDIT: To the downvoters, I guarantee that once you've tried the demo you'll be so surprised at how fast it is that you'll change your vote.