4 ms·

If "LLMs" includes reasoning models, then you're already wrong in your first paragraph: "something that is just MatMul with interspersed nonlinearities."