3 ms·
Strangely, it is super fast on my 16 Plus, but with longer messages it can slow down a LOT, and not because of thermal throttling. I wish I could see some diagn
by mfro 6mo ago
Strangely, it is super fast on my 16 Plus, but with longer messages it can slow down a LOT, and not because of thermal throttling. I wish I could see some diagnostic data.
- steve-atx-7600 6mo agoInference from an LLM is O(tokens^2)
- halJordan 6mo agoOnly in the naive implementations of attention