One of the weaknesses of transformers is that inference is inherently serial, one token at a time, and the entire model needs to be read once per token. This me

One of the weaknesses of transformers is that inference is inherently serial, one token at a time, and the entire model needs to be read once per token. This means that inference (after prompt processing) is bounded by memory bandwidth instead of compute. That said, local solutions always tend to be lower latency than cloud ones just because you get to skip the network.