It is way cheaper to run inference on commodity hardware, because you don't have to keep track of the activations on all the previous tokens (only the 3 previou

It is way cheaper to run inference on commodity hardware, because you don't have to keep track of the activations on all the previous tokens (only the 3 previous ones IIRC).