Memory bandwidth is my constraint too (M4 Max) but prefill and single-token decode don't run at the same time. It's best to use batched prefill so you can benef

Memory bandwidth is my constraint too (M4 Max) but prefill and single-token decode don't run at the same time. It's best to use batched prefill so you can benefit from processing multiple tokens with a single pass through the model weights.