I'm on M4 Max, so I don't have split parameters to tune. I can fit full context, I can fit the full model in unified memory, my only concern has been the very m

I'm on M4 Max, so I don't have split parameters to tune. I can fit full context, I can fit the full model in unified memory, my only concern has been the very meager memory bandwidth of these chips. The only thing that can seem to feed the GPU is very large prefill batches, on the order of many thousands of tokens per batch, and that's with only a 9B model at 3 bpw.