how do you handle models that are bigger than 16GB on that? my m1 max 64gb whips through dolphin-llama3:70b just fine, that's around 42gb usage. I get partial

how do you handle models that are bigger than 16GB on that? my m1 max 64gb whips through dolphin-llama3:70b just fine, that's around 42gb usage. I get partial offload on dolphin-mixtral:8x7b, that's significantly slower but still reasonably usable, whereas even the quantized models chug on my work m1 max 32gb. everything i've seen is that once you seriously cross the threshold of VRAM usage/gpu offload, the performance collapses. and sure, if you stay within that the 4090 (or even 4070 mobile) is a lot faster, but the most popular current families of models don't generally fit on a 4070 or even 4090. on a 41.5gb model, would a 4070 mobile/4060 ti 16GB really beat a 64gb m1 max? it seems doubtful but idk. if there's viable ways to split models across multi-gpus i'm all ears, but afaik there's not (outside NVLink). Other than buying or renting A100s, the only serious option would be bergamo with avx-512 and a bunch of memory, much slower but avx-512 does have inference instructions now, and that gets you into the class of memory you (currently) need for things like snowflake-arctic-instruct:128x3B. (and of course training is a whole different kettle of fish, but everyone knows that at this point.)