It really is not. The 70b model just does not fit into RTX 4090. As seen in the same video for models that fit 4090 is about 2 times faster. It really is. The

It really is not. The 70b model just does not fit into RTX 4090. As seen in the same video for models that fit 4090 is about 2 times faster. It really is. The LLM is benefiting from the high amount of high bandwidth RAM, which is a benefit of UMA in Macs. To get 128GB or more RAM from Nvidia, you'd have to combine two H100 GPUs, which retail for $30k+ each. Incidentally you can get ~ the same memory bandwidth as M3 Max with newly released 7945WX Threadripper for $1400 + the cost of DDR5. And you will be able to scale the amount of RAM as much as you want for bigger models (currently up to 2TB) by just buying new RAM. The CPU would have less compute power. I'd be curious to see the test results of an Epyc 128-core CPU with 12 memory channels though.