3 ms·
Yea unlike other options this is actually scalable. The question isn't if 1 can outperform the Mac Studio but if 3-4 linked together can.
by kcb 2y ago
Yea unlike other options this is actually scalable. The question isn't if 1 can outperform the Mac Studio but if 3-4 linked together can.
- Dylan16807 2y agoWhat kind of scaling do you have in mind there? My naive analysis is: A high end Mac should be able to run each layer of an AI task about twice as fast because of the memory bandwidth. And the data going between layers is tiny enough to run over thunderbolt or even normal ethernet. Is there an AI use case that prefers 250GB/s memory bandwidth plus 25GB/s interconnect over 500GB/s memory and 2GB/s interconnect? Are there other major use cases that prefer it?
- fancyfredbot 2y agoUsually the reason you'd want the network bandwidth would be for distributed training. For inference you can probably get by with 2GB/s assuming you can split the layers up nicely. The interconnect can be a bottleneck for inference but only for networks with loads of activations and large batch sizes, or if you are doing tensor level parallelism.