3 ms·
What kind of scaling do you have in mind there? My naive analysis is: A high end Mac should be able to run each layer of an AI task about twice as fast because
by Dylan16807 2y ago
What kind of scaling do you have in mind there?
My naive analysis is: A high end Mac should be able to run each layer of an AI task about twice as fast because of the memory bandwidth. And the data going between layers is tiny enough to run over thunderbolt or even normal ethernet.
Is there an AI use case that prefers 250GB/s memory bandwidth plus 25GB/s interconnect over 500GB/s memory and 2GB/s interconnect? Are there other major use cases that prefer it?
- fancyfredbot 2y agoUsually the reason you'd want the network bandwidth would be for distributed training. For inference you can probably get by with 2GB/s assuming you can split the layers up nicely. The interconnect can be a bottleneck for inference but only for networks with loads of activations and large batch sizes, or if you are doing tensor level parallelism.