4 ms·
> You could run it on a cluster of nodes Not sure this is a MBP either.
by nl 6mo ago
> You could run it on a cluster of nodes
Not sure this is a MBP either.
- bigyabai 6mo agoNot even a cluster of Mac Pros could run a dense 5T parameter model with RDMA, to my knowledge.
- zozbot234 6mo agoSOTA models are reportedly MoE, not dense.
- bigyabai 6mo agoA 5T MoE model is still bottlenecked by streaming weights from SSD, in addition to compute bottlenecks during prefill and decode.
- zozbot234 6mo agoTrue but a cluster built on pipeline parallelism can naturally stream from multiple SSD's in parallel. That probably makes offload somewhat more effective. And you also have RAM caching available as a natural possibility.
- bigyabai 6mo agoYou won't be RAM caching much of anything with experts that are 220b parameters worth of layers.