3 ms·
Quantized to 4 bits you'll only need ~200GB! 5 4090s should cover it.
by lolinder 2y ago
Quantized to 4 bits you'll only need ~200GB! 5 4090s should cover it.
- angoragoats 2y agoYou'll probably need 9 or more. 4090s have 24GB each.
- woodson 2y agoI wonder if AutoAWQ works out of the box, given no architectural changes (?). That would be most straightforward together with vLLM for serving.
- pat2man 2y agoTwo 128gb Mac studios networked via thunderbolt 4?
- Teknomancer 2y agoThis is actually a promising endeavor. Id love to see someone try that.
- angoragoats 2y agoThere's already at least one project that attempts this: https://github.com/exo-explore/exo https://github.com/exo-explore/exo
- downvotetruth 2y agoIf an implementation had NVidia's Heterogeneous Memory Management implemented, then 192 GB RAM DDR5 + GPU VRAM would seem to be close.