10 ms·
I wonder how many people are scrambling to set this up on their startup infra. 6x24GB NVRAM on 6 GPUs linked with NVSwitch is a little pricey, but totally doab
by q1w2 4y ago
I wonder how many people are scrambling to set this up on their startup infra.
6x24GB NVRAM on 6 GPUs linked with NVSwitch is a little pricey, but totally doable.
- exo-pla-net 4y agoHow pricey would you estimate?
- deleted 4y ago[deleted]
- Tepix 4y agoIf you want to do it the cheap way by buying used stuff, the most expensive parts are: - $2000 for a Threadripper 3xx5WX with a socket sWRX8 mainboard - $5000 for 6x RTX 3090 - $350 for two 1500W PSUs - $700 for 256GB RAM You will also need PCIe extenders and perhaps some watercooling. And find a suitable case. The 2-card NVLink bridges are between $100 and $300 each (you nay want 3). All in all i think less than $10k.
- touristtam 4y agoWould rather put that in AWS.
- Tepix 4y agoAWS is probably among the priciest options. Privacy costs money.
- deleted 4y ago[deleted]
- arthurcolle 4y agoI got it running using Colab Pro+ (immediately got a V100 40GB VRAM GPU) - the 7B model works with batch size of 8 and a max seq len of 1024
- koheripbal 4y agoSure, but the real value here is the 65B. Can you have multiple GPUs on colab?
- arthurcolle 4y agoI can't even get the 13B on colab to do inference with a very small sequence length.