4 ms·
From https://github.com/moyix/fauxpilot https://github.com/moyix/fauxpilot > if you have two NVIDIA RTX 3080 GPUs, you should be able to run the 6B model by pu
by sva_ 4y ago
From https://github.com/moyix/fauxpilot https://github.com/moyix/fauxpilot
> if you have two NVIDIA RTX 3080 GPUs, you should be able to run the 6B model by putting half on each GPU.
Thats quite excessive, but nice that it can be ran locally.
> Note that the VRAM requirements listed by setup.sh are total -- if you have multiple GPUs, you can split the model across them.
This might not work with the upcoming RTX 40x0 GPUs unless they have enough memory, as they abolished NVLink in that series.
Edit: I'm now seeing there are smaller models too. Going to give it a try tomorrow.
- deleted 4y ago[deleted]
- woodson 4y agoYou don’t need NVlink for that to work.
- sva_ 4y agoApologies for my ignorance, but don't you need NVLink to pool the memory of the cards together?
- moyix 4y agoNope. The way it works is FasterTransformer splits the model across the two GPUs and runs both halves in parallel. It periodically has to sync the results from each half, so it will go faster if you have a high-bandwidth link between the GPUs like NVLink, but it will work just fine even if they have to communicate over PCIe peer-to-peer or even communicating via the CPU.