4 ms·
Aside from having to slice and dice things a bit to fit in the accelerator's memory and incurring a higher memory bandwidth cost in doing so, is there any reaso
by hansvm 6y ago
Aside from having to slice and dice things a bit to fit in the accelerator's memory and incurring a higher memory bandwidth cost in doing so, is there any reason a home lab's GPU couldn't be used for GPT-3?
- liuliu 6y agoIt absolutely can. PCIe 3x16 is at around 16GB/s speed, hence, uploading parameters at full speed would be around 50 seconds. That probably is the bottleneck for this case though. A minor issue is that the NVIDIA consumer cards (or any cards besides the new A100) doesn't support bfloat16, I am uncertain fp16 is sufficient for this model or not. Edit: It probably is, considering you only need weights to be fp16, while intermediate layers can be fp32 and reuse these memory.