It absolutely can. PCIe 3x16 is at around 16GB/s speed, hence, uploading parameters at full speed would be around 50 seconds. That probably is the bottleneck fo

It absolutely can. PCIe 3x16 is at around 16GB/s speed, hence, uploading parameters at full speed would be around 50 seconds. That probably is the bottleneck for this case though. A minor issue is that the NVIDIA consumer cards (or any cards besides the new A100) doesn't support bfloat16, I am uncertain fp16 is sufficient for this model or not. Edit: It probably is, considering you only need weights to be fp16, while intermediate layers can be fp32 and reuse these memory.