5 ms·

I don't know. It has 175 billion parameters, thus, about 500GiB for floating point parameters. If these are bfloat16, it is mere 300GiB. We know CPU is about 50

by liuliu 6y ago

I don't know. It has 175 billion parameters, thus, about 500GiB for floating point parameters. If these are bfloat16, it is mere 300GiB. We know CPU is about 50 times slower than GPU for transformer models, hence, we are looking at 4 to 8 minutes per inference (parameter loading on demand from SSD takes 200 seconds or so, and probably the bottleneck here). If the parameters can be loaded into memory (seems you have to be on a >= 8-channel machine with unbuffered RAM or with buffered RAM), it will be probably 1 to 2 minutes per inference. Still in the realm of doing inference in homelab territory (barely).

MiroF 6y ago
> We know CPU is about 50 times slower than GPU for transformer models Where's this heuristic from? Seems handy if true.
- liuliu 6y ago
  It is from my own experience. But it looks like inference is a bit faster on CPU: https://medium.com/huggingface/benchmarking-transformers-pytorch-and-tensorflow-e2917fb891c2 https://medium.com/huggingface/benchmarking-transformers-pyt...
  - hansvm 6y ago
    Those figures make sense as a rough ballpark -- max FLOPS estimates including impacts from FMA and whatnot roughly correspond to the throughput we would expect for a compute-bound linear algebra problem, and current top of the line consumer grade GPUs are 15-30x better in that metric than equivalent CPUs.
hansvm 6y ago
Aside from having to slice and dice things a bit to fit in the accelerator's memory and incurring a higher memory bandwidth cost in doing so, is there any reason a home lab's GPU couldn't be used for GPT-3?
- liuliu 6y ago
  It absolutely can. PCIe 3x16 is at around 16GB/s speed, hence, uploading parameters at full speed would be around 50 seconds. That probably is the bottleneck for this case though. A minor issue is that the NVIDIA consumer cards (or any cards besides the new A100) doesn't support bfloat16, I am uncertain fp16 is sufficient for this model or not. Edit: It probably is, considering you only need weights to be fp16, while intermediate layers can be fp32 and reuse these memory.

by liuliu 6y ago

MiroF 6y ago

> We know CPU is about 50 times slower than GPU for transformer models Where's this heuristic from? Seems handy if true.

liuliu 6y ago
It is from my own experience. But it looks like inference is a bit faster on CPU: https://medium.com/huggingface/benchmarking-transformers-pytorch-and-tensorflow-e2917fb891c2 https://medium.com/huggingface/benchmarking-transformers-pyt...
- hansvm 6y ago
  Those figures make sense as a rough ballpark -- max FLOPS estimates including impacts from FMA and whatnot roughly correspond to the throughput we would expect for a compute-bound linear algebra problem, and current top of the line consumer grade GPUs are 15-30x better in that metric than equivalent CPUs.

hansvm 6y ago

Aside from having to slice and dice things a bit to fit in the accelerator's memory and incurring a higher memory bandwidth cost in doing so, is there any reason a home lab's GPU couldn't be used for GPT-3?

liuliu 6y ago
It absolutely can. PCIe 3x16 is at around 16GB/s speed, hence, uploading parameters at full speed would be around 50 seconds. That probably is the bottleneck for this case though. A minor issue is that the NVIDIA consumer cards (or any cards besides the new A100) doesn't support bfloat16, I am uncertain fp16 is sufficient for this model or not. Edit: It probably is, considering you only need weights to be fp16, while intermediate layers can be fp32 and reuse these memory.