5 ms·

Yes? The original LLaMA-65B on fp16 weight is 130G. There are no NVIDIA GPU with 130G memory. As for why they don't use int8/int4: would you try to optimize y

by rfoo 4y ago

Yes? The original LLaMA-65B on fp16 weight is 130G. There are no NVIDIA GPU with 130G memory. As for why they don't use int8/int4: would you try to optimize your code before you confirmed it can actually run?

alchemist1e9 4y ago
Got it, so looking ahead to building hardware that is useful for inference would you agree with these considerations: - max PCI lanes for more GPUs. AMD cpus and boards probably better - PCIe 5 nvme. ideally enough to saturate bus - max GPUs with max memory. memory more important than FLOPs probably. for a given budget one tries to get as much as possible of above. How long before a consumer component machine for less than $10K can do inference for a gpt 3.5 turbo quality model?
- rfoo 4y ago
  PCIe lanes: it's not that important since the data exchange between host and device during inference is pretty minimal. For model parallel / tensor parallel you want fast interconnect between GPUs, the ages old tech for this is PCIe switch, the fashion ones are NVLink, neither are consumer hardware so don't bother. PCIe 5 NVMe: That's completely irrelevant for inference, as you really want to keep all weights on the GPU VRAM. It only speed up the initial weight load from host to device. max GPUs with max memory: Yeah. For personal use (batch size = 1) VRAM size AND VRAM bandwidth is more important than FLOPs. See https://orenleung.com/is-chatgpt-175-billion-parameters-technical-analysis https://orenleung.com/is-chatgpt-175-billion-parameters-tech.... For production it's a tradeoff. > How long before a consumer component machine for less than $10K can do inference for a gpt 3.5 turbo quality model? $10k is a lot. You can run LLaMA-65B on 2x4090, won't be fast enough for production though. But given that people happily run LLaMA-13B on their MacBooks I guess we are okay with generation being slow. And personally I don't think gpt-3.5-turbo is that big, it should be around or even smaller than LLaMA-65B.
- MacsHeadroom 4y ago
  >How long before a consumer component machine for less than $10K can do inference for a gpt 3.5 turbo quality model? Uh two months ago? LLaMA-13B is GPT 3.5 Turbo quality and runs locally on my Android phone's CPU at acceptable speeds. You can even run LLaMA-65B (which far surpasses GPT 3.5) faster than GPT 3.5 Turbo with two $200 24GB Nvidia Tesla P40 cards, since in 4bit the model is only 39GB with no output quality loss. Total cost $400 plus some junker used PC with two spare PCIe 4 x16 lanes.
  - alchemist1e9 4y ago
    Is that really true! Can LLaMA-65B output code as good quality as GPT 3.5 Turbo? Are P40s only $200! If your information is accurate that is incredible and great tip. I’ll start on building such a rig asap.
- fovc 4y ago
  How much are 2x 3090 + the rig to run them? I think that’s enough to run a 4 bit quantized llama 65B

by rfoo 4y ago

alchemist1e9 4y ago

Got it, so looking ahead to building hardware that is useful for inference would you agree with these considerations: - max PCI lanes for more GPUs. AMD cpus and boards probably better - PCIe 5 nvme. ideally enough to saturate bus - max GPUs with max memory. memory more important than FLOPs probably. for a given budget one tries to get as much as possible of above. How long before a consumer component machine for less than $10K can do inference for a gpt 3.5 turbo quality model?

rfoo 4y ago
PCIe lanes: it's not that important since the data exchange between host and device during inference is pretty minimal. For model parallel / tensor parallel you want fast interconnect between GPUs, the ages old tech for this is PCIe switch, the fashion ones are NVLink, neither are consumer hardware so don't bother. PCIe 5 NVMe: That's completely irrelevant for inference, as you really want to keep all weights on the GPU VRAM. It only speed up the initial weight load from host to device. max GPUs with max memory: Yeah. For personal use (batch size = 1) VRAM size AND VRAM bandwidth is more important than FLOPs. See https://orenleung.com/is-chatgpt-175-billion-parameters-technical-analysis https://orenleung.com/is-chatgpt-175-billion-parameters-tech.... For production it's a tradeoff. > How long before a consumer component machine for less than $10K can do inference for a gpt 3.5 turbo quality model? $10k is a lot. You can run LLaMA-65B on 2x4090, won't be fast enough for production though. But given that people happily run LLaMA-13B on their MacBooks I guess we are okay with generation being slow. And personally I don't think gpt-3.5-turbo is that big, it should be around or even smaller than LLaMA-65B.
MacsHeadroom 4y ago
>How long before a consumer component machine for less than $10K can do inference for a gpt 3.5 turbo quality model? Uh two months ago? LLaMA-13B is GPT 3.5 Turbo quality and runs locally on my Android phone's CPU at acceptable speeds. You can even run LLaMA-65B (which far surpasses GPT 3.5) faster than GPT 3.5 Turbo with two $200 24GB Nvidia Tesla P40 cards, since in 4bit the model is only 39GB with no output quality loss. Total cost $400 plus some junker used PC with two spare PCIe 4 x16 lanes.
- alchemist1e9 4y ago
  Is that really true! Can LLaMA-65B output code as good quality as GPT 3.5 Turbo? Are P40s only $200! If your information is accurate that is incredible and great tip. I’ll start on building such a rig asap.
fovc 4y ago
How much are 2x 3090 + the rig to run them? I think that’s enough to run a 4 bit quantized llama 65B