4 ms·

Well sure, but in other GPU tasks, like Raytracing, the difference between these GPUs is far more pronounced. And AMD has passable Raytracing units (NVidias ar

Well sure, but in other GPU tasks, like Raytracing, the difference between these GPUs is far more pronounced. And AMD has passable Raytracing units (NVidias are better but the difference is bigger than these LLM results). If RAM is the main bottleneck then CPUs should be on the table.

webmaven 2y ago
RAM is (often) the bottleneck for highly parallel GPUs, but not for CPUs. Though the distinction between the two categories is blurring.
- ryao 2y ago
  Memory bandwidth is the bottleneck for both when running GEMV, which is the main operation used by token generation in inference. It has always been this way.
IX-103 2y ago
> If RAM is the main bottleneck then CPUs should be on the table That's certainly not the case. The graphics memory model is very different from the CPU memory model. Graphics memory is explicitly designed for multiple simultaneous reads (spread across several different buses) at the cost of generality (only portions of memory may be available on each bus) and speed (the extra complexity means reads are slower). This makes then fast at doing simple operations on a large amount of data. CPU memory only has one bus, so only a single read can happen at a time (a cache line read), but can happen relatively quickly. So CPUs are better for workloads with high memory locality and frequent reuse of memory locations (as is common in procedural programs).
- dragontamer 2y ago
  > CPU memory only has one bus If people are paying $15,000 or more per GPU, then I can choose $15,000 CPUs like EPYC that have 12-channels or dual-socket 24-channel RAM. Even desktop CPUs are dual-channel at a minimum, and arguably DDR5 is closer to 2 or 4 buses per channel. Now yes, GPU RAM can be faster, but guess what? https://www.tomshardware.com/pc-components/cpus/amd-crafts-custom-epyc-cpu-for-microsoft-azure-with-hbm3-memory-cpu-with-88-zen-4-cores-and-450gb-of-hbm3-may-be-repurposed-mi300c-four-chips-hit-7-tb-s https://www.tomshardware.com/pc-components/cpus/amd-crafts-c... GPUs are about extremely parallel performance, above and beyond what traditional single-threaded (or limited-SIMD) CPUs can do. But if you're waiting on RAM anyway?? Then the compute-method doesn't matter. Its all about RAM.
- ryao 2y ago
  Where are these GPUs with multiple buses? I only know of GPUs with wide buses.
schmidtleonard 2y ago
CPUs have pitiful RAM bandwidth compared to GPUs. The speeds aren't so different but GPU RAM busses are wiiiiiiiide.
- teleforce 2y ago
  Compute Express Link (CXL) should mostly solve limited RAM with CPU: 1) Compute Express Link (CXL): https://en.wikipedia.org/wiki/Compute_Express_Link https://en.wikipedia.org/wiki/Compute_Express_Link PCIe vs. CXL for Memory and Storage: https://news.ycombinator.com/item?id=38125885 https://news.ycombinator.com/item?id=38125885
  - schmidtleonard 2y ago
    Gigabytes per second? What is this, bandwidth for ants? My years old pleb tier non-HBM GPU has more than 4 times the bandwidth you would get from a PCIe Gen 7 x16 link, which doesn't even officially exist yet.
    - teleforce 2y ago
      Yes CXL will soon benefit from PCIe Gen 7 x16 with expected 64GB/s in 2025 and the non-HBM bandwidth I/O alternative is increasing rapidly by the day. For most inferences of near real-time LLM it will be feasible. For majority of SME companies and other DIY users (humans or ants) with their localized LLM should not be any issues [1],[2]. In addition new techniques for more efficient LLM are being discover to reduce the memory consumption [3]. [1] Forget ChatGPT: why researchers now run small AIs on their laptops: https://news.ycombinator.com/item?id=41609393 https://news.ycombinator.com/item?id=41609393 [2] Welcome to LLMflation – LLM inference cost is going down fast: https://a16z.com/llmflation-llm-inference-cost/ https://a16z.com/llmflation-llm-inference-cost/ [3] New LLM optimization technique slashes memory costs up to 75%: https://news.ycombinator.com/item?id=42411409 https://news.ycombinator.com/item?id=42411409
      - schmidtleonard 2y ago
ryao 2y ago
The main bottleneck is memory bandwidth. CPUs have less memory bandwidth than GPUs.

webmaven 2y ago

RAM is (often) the bottleneck for highly parallel GPUs, but not for CPUs. Though the distinction between the two categories is blurring.

ryao 2y ago
Memory bandwidth is the bottleneck for both when running GEMV, which is the main operation used by token generation in inference. It has always been this way.

IX-103 2y ago

> If RAM is the main bottleneck then CPUs should be on the table That's certainly not the case. The graphics memory model is very different from the CPU memory model. Graphics memory is explicitly designed for multiple simultaneous reads (spread across several different buses) at the cost of generality (only portions of memory may be available on each bus) and speed (the extra complexity means reads are slower). This makes then fast at doing simple operations on a large amount of data. CPU memory only has one bus, so only a single read can happen at a time (a cache line read), but can happen relatively quickly. So CPUs are better for workloads with high memory locality and frequent reuse of memory locations (as is common in procedural programs).

dragontamer 2y ago
> CPU memory only has one bus If people are paying $15,000 or more per GPU, then I can choose $15,000 CPUs like EPYC that have 12-channels or dual-socket 24-channel RAM. Even desktop CPUs are dual-channel at a minimum, and arguably DDR5 is closer to 2 or 4 buses per channel. Now yes, GPU RAM can be faster, but guess what? https://www.tomshardware.com/pc-components/cpus/amd-crafts-custom-epyc-cpu-for-microsoft-azure-with-hbm3-memory-cpu-with-88-zen-4-cores-and-450gb-of-hbm3-may-be-repurposed-mi300c-four-chips-hit-7-tb-s https://www.tomshardware.com/pc-components/cpus/amd-crafts-c... GPUs are about extremely parallel performance, above and beyond what traditional single-threaded (or limited-SIMD) CPUs can do. But if you're waiting on RAM anyway?? Then the compute-method doesn't matter. Its all about RAM.
ryao 2y ago
Where are these GPUs with multiple buses? I only know of GPUs with wide buses.

schmidtleonard 2y ago

CPUs have pitiful RAM bandwidth compared to GPUs. The speeds aren't so different but GPU RAM busses are wiiiiiiiide.

teleforce 2y ago
Compute Express Link (CXL) should mostly solve limited RAM with CPU: 1) Compute Express Link (CXL): https://en.wikipedia.org/wiki/Compute_Express_Link https://en.wikipedia.org/wiki/Compute_Express_Link PCIe vs. CXL for Memory and Storage: https://news.ycombinator.com/item?id=38125885 https://news.ycombinator.com/item?id=38125885
- schmidtleonard 2y ago
  Gigabytes per second? What is this, bandwidth for ants? My years old pleb tier non-HBM GPU has more than 4 times the bandwidth you would get from a PCIe Gen 7 x16 link, which doesn't even officially exist yet.
  - teleforce 2y ago
    Yes CXL will soon benefit from PCIe Gen 7 x16 with expected 64GB/s in 2025 and the non-HBM bandwidth I/O alternative is increasing rapidly by the day. For most inferences of near real-time LLM it will be feasible. For majority of SME companies and other DIY users (humans or ants) with their localized LLM should not be any issues [1],[2]. In addition new techniques for more efficient LLM are being discover to reduce the memory consumption [3]. [1] Forget ChatGPT: why researchers now run small AIs on their laptops: https://news.ycombinator.com/item?id=41609393 https://news.ycombinator.com/item?id=41609393 [2] Welcome to LLMflation – LLM inference cost is going down fast: https://a16z.com/llmflation-llm-inference-cost/ https://a16z.com/llmflation-llm-inference-cost/ [3] New LLM optimization technique slashes memory costs up to 75%: https://news.ycombinator.com/item?id=42411409 https://news.ycombinator.com/item?id=42411409
    - schmidtleonard 2y ago

ryao 2y ago

The main bottleneck is memory bandwidth. CPUs have less memory bandwidth than GPUs.