3 ms·

5k for a computer with 128gb doesn’t seem like a great deal. You can get that much ddr5 today for like $400 sometimes less than that on sale.

theGnuMe 3y ago
yeah but you can't get an nvidia gpu with at least 128gb (H200) for less than $40k
- kjkjadksj 3y ago
  You can set how much of your system ram your gpu gets in windows and linux
  - xcv123 3y ago
    Useless for LLMs. Limited system memory bandwidth kills it. GPU will be mostly idle waiting for memory access. The H200 has 4.8 TB/s bandwidth. System memory is below 90 GB/s
    - kjkjadksj 3y ago
      If you have a use case for an h200 class gpu you are probably making enough money off of it to not care about $40k and the 900gb/s bandwidth mentioned in another comment would seem absolutely pedestrian in comparison. If these things were really that compelling of a performance per dollar ratio you wouldn’t be able to buy them. They’d be sold out because people would be filling datacenters with them.
      - xcv123 3y ago
        M2 Ultra sits between a lower end Intel setup and an H200 for this purpose. I reject this all-or-nothing black and white childish framing of it.
xcv123 3y ago
> You can get that much ddr5 today for like $400 Not comparable. M2 Ultra with 128GB RAM has 800 GB/s bandwidth. Maximum bandwidth for a DDR5 Intel 14900K system is 89.6 GB/s. https://www.intel.com/content/www/us/en/products/sku/236773/intel-core-i9-processor-14900k-36m-cache-up-to-6-00-ghz/specifications.html https://www.intel.com/content/www/us/en/products/sku/236773/...
- kjkjadksj 3y ago
  You can only get 2tb of storage max i believe, you are probably bottlenecked elsewhere if you are actually needing to move that much data that fast into memory in practice. I wonder what the temps would even be doing something that maxes that ram bandwidth
  - xcv123 3y ago
    > I wonder what the temps would even be doing something that maxes that ram bandwidth The temps will be completely normal since that is normal operation. TDP is only 80 watts for M2 Ultra. > You can only get 2tb of storage max i believe, you are probably bottlenecked elsewhere No. Just no. This is not the issue. The models are loaded into GPU memory.

theGnuMe 3y ago

yeah but you can't get an nvidia gpu with at least 128gb (H200) for less than $40k

kjkjadksj 3y ago
You can set how much of your system ram your gpu gets in windows and linux
- xcv123 3y ago
  Useless for LLMs. Limited system memory bandwidth kills it. GPU will be mostly idle waiting for memory access. The H200 has 4.8 TB/s bandwidth. System memory is below 90 GB/s
  - kjkjadksj 3y ago
    If you have a use case for an h200 class gpu you are probably making enough money off of it to not care about $40k and the 900gb/s bandwidth mentioned in another comment would seem absolutely pedestrian in comparison. If these things were really that compelling of a performance per dollar ratio you wouldn’t be able to buy them. They’d be sold out because people would be filling datacenters with them.
    - xcv123 3y ago
      M2 Ultra sits between a lower end Intel setup and an H200 for this purpose. I reject this all-or-nothing black and white childish framing of it.

xcv123 3y ago

> You can get that much ddr5 today for like $400 Not comparable. M2 Ultra with 128GB RAM has 800 GB/s bandwidth. Maximum bandwidth for a DDR5 Intel 14900K system is 89.6 GB/s. https://www.intel.com/content/www/us/en/products/sku/236773/intel-core-i9-processor-14900k-36m-cache-up-to-6-00-ghz/specifications.html https://www.intel.com/content/www/us/en/products/sku/236773/...

kjkjadksj 3y ago
You can only get 2tb of storage max i believe, you are probably bottlenecked elsewhere if you are actually needing to move that much data that fast into memory in practice. I wonder what the temps would even be doing something that maxes that ram bandwidth
- xcv123 3y ago
  > I wonder what the temps would even be doing something that maxes that ram bandwidth The temps will be completely normal since that is normal operation. TDP is only 80 watts for M2 Ultra. > You can only get 2tb of storage max i believe, you are probably bottlenecked elsewhere No. Just no. This is not the issue. The models are loaded into GPU memory.