4 ms·

I don't think it's needs 'call us' just a separate SKU, I mean if they called it Xserve Ultra and it was just a studio in a 1u format with dual PSUs and extra R

by matt-p 4mo ago

I don't think it's needs 'call us' just a separate SKU, I mean if they called it Xserve Ultra and it was just a studio in a 1u format with dual PSUs and extra RAM, it would fly off the shelves.

AnthonyMouse 4mo ago
Isn't that the same thing you can get from ordinary 2S Epyc/Xeon servers at a similar price that have 24 memory channels (when the M3 Ultra has the equivalent of 16)? And the reason people rarely use that for AI is that the enterprise GPUs from AMD and Nvidia are only moderately more expensive but are significantly faster because they use HBM instead of DDR5.
- matt-p 4mo ago
  Yeah kind of, I think a 24 channels DDR5 works out approx 1TB/s, but the cost is astronomical, a M5 studio would probably beat that performance for around half the cost. You also get to use the GPU/NPU cores of the mac vs CPU only on the servers. M5 ultra studio with 128GB RAM could probably beat out a sever with a RTX 6000 pro at half the price.
  - AnthonyMouse 4mo ago
    24 channels of DDR5-6400 is 1.2TB/s, M3 ultra is 0.8TB/s. They both use DDR5-6400. > a M5 studio would probably beat that performance for around half the cost. A barebones 2S system with no CPUs or memory is ~$2000, a pair of 16 core CPUs another ~$1000 each, and then however much memory you want. The price seems pretty comparable. The "problem" with doing this is actually that 128GB is too little memory, because you want to populate all the channels, but even using 16GB sticks, 24x16GB is already 384GB. > You also get to use the GPU/NPU cores of the mac vs CPU only on the servers. You only need enough cores to make sure the bottleneck is memory bandwidth.
    - spwa4 4mo ago
      > but even using 16GB sticks, 24x16GB is already 384GB. Question: you need 16GB sticks because they're the smallest doublesided ones, which you need for maximum BW, right? Otherwise why not 8G?
      - AnthonyMouse 4mo ago
        If 8GB sticks of registered DDR5-6400 even exist, they're not common.
        ethbr1 4mo ago
        That was because memory used to be cheap. Will be curious if smaller capacities come back en vogue as a cost cutting mechanism.
        AnthonyMouse 4mo ago
        Using a server system designed for having hundreds of cores and TBs of RAM for LLMs because it has a medium-high amount of memory bandwidth is a hack for enthusiasts who want to take the price/performance trade off to run big models without paying for enterprise GPUs. The market for 8GB RDIMMs would be specifically that, since the more typical server workloads that need that amount of memory bandwidth also want the larger memory sticks (e.g. database servers, systems running hundreds of VMs), or aren't bounded by memory bandwidth to begin with and then don't need to populate all the channels. And if you wanted that market segment then what you'd really do is produce a consumer GPU with 128GB of RAM.
    - matt-p 4mo ago
      M3 ultra is obviously 1-2 generations behind and new the studio is expected 'any day now. Even if this was M4 Ultra it would still be ~comparable to any EPYC system in bandwidth, but get to use the GPU for compute so potentially faster than the EPYC. Total Cost of Ownership in the Epyc is going to be WAY higher because of electricity costs, the EYPC is going to be consuming probably 5X the electricity and is probably not going to sit quietly on your desk. More RAM though, but again it's more about the ratio of RAM (size) to RAM (Memory Bandwith) to Compute and you may find a model bigger than e.g 70b suddenly is bottlenecked by the CPUs or memory bandwidth and therefore the extra RAM (size) is wasted. But maybe not, different use cases will yeild different results I guess. > A barebones 2S system with no CPUs or memory is ~$2000, a pair of 16 core CPUs another ~$1000 each, and then however much memory you want. As you say, the thing is it's not 'however much memory you want' it's 24 sticks which at $300 a stick for 16GB is $7200, then you also need at least one NVME disk so you're looking at what $13,000?

by matt-p 4mo ago

I don't think it's needs 'call us' just a separate SKU, I mean if they called it Xserve Ultra and it was just a studio in a 1u format with dual PSUs and extra RAM, it would fly off the shelves.

AnthonyMouse 4mo ago

Isn't that the same thing you can get from ordinary 2S Epyc/Xeon servers at a similar price that have 24 memory channels (when the M3 Ultra has the equivalent of 16)? And the reason people rarely use that for AI is that the enterprise GPUs from AMD and Nvidia are only moderately more expensive but are significantly faster because they use HBM instead of DDR5.

matt-p 4mo ago
Yeah kind of, I think a 24 channels DDR5 works out approx 1TB/s, but the cost is astronomical, a M5 studio would probably beat that performance for around half the cost. You also get to use the GPU/NPU cores of the mac vs CPU only on the servers. M5 ultra studio with 128GB RAM could probably beat out a sever with a RTX 6000 pro at half the price.
- AnthonyMouse 4mo ago
  24 channels of DDR5-6400 is 1.2TB/s, M3 ultra is 0.8TB/s. They both use DDR5-6400. > a M5 studio would probably beat that performance for around half the cost. A barebones 2S system with no CPUs or memory is ~$2000, a pair of 16 core CPUs another ~$1000 each, and then however much memory you want. The price seems pretty comparable. The "problem" with doing this is actually that 128GB is too little memory, because you want to populate all the channels, but even using 16GB sticks, 24x16GB is already 384GB. > You also get to use the GPU/NPU cores of the mac vs CPU only on the servers. You only need enough cores to make sure the bottleneck is memory bandwidth.
  - spwa4 4mo ago
    > but even using 16GB sticks, 24x16GB is already 384GB. Question: you need 16GB sticks because they're the smallest doublesided ones, which you need for maximum BW, right? Otherwise why not 8G?
    - AnthonyMouse 4mo ago
      If 8GB sticks of registered DDR5-6400 even exist, they're not common.
      - ethbr1 4mo ago
        That was because memory used to be cheap. Will be curious if smaller capacities come back en vogue as a cost cutting mechanism.
        AnthonyMouse 4mo ago
        Using a server system designed for having hundreds of cores and TBs of RAM for LLMs because it has a medium-high amount of memory bandwidth is a hack for enthusiasts who want to take the price/performance trade off to run big models without paying for enterprise GPUs. The market for 8GB RDIMMs would be specifically that, since the more typical server workloads that need that amount of memory bandwidth also want the larger memory sticks (e.g. database servers, systems running hundreds of VMs), or aren't bounded by memory bandwidth to begin with and then don't need to populate all the channels. And if you wanted that market segment then what you'd really do is produce a consumer GPU with 128GB of RAM.
  - matt-p 4mo ago
    M3 ultra is obviously 1-2 generations behind and new the studio is expected 'any day now. Even if this was M4 Ultra it would still be ~comparable to any EPYC system in bandwidth, but get to use the GPU for compute so potentially faster than the EPYC. Total Cost of Ownership in the Epyc is going to be WAY higher because of electricity costs, the EYPC is going to be consuming probably 5X the electricity and is probably not going to sit quietly on your desk. More RAM though, but again it's more about the ratio of RAM (size) to RAM (Memory Bandwith) to Compute and you may find a model bigger than e.g 70b suddenly is bottlenecked by the CPUs or memory bandwidth and therefore the extra RAM (size) is wasted. But maybe not, different use cases will yeild different results I guess. > A barebones 2S system with no CPUs or memory is ~$2000, a pair of 16 core CPUs another ~$1000 each, and then however much memory you want. As you say, the thing is it's not 'however much memory you want' it's 24 sticks which at $300 a stick for 16GB is $7200, then you also need at least one NVME disk so you're looking at what $13,000?